2024-03-22T12:35:50.683 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-22T12:35:50.683 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325 --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --owner mchangir --description fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} --verbose -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/orig.config.yaml 2024-03-22T12:35:50.732 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T12:35:50.811 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325 branch: wip-mchangir-testing-main-20240318.032620 description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} email: mchangir@redhat.com first_in_suite: false job_id: '7616325' kernel: client: branch: testing flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mds beacon grace: 60 mds heartbeat grace: 60 ms die on skipped message: false osd heartbeat grace: 60 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds valgrind exit: true mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon osd crush smoke test: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow request - SLOW_OPS - MON_DOWN sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes watchdog: daemon_restart: normal ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: valgrind: - --tool=memcheck - --leak-check=full - --show-reachable=yes check-counter: counters: mds: - mds.exported - mds.imported install: ceph: debuginfo: true flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6BhfP/x4l8nDPTZnbV3xnODLJoELpLhMe8jZfm+FBBZzddHhUHg7GKzQLOknR/zo4xfve+LI+p66Lb2RTroU0= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4IWQLXvvgXJLm0v+K1UZa29kuyXyIouGFDfsGfDWmjnNHWLCxxQMQGuVH9JaD0cFOC6fhXVNlYmFKQo0WSwsQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-22T12:35:50.812 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-22T12:35:50.812 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-22T12:35:50.813 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-22T12:35:50.814 INFO:teuthology.task.internal:Checking packages... 2024-03-22T12:35:50.838 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-22T12:35:50.839 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T12:35:50.839 INFO:teuthology.packaging:ref: None 2024-03-22T12:35:50.839 INFO:teuthology.packaging:tag: None 2024-03-22T12:35:50.839 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-22T12:35:50.839 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-22T12:35:50.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-22T12:35:51.059 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-22T12:35:51.062 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-22T12:35:51.072 INFO:teuthology.task.internal:no buildpackages task found 2024-03-22T12:35:51.073 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-22T12:35:51.126 INFO:teuthology.task.internal:Saving configuration 2024-03-22T12:35:51.141 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-22T12:35:51.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T12:35:51.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 12:30:53.725282', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6BhfP/x4l8nDPTZnbV3xnODLJoELpLhMe8jZfm+FBBZzddHhUHg7GKzQLOknR/zo4xfve+LI+p66Lb2RTroU0='} 2024-03-22T12:35:51.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 12:30:53.726772', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4IWQLXvvgXJLm0v+K1UZa29kuyXyIouGFDfsGfDWmjnNHWLCxxQMQGuVH9JaD0cFOC6fhXVNlYmFKQo0WSwsQ='} 2024-03-22T12:35:51.249 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-22T12:35:51.262 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-22T12:35:51.262 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-22T12:35:51.262 INFO:teuthology.run_tasks:Running task console_log... 2024-03-22T12:35:51.356 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6c73d96f70>, signals=[15]) 2024-03-22T12:35:51.357 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-22T12:35:51.367 INFO:teuthology.task.internal:Opening connections... 2024-03-22T12:35:51.367 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T12:35:51.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T12:35:51.476 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T12:35:51.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T12:35:51.564 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-22T12:35:51.574 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-03-22T12:35:51.592 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-03-22T12:35:51.592 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-03-22T12:35:51.649 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-03-22T12:35:51.649 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T12:35:51.650 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T12:35:51.651 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-03-22T12:35:51.682 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-03-22T12:35:51.699 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-03-22T12:35:51.699 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T12:35:51.756 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-03-22T12:35:51.757 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-03-22T12:35:51.757 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T12:35:51.757 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-03-22T12:35:51.757 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T12:35:51.757 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T12:35:51.757 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T12:35:51.757 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-03-22T12:35:51.782 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-22T12:35:51.792 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-22T12:35:51.826 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-22T12:35:51.826 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-03-22T12:35:51.829 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-03-22T12:35:51.845 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-22T12:35:51.854 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-22T12:35:51.854 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-03-22T12:35:51.887 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-03-22T12:35:52.012 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-22T12:35:52.067 INFO:teuthology.run_tasks:Running task kernel... 2024-03-22T12:35:52.089 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-22T12:35:52.090 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-22T12:35:52.090 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-22T12:35:52.090 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T12:35:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T12:35:52.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-22T12:35:52.399 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-22T12:35:52.399 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-22T12:35:52.399 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-22T12:35:52.417 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-03-22T12:35:52.417 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-22T12:35:52.418 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-22T12:35:52.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum upgrade -y linux-firmware 2024-03-22T12:35:54.488 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-22T12:35:55.931 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-22T12:36:01.437 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-03-22T12:36:04.213 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-22T12:36:09.397 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-22T12:36:10.781 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 16 kB 00:00 2024-03-22T12:36:12.933 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-22T12:36:17.958 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-22T12:36:20.322 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-22T12:36:20.336 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-22T12:36:20.336 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-22T12:36:20.336 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-22T12:36:20.336 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-22T12:36:20.336 INFO:teuthology.orchestra.run.smithi006.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-22T12:36:20.336 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:36:20.337 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-22T12:36:20.337 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-22T12:36:20.337 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 1 Package 2024-03-22T12:36:20.337 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:36:20.337 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 375 M 2024-03-22T12:36:20.337 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-22T12:36:28.367 INFO:teuthology.orchestra.run.smithi006.stdout:linux-firmware-20240219-143.el9.noarch.rpm 47 MB/s | 375 MB 00:08 2024-03-22T12:36:28.368 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-22T12:36:28.368 INFO:teuthology.orchestra.run.smithi006.stdout:Total 47 MB/s | 375 MB 00:08 2024-03-22T12:36:28.369 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-22T12:36:28.462 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-22T12:36:28.462 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-22T12:36:30.966 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-22T12:36:30.966 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-22T12:36:37.516 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-22T12:36:37.774 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-22T12:36:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-22T12:36:39.251 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-22T12:36:39.251 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-22T12:36:39.645 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-22T12:36:39.645 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:36:39.645 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-22T12:36:39.645 INFO:teuthology.orchestra.run.smithi006.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-22T12:36:39.645 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:36:39.645 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-22T12:36:39.924 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-22T12:36:39.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-22T12:36:40.074 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-22T12:36:40.074 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-22T12:36:40.077 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi006.front.sepia.ceph.com to complete... 2024-03-22T12:37:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22 12:37:09 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-22T12:37:09.410 INFO:teuthology.orchestra.run.smithi006.stderr:FINISHED --2024-03-22 12:37:09-- 2024-03-22T12:37:09.410 INFO:teuthology.orchestra.run.smithi006.stderr:Total wall clock time: 29s 2024-03-22T12:37:09.410 INFO:teuthology.orchestra.run.smithi006.stderr:Downloaded: 1 files, 321M in 29s (11.1 MB/s) 2024-03-22T12:37:09.412 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-22T12:37:09.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-22T12:37:11.441 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying... ######################################## 2024-03-22T12:37:11.506 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing... ######################################## 2024-03-22T12:37:11.593 INFO:teuthology.orchestra.run.smithi006.stdout:Updating / installing... 2024-03-22T12:37:19.419 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-22T12:37:33.398 INFO:teuthology.orchestra.run.smithi006.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-22T12:37:56.651 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-22T12:37:56.651 DEBUG:teuthology.orchestra.run.smithi006:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-22T12:37:56.761 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-22T12:37:56.761 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-22T12:37:56.761 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-22T12:37:56.762 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-22T12:37:56.763 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-22T12:37:56.764 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-22T12:37:56.765 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-22T12:37:56.766 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-22T12:37:56.767 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-22T12:37:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-22T12:37:56.769 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-22T12:37:56.770 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-22T12:37:56.771 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-22T12:37:56.772 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-22T12:37:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-22T12:37:56.774 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-22T12:37:56.775 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-22T12:37:56.776 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-22T12:37:56.777 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-22T12:37:56.778 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-22T12:37:56.779 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-22T12:37:56.780 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-22T12:37:56.781 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-22T12:37:56.782 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-22T12:37:56.783 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-22T12:37:56.784 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-22T12:37:56.785 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-22T12:37:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-22T12:37:56.787 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-22T12:37:56.788 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-22T12:37:56.789 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-22T12:37:56.790 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-22T12:37:56.791 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-22T12:37:56.792 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-22T12:37:56.793 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-22T12:37:56.794 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-22T12:37:56.795 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-22T12:37:56.796 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-22T12:37:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-22T12:37:56.798 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-22T12:37:56.799 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-22T12:37:56.800 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-22T12:37:56.801 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-22T12:37:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-22T12:37:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-22T12:37:56.804 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-22T12:37:56.805 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-22T12:37:56.806 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-22T12:37:56.807 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-22T12:37:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-22T12:37:56.809 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-22T12:37:56.810 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-22T12:37:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-22T12:37:56.812 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-22T12:37:56.813 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-22T12:37:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-22T12:37:56.815 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-22T12:37:56.816 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-22T12:37:56.817 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-22T12:37:56.818 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-22T12:37:56.819 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-22T12:37:56.820 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-22T12:37:56.821 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-22T12:37:56.822 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-22T12:37:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-22T12:37:56.824 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-22T12:37:56.825 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-22T12:37:56.826 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-22T12:37:56.827 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-22T12:37:56.828 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-22T12:37:56.829 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-22T12:37:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-22T12:37:56.831 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-22T12:37:56.832 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-22T12:37:56.833 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-22T12:37:56.834 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-22T12:37:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-22T12:37:56.836 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-22T12:37:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-22T12:37:56.838 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-22T12:37:56.839 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-22T12:37:56.840 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-22T12:37:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-22T12:37:56.842 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-22T12:37:56.843 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-22T12:37:56.844 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-22T12:37:56.845 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-22T12:37:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-22T12:37:56.847 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-22T12:37:56.848 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-22T12:37:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-22T12:37:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-22T12:37:56.851 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-22T12:37:56.852 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-22T12:37:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-22T12:37:56.854 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-22T12:37:56.855 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-22T12:37:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-22T12:37:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-22T12:37:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-22T12:37:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-22T12:37:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-22T12:37:56.861 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-22T12:37:56.862 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-22T12:37:56.863 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-22T12:37:56.864 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-22T12:37:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-22T12:37:56.866 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-22T12:37:56.867 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-22T12:37:56.868 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-22T12:37:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-22T12:37:56.870 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-22T12:37:56.871 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-22T12:37:56.872 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-22T12:37:56.873 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-22T12:37:56.874 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-22T12:37:56.875 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-22T12:37:56.876 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-22T12:37:56.877 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-22T12:37:56.878 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-22T12:37:56.879 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-22T12:37:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-22T12:37:56.881 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-22T12:37:56.882 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-22T12:37:56.883 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-22T12:37:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-22T12:37:56.885 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-22T12:37:56.886 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-22T12:37:56.887 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-22T12:37:56.888 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-22T12:37:56.892 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-22T12:37:56.892 DEBUG:teuthology.orchestra.run.smithi006:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout:postinstall scriptlet (using /bin/sh): 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout:for file in vmlinuz System.map config; do 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-22T12:37:56.914 INFO:teuthology.orchestra.run.smithi006.stdout: fi 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-22T12:37:56.915 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-22T12:37:56.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-03-22T12:37:56.994 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-03-22T12:37:56.994 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-03-22T12:37:56.994 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-03-22T12:37:56.995 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:37:56.996 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-22T12:37:56.996 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-22T12:37:56.996 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-22T12:37:56.996 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-03-22T12:37:56.996 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:37:56.996 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-03-22T12:37:56.997 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-22T12:37:56.997 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 6.8.0-g5ec3e6e88b2e 2024-03-22T12:37:56.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-22T12:37:57.626 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-03-22T12:37:59.083 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-22T12:37:59.127 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-03-22T12:37:59.138 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-22T12:37:59.154 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.MtbciBGYmJ 2024-03-22T12:37:59.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MtbciBGYmJ 2024-03-22T12:37:59.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.MtbciBGYmJ 2024-03-22T12:37:59.376 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.MtbciBGYmJ is 6KB 2024-03-22T12:37:59.425 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.MtbciBGYmJ 2024-03-22T12:37:59.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-03-22T12:37:59.507 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-22T12:37:59.507 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-22T12:37:59.507 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-22T12:37:59.508 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-22T12:37:59.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-22T12:37:59.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-03-22T12:38:29.671 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-22T12:38:29.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-22T12:38:29.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T12:38:48.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-03-22T12:38:57.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-22T12:38:57.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T12:39:57.101 DEBUG:teuthology.orchestra.remote:timed out 2024-03-22T12:40:09.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-22T12:40:09.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T12:40:09.593 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-22T12:40:10.229 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-22T12:40:10.229 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g5ec3e6e88b2e"... 2024-03-22T12:40:10.230 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-22T12:40:10.283 INFO:teuthology.orchestra.run.smithi006.stdout:6.8.0-g5ec3e6e88b2e 2024-03-22T12:40:10.283 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-22T12:40:10.283 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-22T12:40:10.283 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-22T12:40:11.286 DEBUG:teuthology.parallel:result is None 2024-03-22T12:40:11.287 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-22T12:40:11.295 INFO:teuthology.task.internal:Creating test directory... 2024-03-22T12:40:11.295 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T12:40:11.302 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T12:40:11.355 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-22T12:40:11.363 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-22T12:40:11.386 INFO:teuthology.task.internal:Creating archive directory... 2024-03-22T12:40:11.387 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T12:40:11.393 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T12:40:11.493 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-22T12:40:11.501 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-22T12:40:11.501 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T12:40:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T12:40:11.518 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T12:40:11.525 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T12:40:11.621 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T12:40:11.641 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T12:40:11.656 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T12:40:11.729 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T12:40:11.734 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-22T12:40:11.742 INFO:teuthology.task.internal:Configuring sudo... 2024-03-22T12:40:11.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T12:40:11.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T12:40:11.845 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-22T12:40:11.854 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-22T12:40:11.855 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T12:40:11.862 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T12:40:11.911 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T12:40:12.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T12:40:12.112 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T12:40:12.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T12:40:12.304 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:40:12.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T12:40:12.400 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T12:40:12.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T12:40:12.503 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T12:40:12.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T12:40:12.631 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:40:12.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T12:40:12.700 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-03-22T12:40:12.707 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-03-22T12:40:12.781 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T12:40:12.830 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T12:40:13.200 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-22T12:40:13.209 INFO:teuthology.task.internal:Starting timer... 2024-03-22T12:40:13.209 INFO:teuthology.run_tasks:Running task pcp... 2024-03-22T12:40:13.243 INFO:teuthology.run_tasks:Running task selinux... 2024-03-22T12:40:13.274 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-22T12:40:13.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-03-22T12:40:13.404 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-03-22T12:40:13.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-03-22T12:40:13.462 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-03-22T12:40:13.464 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-22T12:40:13.464 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-03-22T12:40:13.524 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-03-22T12:40:13.524 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-03-22T12:40:13.554 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-03-22T12:40:13.555 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive'} 2024-03-22T12:40:13.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T12:40:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T12:40:13.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T12:40:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T12:40:13.676 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-22T12:40:13.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-03-22T12:40:13.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-03-22T12:40:13.806 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-22T12:40:13.817 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-22T12:40:13.823 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-22T12:40:13.935 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-22T12:40:13.950 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-22T12:40:13.951 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi092.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-22T12:46:16.937 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com')] 2024-03-22T12:46:16.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-22T12:46:16.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T12:46:17.088 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-22T12:46:17.243 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-22T12:46:17.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-22T12:46:17.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T12:46:17.331 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-03-22T12:46:17.408 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-22T12:46:17.408 INFO:teuthology.run_tasks:Running task clock... 2024-03-22T12:46:17.419 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-22T12:46:17.419 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T12:46:17.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T12:46:17.427 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T12:46:17.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T12:46:17.483 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T12:46:17.502 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T12:46:17.524 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T12:46:17.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-03-22T12:46:17.552 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-03-22T12:46:17.570 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T12:46:17.580 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T12:46:17.588 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T12:46:17.634 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-22T12:46:17.637 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T12:46:17.637 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-22T12:46:17.637 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.638 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.638 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.638 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.680 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-03-22T12:46:17.726 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-03-22T12:46:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T12:46:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T12:46:17.972 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-22T12:46:17.983 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T12:46:17.983 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-22T12:46:17.983 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.983 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.983 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.983 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T12:46:17.984 INFO:teuthology.run_tasks:Running task install... 2024-03-22T12:46:17.992 DEBUG:teuthology.task.install:project ceph 2024-03-22T12:46:17.993 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-22T12:46:17.993 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-22T12:46:17.993 INFO:teuthology.task.install:Using flavor: default 2024-03-22T12:46:17.998 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-22T12:46:17.999 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-22T12:46:17.999 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-22T12:46:18.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-22T12:46:18.002 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-22T12:46:18.002 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-22T12:46:18.153 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-22T12:46:18.154 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-22T12:46:18.216 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-22T12:46:18.216 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-22T12:46:18.324 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T12:46:18.324 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:46:18.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T12:46:18.326 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T12:46:18.326 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:46:18.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T12:46:18.358 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T12:46:18.358 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T12:46:18.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T12:46:18.430 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T12:46:18.431 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T12:46:18.511 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T12:46:18.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T12:46:18.579 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2024-03-22T12:46:18.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2024-03-22T12:46:18.721 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T12:46:18.816 INFO:teuthology.orchestra.run.smithi092.stdout:74 files removed 2024-03-22T12:46:18.816 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-03-22T12:46:18.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-03-22T12:46:18.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T12:46:19.184 INFO:teuthology.orchestra.run.smithi006.stdout:74 files removed 2024-03-22T12:46:19.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T12:46:19.587 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 181 kB/s | 83 kB 00:00 2024-03-22T12:46:19.946 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-22T12:46:20.224 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 171 kB/s | 83 kB 00:00 2024-03-22T12:46:20.247 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-03-22T12:46:20.675 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-22T12:46:20.875 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 18 kB/s | 10 kB 00:00 2024-03-22T12:46:21.451 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-22T12:46:22.046 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-22T12:46:22.083 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-22T12:46:23.077 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-22T12:46:23.688 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-22T12:46:24.651 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-22T12:46:26.550 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-03-22T12:46:26.669 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-22T12:46:30.508 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-22T12:46:31.595 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.1 MB 00:01 2024-03-22T12:46:32.018 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-22T12:46:33.262 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-22T12:46:33.405 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-22T12:46:37.431 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 21 MB 00:03 2024-03-22T12:46:38.302 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-22T12:46:40.672 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T12:46:40.673 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T12:46:40.702 INFO:teuthology.orchestra.run.smithi092.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T12:46:40.703 INFO:teuthology.orchestra.run.smithi092.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T12:46:40.704 INFO:teuthology.orchestra.run.smithi092.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T12:46:40.705 INFO:teuthology.orchestra.run.smithi092.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T12:46:40.707 INFO:teuthology.orchestra.run.smithi092.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T12:46:40.709 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T12:46:40.710 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T12:46:40.793 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout:==================================================================================================== 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout:==================================================================================================== 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-22T12:46:40.822 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 69 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 439 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.1 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.4 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 52 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 M 2024-03-22T12:46:40.823 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 290 M 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 178 M 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 679 M 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-22T12:46:40.824 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.5 M 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 109 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 M 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 52 M 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 615 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 769 k 2024-03-22T12:46:40.825 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.1 M 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 338 k 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 516 k 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 52 M 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.3 M 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T12:46:40.826 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-22T12:46:40.827 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-22T12:46:40.828 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 23 M 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T12:46:40.829 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T12:46:40.830 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T12:46:40.831 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T12:46:40.832 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T12:46:40.833 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T12:46:40.834 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T12:46:40.835 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout:==================================================================================================== 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout:Install 172 Packages 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 24 Packages 2024-03-22T12:46:40.836 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:46:40.837 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 2.2 G 2024-03-22T12:46:40.837 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-22T12:46:42.353 INFO:teuthology.orchestra.run.smithi092.stdout:(1/196): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-22T12:46:42.545 INFO:teuthology.orchestra.run.smithi092.stdout:(2/196): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 5.4 MB 00:00 2024-03-22T12:46:43.389 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-22T12:46:46.089 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T12:46:46.091 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T12:46:46.134 INFO:teuthology.orchestra.run.smithi006.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T12:46:46.135 INFO:teuthology.orchestra.run.smithi006.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T12:46:46.136 INFO:teuthology.orchestra.run.smithi006.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T12:46:46.137 INFO:teuthology.orchestra.run.smithi006.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T12:46:46.139 INFO:teuthology.orchestra.run.smithi006.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T12:46:46.142 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T12:46:46.144 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T12:46:46.239 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout:==================================================================================================== 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout:==================================================================================================== 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 69 M 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 439 M 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-22T12:46:46.265 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.1 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.4 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 52 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 290 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-22T12:46:46.266 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 178 M 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 679 M 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.5 M 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-22T12:46:46.267 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 109 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 M 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 52 M 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 615 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 769 k 2024-03-22T12:46:46.268 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.1 M 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 338 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 516 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 52 M 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.3 M 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T12:46:46.269 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T12:46:46.270 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-debuginfo x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 23 M 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-22T12:46:46.271 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T12:46:46.272 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T12:46:46.273 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T12:46:46.274 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T12:46:46.275 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T12:46:46.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T12:46:46.277 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T12:46:46.278 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout:==================================================================================================== 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout:Install 172 Packages 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 24 Packages 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:46:46.279 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.2 G 2024-03-22T12:46:46.280 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-22T12:46:47.691 INFO:teuthology.orchestra.run.smithi006.stdout:(1/196): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-22T12:46:48.694 INFO:teuthology.orchestra.run.smithi092.stdout:(3/196): ceph-common-19.0.0-2145.g3be9a5c9.el9. 3.2 MB/s | 20 MB 00:06 2024-03-22T12:46:49.741 INFO:teuthology.orchestra.run.smithi006.stdout:(2/196): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-03-22T12:46:53.251 INFO:teuthology.orchestra.run.smithi092.stdout:(4/196): ceph-base-debuginfo-19.0.0-2145.g3be9a 6.1 MB/s | 69 MB 00:11 2024-03-22T12:46:53.619 INFO:teuthology.orchestra.run.smithi092.stdout:(5/196): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.4 MB/s | 890 kB 00:00 2024-03-22T12:46:54.711 INFO:teuthology.orchestra.run.smithi092.stdout:(6/196): ceph-fuse-debuginfo-19.0.0-2145.g3be9a 5.6 MB/s | 6.1 MB 00:01 2024-03-22T12:46:54.803 INFO:teuthology.orchestra.run.smithi092.stdout:(7/196): ceph-immutable-object-cache-19.0.0-214 1.6 MB/s | 145 kB 00:00 2024-03-22T12:46:55.095 INFO:teuthology.orchestra.run.smithi092.stdout:(8/196): ceph-immutable-object-cache-debuginfo- 4.9 MB/s | 1.4 MB 00:00 2024-03-22T12:46:55.528 INFO:teuthology.orchestra.run.smithi092.stdout:(9/196): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 5.3 MB/s | 2.3 MB 00:00 2024-03-22T12:46:57.707 INFO:teuthology.orchestra.run.smithi006.stdout:(3/196): ceph-common-19.0.0-2145.g3be9a5c9.el9. 2.0 MB/s | 20 MB 00:10 2024-03-22T12:47:04.052 INFO:teuthology.orchestra.run.smithi092.stdout:(10/196): ceph-mds-debuginfo-19.0.0-2145.g3be9a 6.2 MB/s | 52 MB 00:08 2024-03-22T12:47:04.278 INFO:teuthology.orchestra.run.smithi092.stdout:(11/196): ceph-debuginfo-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 23 MB 00:15 2024-03-22T12:47:04.628 INFO:teuthology.orchestra.run.smithi092.stdout:(12/196): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x8 2.6 MB/s | 1.5 MB 00:00 2024-03-22T12:47:04.866 INFO:teuthology.orchestra.run.smithi006.stdout:(4/196): ceph-debuginfo-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 23 MB 00:07 2024-03-22T12:47:05.349 INFO:teuthology.orchestra.run.smithi006.stdout:(5/196): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.8 MB/s | 890 kB 00:00 2024-03-22T12:47:05.461 INFO:teuthology.orchestra.run.smithi092.stdout:(13/196): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x8 5.6 MB/s | 4.7 MB 00:00 2024-03-22T12:47:07.166 INFO:teuthology.orchestra.run.smithi006.stdout:(6/196): ceph-base-debuginfo-19.0.0-2145.g3be9a 3.5 MB/s | 69 MB 00:19 2024-03-22T12:47:07.346 INFO:teuthology.orchestra.run.smithi006.stdout:(7/196): ceph-immutable-object-cache-19.0.0-214 813 kB/s | 145 kB 00:00 2024-03-22T12:47:07.525 INFO:teuthology.orchestra.run.smithi006.stdout:(8/196): ceph-fuse-debuginfo-19.0.0-2145.g3be9a 2.8 MB/s | 6.1 MB 00:02 2024-03-22T12:47:07.842 INFO:teuthology.orchestra.run.smithi006.stdout:(9/196): ceph-immutable-object-cache-debuginfo- 2.9 MB/s | 1.4 MB 00:00 2024-03-22T12:47:08.367 INFO:teuthology.orchestra.run.smithi006.stdout:(10/196): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x8 2.7 MB/s | 2.3 MB 00:00 2024-03-22T12:47:08.825 INFO:teuthology.orchestra.run.smithi006.stdout:(11/196): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x8 3.3 MB/s | 1.5 MB 00:00 2024-03-22T12:47:12.502 INFO:teuthology.orchestra.run.smithi092.stdout:(14/196): ceph-mgr-debuginfo-19.0.0-2145.g3be9a 2.5 MB/s | 20 MB 00:08 2024-03-22T12:47:14.001 INFO:teuthology.orchestra.run.smithi006.stdout:(12/196): ceph-mgr-debuginfo-19.0.0-2145.g3be9a 3.9 MB/s | 20 MB 00:05 2024-03-22T12:47:15.442 INFO:teuthology.orchestra.run.smithi006.stdout:(13/196): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x8 3.3 MB/s | 4.7 MB 00:01 2024-03-22T12:47:16.618 INFO:teuthology.orchestra.run.smithi006.stdout:(14/196): ceph-mds-debuginfo-19.0.0-2145.g3be9a 6.0 MB/s | 52 MB 00:08 2024-03-22T12:47:18.143 INFO:teuthology.orchestra.run.smithi092.stdout:(15/196): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 3.0 MB/s | 17 MB 00:05 2024-03-22T12:47:18.451 INFO:teuthology.orchestra.run.smithi006.stdout:(15/196): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 9.2 MB/s | 17 MB 00:01 2024-03-22T12:47:27.475 INFO:teuthology.orchestra.run.smithi092.stdout:(16/196): ceph-mon-debuginfo-19.0.0-2145.g3be9a 4.5 MB/s | 99 MB 00:22 2024-03-22T12:47:30.634 INFO:teuthology.orchestra.run.smithi092.stdout:(17/196): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 3.1 MB/s | 9.9 MB 00:03 2024-03-22T12:47:33.943 INFO:teuthology.orchestra.run.smithi006.stdout:(16/196): ceph-mon-debuginfo-19.0.0-2145.g3be9a 5.4 MB/s | 99 MB 00:18 2024-03-22T12:47:36.226 INFO:teuthology.orchestra.run.smithi006.stdout:(17/196): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 4.3 MB/s | 9.9 MB 00:02 2024-03-22T12:48:04.637 INFO:teuthology.orchestra.run.smithi092.stdout:(18/196): ceph-osd-debuginfo-19.0.0-2145.g3be9a 6.2 MB/s | 290 MB 00:46 2024-03-22T12:48:05.061 INFO:teuthology.orchestra.run.smithi092.stdout:(19/196): ceph-selinux-19.0.0-2145.g3be9a5c9.el 59 kB/s | 25 kB 00:00 2024-03-22T12:48:07.474 INFO:teuthology.orchestra.run.smithi006.stdout:(18/196): ceph-common-debuginfo-19.0.0-2145.g3b 5.7 MB/s | 439 MB 01:17 2024-03-22T12:48:07.557 INFO:teuthology.orchestra.run.smithi006.stdout:(19/196): ceph-selinux-19.0.0-2145.g3be9a5c9.el 306 kB/s | 25 kB 00:00 2024-03-22T12:48:13.443 INFO:teuthology.orchestra.run.smithi092.stdout:(20/196): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 5.7 MB/s | 48 MB 00:08 2024-03-22T12:48:13.640 INFO:teuthology.orchestra.run.smithi006.stdout:(20/196): ceph-osd-debuginfo-19.0.0-2145.g3be9a 5.2 MB/s | 290 MB 00:55 2024-03-22T12:48:14.515 INFO:teuthology.orchestra.run.smithi006.stdout:(21/196): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 6.8 MB/s | 48 MB 00:06 2024-03-22T12:48:14.665 INFO:teuthology.orchestra.run.smithi006.stdout:(22/196): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.4 MB/s | 220 kB 00:00 2024-03-22T12:48:14.757 INFO:teuthology.orchestra.run.smithi006.stdout:(23/196): cephfs-shell-19.0.0-2145.g3be9a5c9.el 299 kB/s | 27 kB 00:00 2024-03-22T12:48:14.841 INFO:teuthology.orchestra.run.smithi006.stdout:(24/196): libcephfs-devel-19.0.0-2145.g3be9a5c9 380 kB/s | 31 kB 00:00 2024-03-22T12:48:15.041 INFO:teuthology.orchestra.run.smithi006.stdout:(25/196): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.7 MB/s | 746 kB 00:00 2024-03-22T12:48:15.950 INFO:teuthology.orchestra.run.smithi006.stdout:(26/196): libcephfs2-debuginfo-19.0.0-2145.g3be 6.0 MB/s | 5.5 MB 00:00 2024-03-22T12:48:16.734 INFO:teuthology.orchestra.run.smithi006.stdout:(27/196): ceph-radosgw-debuginfo-19.0.0-2145.g3 4.4 MB/s | 178 MB 00:40 2024-03-22T12:48:16.767 INFO:teuthology.orchestra.run.smithi006.stdout:(28/196): libcephsqlite-19.0.0-2145.g3be9a5c9.e 200 kB/s | 163 kB 00:00 2024-03-22T12:48:16.834 INFO:teuthology.orchestra.run.smithi006.stdout:(29/196): librados-devel-19.0.0-2145.g3be9a5c9. 1.3 MB/s | 127 kB 00:00 2024-03-22T12:48:16.867 INFO:teuthology.orchestra.run.smithi006.stdout:(30/196): librados-devel-debuginfo-19.0.0-2145. 1.1 MB/s | 109 kB 00:00 2024-03-22T12:48:17.043 INFO:teuthology.orchestra.run.smithi006.stdout:(31/196): libradosstriper1-19.0.0-2145.g3be9a5c 2.8 MB/s | 501 kB 00:00 2024-03-22T12:48:18.151 INFO:teuthology.orchestra.run.smithi092.stdout:(21/196): ceph-common-debuginfo-19.0.0-2145.g3b 4.6 MB/s | 439 MB 01:35 2024-03-22T12:48:18.409 INFO:teuthology.orchestra.run.smithi092.stdout:(22/196): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 854 kB/s | 220 kB 00:00 2024-03-22T12:48:18.493 INFO:teuthology.orchestra.run.smithi092.stdout:(23/196): cephfs-shell-19.0.0-2145.g3be9a5c9.el 326 kB/s | 27 kB 00:00 2024-03-22T12:48:18.585 INFO:teuthology.orchestra.run.smithi092.stdout:(24/196): libcephfs-devel-19.0.0-2145.g3be9a5c9 341 kB/s | 31 kB 00:00 2024-03-22T12:48:18.795 INFO:teuthology.orchestra.run.smithi092.stdout:(25/196): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.5 MB/s | 746 kB 00:00 2024-03-22T12:48:19.662 INFO:teuthology.orchestra.run.smithi092.stdout:(26/196): libcephfs2-debuginfo-19.0.0-2145.g3be 6.3 MB/s | 5.5 MB 00:00 2024-03-22T12:48:19.754 INFO:teuthology.orchestra.run.smithi092.stdout:(27/196): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-22T12:48:19.846 INFO:teuthology.orchestra.run.smithi092.stdout:(28/196): librados-devel-19.0.0-2145.g3be9a5c9. 1.3 MB/s | 127 kB 00:00 2024-03-22T12:48:19.946 INFO:teuthology.orchestra.run.smithi092.stdout:(29/196): librados-devel-debuginfo-19.0.0-2145. 1.1 MB/s | 109 kB 00:00 2024-03-22T12:48:23.410 INFO:teuthology.orchestra.run.smithi006.stdout:(32/196): librbd1-debuginfo-19.0.0-2145.g3be9a5 8.2 MB/s | 52 MB 00:06 2024-03-22T12:48:24.785 INFO:teuthology.orchestra.run.smithi006.stdout:(33/196): librados2-debuginfo-19.0.0-2145.g3be9 5.7 MB/s | 45 MB 00:07 2024-03-22T12:48:24.902 INFO:teuthology.orchestra.run.smithi006.stdout:(34/196): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.5 MB/s | 5.1 MB 00:01 2024-03-22T12:48:24.927 INFO:teuthology.orchestra.run.smithi006.stdout:(35/196): python3-ceph-argparse-19.0.0-2145.g3b 321 kB/s | 45 kB 00:00 2024-03-22T12:48:24.961 INFO:teuthology.orchestra.run.smithi006.stdout:(36/196): python3-ceph-common-19.0.0-2145.g3be9 2.2 MB/s | 131 kB 00:00 2024-03-22T12:48:24.994 INFO:teuthology.orchestra.run.smithi006.stdout:(37/196): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-22T12:48:25.096 INFO:teuthology.orchestra.run.smithi006.stdout:(38/196): python3-cephfs-debuginfo-19.0.0-2145. 4.5 MB/s | 615 kB 00:00 2024-03-22T12:48:25.128 INFO:teuthology.orchestra.run.smithi006.stdout:(39/196): python3-rados-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 321 kB 00:00 2024-03-22T12:48:25.238 INFO:teuthology.orchestra.run.smithi092.stdout:(30/196): ceph-radosgw-debuginfo-19.0.0-2145.g3 3.3 MB/s | 178 MB 00:54 2024-03-22T12:48:25.479 INFO:teuthology.orchestra.run.smithi006.stdout:(40/196): python3-rados-debuginfo-19.0.0-2145.g 2.0 MB/s | 769 kB 00:00 2024-03-22T12:48:25.512 INFO:teuthology.orchestra.run.smithi006.stdout:(41/196): python3-rbd-19.0.0-2145.g3be9a5c9.el9 780 kB/s | 298 kB 00:00 2024-03-22T12:48:25.538 INFO:teuthology.orchestra.run.smithi092.stdout:(31/196): libradosstriper1-19.0.0-2145.g3be9a5c 1.6 MB/s | 501 kB 00:00 2024-03-22T12:48:25.579 INFO:teuthology.orchestra.run.smithi006.stdout:(42/196): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 99 kB 00:00 2024-03-22T12:48:25.713 INFO:teuthology.orchestra.run.smithi006.stdout:(43/196): python3-rbd-debuginfo-19.0.0-2145.g3b 4.8 MB/s | 1.1 MB 00:00 2024-03-22T12:48:25.746 INFO:teuthology.orchestra.run.smithi006.stdout:(44/196): python3-rgw-debuginfo-19.0.0-2145.g3b 2.0 MB/s | 338 kB 00:00 2024-03-22T12:48:25.788 INFO:teuthology.orchestra.run.smithi006.stdout:(45/196): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-22T12:48:25.922 INFO:teuthology.orchestra.run.smithi006.stdout:(46/196): rbd-fuse-debuginfo-19.0.0-2145.g3be9a 2.9 MB/s | 516 kB 00:00 2024-03-22T12:48:26.331 INFO:teuthology.orchestra.run.smithi006.stdout:(47/196): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 5.7 MB/s | 3.1 MB 00:00 2024-03-22T12:48:26.406 INFO:teuthology.orchestra.run.smithi006.stdout:(48/196): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.3 MB/s | 170 kB 00:00 2024-03-22T12:48:26.488 INFO:teuthology.orchestra.run.smithi092.stdout:(32/196): librados2-debuginfo-19.0.0-2145.g3be9 6.9 MB/s | 45 MB 00:06 2024-03-22T12:48:26.648 INFO:teuthology.orchestra.run.smithi006.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-2145.g3be9a5 5.3 MB/s | 1.3 MB 00:00 2024-03-22T12:48:26.739 INFO:teuthology.orchestra.run.smithi006.stdout:(50/196): ceph-grafana-dashboards-19.0.0-2145.g 374 kB/s | 34 kB 00:00 2024-03-22T12:48:26.840 INFO:teuthology.orchestra.run.smithi006.stdout:(51/196): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.4 MB/s | 139 kB 00:00 2024-03-22T12:48:27.249 INFO:teuthology.orchestra.run.smithi006.stdout:(52/196): ceph-mgr-dashboard-19.0.0-2145.g3be9a 4.4 MB/s | 1.8 MB 00:00 2024-03-22T12:48:27.280 INFO:teuthology.orchestra.run.smithi092.stdout:(33/196): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 6.5 MB/s | 5.1 MB 00:00 2024-03-22T12:48:27.338 INFO:teuthology.orchestra.run.smithi092.stdout:(34/196): python3-ceph-argparse-19.0.0-2145.g3b 777 kB/s | 45 kB 00:00 2024-03-22T12:48:27.406 INFO:teuthology.orchestra.run.smithi092.stdout:(35/196): python3-ceph-common-19.0.0-2145.g3be9 1.9 MB/s | 131 kB 00:00 2024-03-22T12:48:27.481 INFO:teuthology.orchestra.run.smithi092.stdout:(36/196): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 162 kB 00:00 2024-03-22T12:48:27.615 INFO:teuthology.orchestra.run.smithi092.stdout:(37/196): python3-cephfs-debuginfo-19.0.0-2145. 4.5 MB/s | 615 kB 00:00 2024-03-22T12:48:27.698 INFO:teuthology.orchestra.run.smithi092.stdout:(38/196): python3-rados-19.0.0-2145.g3be9a5c9.e 3.8 MB/s | 321 kB 00:00 2024-03-22T12:48:27.840 INFO:teuthology.orchestra.run.smithi092.stdout:(39/196): python3-rados-debuginfo-19.0.0-2145.g 5.3 MB/s | 769 kB 00:00 2024-03-22T12:48:27.924 INFO:teuthology.orchestra.run.smithi092.stdout:(40/196): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.5 MB/s | 298 kB 00:00 2024-03-22T12:48:28.133 INFO:teuthology.orchestra.run.smithi092.stdout:(41/196): python3-rbd-debuginfo-19.0.0-2145.g3b 5.4 MB/s | 1.1 MB 00:00 2024-03-22T12:48:28.208 INFO:teuthology.orchestra.run.smithi092.stdout:(42/196): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-22T12:48:28.333 INFO:teuthology.orchestra.run.smithi092.stdout:(43/196): python3-rgw-debuginfo-19.0.0-2145.g3b 2.6 MB/s | 338 kB 00:00 2024-03-22T12:48:28.400 INFO:teuthology.orchestra.run.smithi092.stdout:(44/196): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-22T12:48:28.484 INFO:teuthology.orchestra.run.smithi092.stdout:(45/196): rbd-fuse-debuginfo-19.0.0-2145.g3be9a 6.1 MB/s | 516 kB 00:00 2024-03-22T12:48:28.490 INFO:teuthology.orchestra.run.smithi006.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 6.0 MB/s | 7.4 MB 00:01 2024-03-22T12:48:28.591 INFO:teuthology.orchestra.run.smithi006.stdout:(54/196): ceph-mgr-modules-core-19.0.0-2145.g3b 2.5 MB/s | 248 kB 00:00 2024-03-22T12:48:28.666 INFO:teuthology.orchestra.run.smithi006.stdout:(55/196): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 665 kB/s | 49 kB 00:00 2024-03-22T12:48:28.749 INFO:teuthology.orchestra.run.smithi006.stdout:(56/196): ceph-prometheus-alerts-19.0.0-2145.g3 200 kB/s | 16 kB 00:00 2024-03-22T12:48:28.841 INFO:teuthology.orchestra.run.smithi006.stdout:(57/196): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 264 kB 00:00 2024-03-22T12:48:28.917 INFO:teuthology.orchestra.run.smithi092.stdout:(46/196): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 7.1 MB/s | 3.1 MB 00:00 2024-03-22T12:48:29.042 INFO:teuthology.orchestra.run.smithi006.stdout:(58/196): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.7 MB/s | 748 kB 00:00 2024-03-22T12:48:29.142 INFO:teuthology.orchestra.run.smithi006.stdout:(59/196): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 184 kB/s | 18 kB 00:00 2024-03-22T12:48:29.334 INFO:teuthology.orchestra.run.smithi006.stdout:(60/196): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-22T12:48:29.401 INFO:teuthology.orchestra.run.smithi006.stdout:(61/196): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-22T12:48:29.443 INFO:teuthology.orchestra.run.smithi006.stdout:(62/196): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-22T12:48:29.485 INFO:teuthology.orchestra.run.smithi006.stdout:(63/196): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2024-03-22T12:48:29.519 INFO:teuthology.orchestra.run.smithi006.stdout:(64/196): flexiblas-3.0.4-8.el9.x86_64.rpm 969 kB/s | 32 kB 00:00 2024-03-22T12:48:29.636 INFO:teuthology.orchestra.run.smithi006.stdout:(65/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-22T12:48:29.669 INFO:teuthology.orchestra.run.smithi006.stdout:(66/196): flexiblas-openblas-openmp-3.0.4-8.el9 528 kB/s | 17 kB 00:00 2024-03-22T12:48:29.728 INFO:teuthology.orchestra.run.smithi006.stdout:(67/196): libaio-devel-0.3.111-13.el9.x86_64.rp 175 kB/s | 10 kB 00:00 2024-03-22T12:48:29.762 INFO:teuthology.orchestra.run.smithi006.stdout:(68/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.8 MB/s | 159 kB 00:00 2024-03-22T12:48:29.795 INFO:teuthology.orchestra.run.smithi006.stdout:(69/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-22T12:48:29.845 INFO:teuthology.orchestra.run.smithi006.stdout:(70/196): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-22T12:48:29.896 INFO:teuthology.orchestra.run.smithi006.stdout:(71/196): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-22T12:48:29.921 INFO:teuthology.orchestra.run.smithi006.stdout:(72/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.9 MB/s | 46 kB 00:00 2024-03-22T12:48:29.963 INFO:teuthology.orchestra.run.smithi006.stdout:(73/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 282 kB 00:00 2024-03-22T12:48:30.005 INFO:teuthology.orchestra.run.smithi006.stdout:(74/196): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-22T12:48:30.056 INFO:teuthology.orchestra.run.smithi006.stdout:(75/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-22T12:48:30.089 INFO:teuthology.orchestra.run.smithi006.stdout:(76/196): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-22T12:48:30.248 INFO:teuthology.orchestra.run.smithi006.stdout:(77/196): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-22T12:48:30.332 INFO:teuthology.orchestra.run.smithi006.stdout:(78/196): postgresql-13.11-1.el9.x86_64.rpm 18 MB/s | 1.5 MB 00:00 2024-03-22T12:48:30.390 INFO:teuthology.orchestra.run.smithi006.stdout:(79/196): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-22T12:48:30.432 INFO:teuthology.orchestra.run.smithi006.stdout:(80/196): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-22T12:48:30.591 INFO:teuthology.orchestra.run.smithi006.stdout:(81/196): postgresql-server-13.11-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-22T12:48:30.658 INFO:teuthology.orchestra.run.smithi006.stdout:(82/196): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-22T12:48:30.700 INFO:teuthology.orchestra.run.smithi006.stdout:(83/196): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-03-22T12:48:30.883 INFO:teuthology.orchestra.run.smithi006.stdout:(84/196): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-22T12:48:30.925 INFO:teuthology.orchestra.run.smithi006.stdout:(85/196): python3-jsonpatch-1.21-16.el9.noarch. 602 kB/s | 24 kB 00:00 2024-03-22T12:48:30.959 INFO:teuthology.orchestra.run.smithi006.stdout:(86/196): python3-jsonpointer-2.0-4.el9.noarch. 559 kB/s | 18 kB 00:00 2024-03-22T12:48:31.001 INFO:teuthology.orchestra.run.smithi006.stdout:(87/196): python3-mako-1.1.4-6.el9.noarch.rpm 3.5 MB/s | 144 kB 00:00 2024-03-22T12:48:31.043 INFO:teuthology.orchestra.run.smithi006.stdout:(88/196): python3-markupsafe-1.1.1-12.el9.x86_6 770 kB/s | 32 kB 00:00 2024-03-22T12:48:31.194 INFO:teuthology.orchestra.run.smithi006.stdout:(89/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-22T12:48:31.236 INFO:teuthology.orchestra.run.smithi006.stdout:(90/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.2 MB/s | 217 kB 00:00 2024-03-22T12:48:31.278 INFO:teuthology.orchestra.run.smithi006.stdout:(91/196): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-22T12:48:31.311 INFO:teuthology.orchestra.run.smithi006.stdout:(92/196): python3-packaging-20.9-5.el9.noarch.r 2.1 MB/s | 69 kB 00:00 2024-03-22T12:48:31.345 INFO:teuthology.orchestra.run.smithi006.stdout:(93/196): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-22T12:48:31.387 INFO:teuthology.orchestra.run.smithi006.stdout:(94/196): python3-protobuf-3.14.0-13.el9.noarch 5.7 MB/s | 238 kB 00:00 2024-03-22T12:48:31.454 INFO:teuthology.orchestra.run.smithi006.stdout:(95/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-22T12:48:31.496 INFO:teuthology.orchestra.run.smithi006.stdout:(96/196): python3-pyasn1-modules-0.4.8-6.el9.no 5.1 MB/s | 211 kB 00:00 2024-03-22T12:48:31.538 INFO:teuthology.orchestra.run.smithi006.stdout:(97/196): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-22T12:48:31.580 INFO:teuthology.orchestra.run.smithi006.stdout:(98/196): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-22T12:48:32.064 INFO:teuthology.orchestra.run.smithi006.stdout:(99/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-22T12:48:32.105 INFO:teuthology.orchestra.run.smithi006.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 956 kB/s | 39 kB 00:00 2024-03-22T12:48:32.147 INFO:teuthology.orchestra.run.smithi006.stdout:(101/196): python3-wcwidth-0.2.5-8.el9.noarch.r 1.0 MB/s | 41 kB 00:00 2024-03-22T12:48:32.206 INFO:teuthology.orchestra.run.smithi006.stdout:(102/196): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-22T12:48:32.248 INFO:teuthology.orchestra.run.smithi006.stdout:(103/196): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 56 kB 00:00 2024-03-22T12:48:32.290 INFO:teuthology.orchestra.run.smithi006.stdout:(104/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-22T12:48:32.332 INFO:teuthology.orchestra.run.smithi006.stdout:(105/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-22T12:48:32.374 INFO:teuthology.orchestra.run.smithi006.stdout:(106/196): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-22T12:48:32.425 INFO:teuthology.orchestra.run.smithi006.stdout:(107/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-03-22T12:48:32.508 INFO:teuthology.orchestra.run.smithi006.stdout:(108/196): python3-cryptography-36.0.1-4.el9.x8 14 MB/s | 1.1 MB 00:00 2024-03-22T12:48:32.541 INFO:teuthology.orchestra.run.smithi006.stdout:(109/196): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 103 kB 00:00 2024-03-22T12:48:32.600 INFO:teuthology.orchestra.run.smithi006.stdout:(110/196): python3-pycparser-2.20-6.el9.noarch. 2.1 MB/s | 124 kB 00:00 2024-03-22T12:48:32.642 INFO:teuthology.orchestra.run.smithi006.stdout:(111/196): python3-urllib3-1.26.5-5.el9.noarch. 4.5 MB/s | 187 kB 00:00 2024-03-22T12:48:32.959 INFO:teuthology.orchestra.run.smithi006.stdout:(112/196): jq-1.6-16.el9.x86_64.rpm 593 kB/s | 187 kB 00:00 2024-03-22T12:48:33.093 INFO:teuthology.orchestra.run.smithi006.stdout:(113/196): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2024-03-22T12:48:33.177 INFO:teuthology.orchestra.run.smithi006.stdout:(114/196): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-22T12:48:33.244 INFO:teuthology.orchestra.run.smithi006.stdout:(115/196): python3-requests-2.25.1-8.el9.noarch 1.9 MB/s | 125 kB 00:00 2024-03-22T12:48:33.352 INFO:teuthology.orchestra.run.smithi006.stdout:(116/196): elfutils-libelf-devel-0.190-2.el9.x8 200 kB/s | 21 kB 00:00 2024-03-22T12:48:33.686 INFO:teuthology.orchestra.run.smithi006.stdout:(117/196): libuuid-devel-2.37.4-18.el9.x86_64.r 76 kB/s | 25 kB 00:00 2024-03-22T12:48:34.659 INFO:teuthology.orchestra.run.smithi092.stdout:(47/196): rbd-mirror-debuginfo-19.0.0-2145.g3be 9.0 MB/s | 52 MB 00:05 2024-03-22T12:48:34.709 INFO:teuthology.orchestra.run.smithi092.stdout:(48/196): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.4 MB/s | 170 kB 00:00 2024-03-22T12:48:34.770 INFO:teuthology.orchestra.run.smithi006.stdout:(118/196): rbd-mirror-debuginfo-19.0.0-2145.g3b 5.9 MB/s | 52 MB 00:08 2024-03-22T12:48:34.859 INFO:teuthology.orchestra.run.smithi092.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-2145.g3be9a5 8.6 MB/s | 1.3 MB 00:00 2024-03-22T12:48:34.909 INFO:teuthology.orchestra.run.smithi092.stdout:(50/196): ceph-grafana-dashboards-19.0.0-2145.g 680 kB/s | 34 kB 00:00 2024-03-22T12:48:34.961 INFO:teuthology.orchestra.run.smithi006.stdout:(119/196): python3-devel-3.9.17-2.el9.x86_64.rp 1.3 MB/s | 245 kB 00:00 2024-03-22T12:48:34.976 INFO:teuthology.orchestra.run.smithi092.stdout:(51/196): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-22T12:48:35.020 INFO:teuthology.orchestra.run.smithi006.stdout:(120/196): python3-jinja2-2.11.3-5.el9.noarch.r 4.2 MB/s | 248 kB 00:00 2024-03-22T12:48:35.079 INFO:teuthology.orchestra.run.smithi006.stdout:(121/196): qatlib-23.11.0-1.el9.x86_64.rpm 3.7 MB/s | 216 kB 00:00 2024-03-22T12:48:35.177 INFO:teuthology.orchestra.run.smithi092.stdout:(52/196): ceph-mgr-dashboard-19.0.0-2145.g3be9a 9.0 MB/s | 1.8 MB 00:00 2024-03-22T12:48:35.179 INFO:teuthology.orchestra.run.smithi006.stdout:(122/196): qatlib-service-23.11.0-1.el9.x86_64. 353 kB/s | 35 kB 00:00 2024-03-22T12:48:35.354 INFO:teuthology.orchestra.run.smithi006.stdout:(123/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 255 kB/s | 44 kB 00:00 2024-03-22T12:48:35.421 INFO:teuthology.orchestra.run.smithi006.stdout:(124/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-22T12:48:35.564 INFO:teuthology.orchestra.run.smithi006.stdout:(125/196): lua-devel-5.4.4-4.el9.x86_64.rpm 158 kB/s | 22 kB 00:00 2024-03-22T12:48:35.852 INFO:teuthology.orchestra.run.smithi092.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-22T12:48:35.919 INFO:teuthology.orchestra.run.smithi092.stdout:(54/196): ceph-mgr-modules-core-19.0.0-2145.g3b 3.6 MB/s | 248 kB 00:00 2024-03-22T12:48:35.964 INFO:teuthology.orchestra.run.smithi006.stdout:(126/196): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-22T12:48:35.969 INFO:teuthology.orchestra.run.smithi092.stdout:(55/196): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 991 kB/s | 49 kB 00:00 2024-03-22T12:48:36.019 INFO:teuthology.orchestra.run.smithi092.stdout:(56/196): ceph-prometheus-alerts-19.0.0-2145.g3 329 kB/s | 16 kB 00:00 2024-03-22T12:48:36.087 INFO:teuthology.orchestra.run.smithi092.stdout:(57/196): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 264 kB 00:00 2024-03-22T12:48:36.187 INFO:teuthology.orchestra.run.smithi092.stdout:(58/196): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-22T12:48:36.237 INFO:teuthology.orchestra.run.smithi092.stdout:(59/196): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 364 kB/s | 18 kB 00:00 2024-03-22T12:48:36.438 INFO:teuthology.orchestra.run.smithi092.stdout:(60/196): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 662 kB 00:00 2024-03-22T12:48:36.513 INFO:teuthology.orchestra.run.smithi092.stdout:(61/196): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-22T12:48:36.547 INFO:teuthology.orchestra.run.smithi006.stdout:(127/196): abseil-cpp-20211102.0-3.el9.x86_64.r 941 kB/s | 548 kB 00:00 2024-03-22T12:48:36.555 INFO:teuthology.orchestra.run.smithi092.stdout:(62/196): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-22T12:48:36.605 INFO:teuthology.orchestra.run.smithi092.stdout:(63/196): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-22T12:48:36.647 INFO:teuthology.orchestra.run.smithi092.stdout:(64/196): flexiblas-3.0.4-8.el9.x86_64.rpm 762 kB/s | 32 kB 00:00 2024-03-22T12:48:36.697 INFO:teuthology.orchestra.run.smithi006.stdout:(128/196): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-03-22T12:48:36.789 INFO:teuthology.orchestra.run.smithi092.stdout:(65/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-22T12:48:36.814 INFO:teuthology.orchestra.run.smithi006.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 168 kB/s | 19 kB 00:00 2024-03-22T12:48:36.831 INFO:teuthology.orchestra.run.smithi092.stdout:(66/196): flexiblas-openblas-openmp-3.0.4-8.el9 411 kB/s | 17 kB 00:00 2024-03-22T12:48:36.873 INFO:teuthology.orchestra.run.smithi092.stdout:(67/196): libaio-devel-0.3.111-13.el9.x86_64.rp 241 kB/s | 10 kB 00:00 2024-03-22T12:48:36.923 INFO:teuthology.orchestra.run.smithi092.stdout:(68/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-22T12:48:36.966 INFO:teuthology.orchestra.run.smithi092.stdout:(69/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-22T12:48:37.027 INFO:teuthology.orchestra.run.smithi092.stdout:(70/196): librdkafka-1.6.1-102.el9.x86_64.rpm 10 MB/s | 661 kB 00:00 2024-03-22T12:48:37.095 INFO:teuthology.orchestra.run.smithi092.stdout:(71/196): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-22T12:48:37.128 INFO:teuthology.orchestra.run.smithi092.stdout:(72/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-22T12:48:37.178 INFO:teuthology.orchestra.run.smithi092.stdout:(73/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-22T12:48:37.220 INFO:teuthology.orchestra.run.smithi092.stdout:(74/196): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-22T12:48:37.262 INFO:teuthology.orchestra.run.smithi092.stdout:(75/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-22T12:48:37.305 INFO:teuthology.orchestra.run.smithi092.stdout:(76/196): openblas-0.3.21-2.el9.x86_64.rpm 815 kB/s | 34 kB 00:00 2024-03-22T12:48:37.430 INFO:teuthology.orchestra.run.smithi006.stdout:(130/196): libarrow-9.0.0-10.el9.x86_64.rpm 7.2 MB/s | 4.4 MB 00:00 2024-03-22T12:48:37.471 INFO:teuthology.orchestra.run.smithi092.stdout:(77/196): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-22T12:48:37.539 INFO:teuthology.orchestra.run.smithi092.stdout:(78/196): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-22T12:48:37.547 INFO:teuthology.orchestra.run.smithi006.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 225 kB/s | 26 kB 00:00 2024-03-22T12:48:37.606 INFO:teuthology.orchestra.run.smithi092.stdout:(79/196): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-22T12:48:37.648 INFO:teuthology.orchestra.run.smithi092.stdout:(80/196): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-22T12:48:37.672 INFO:teuthology.orchestra.run.smithi006.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 391 kB/s | 49 kB 00:00 2024-03-22T12:48:37.789 INFO:teuthology.orchestra.run.smithi006.stdout:(133/196): libunwind-1.6.2-1.el9.x86_64.rpm 582 kB/s | 67 kB 00:00 2024-03-22T12:48:37.864 INFO:teuthology.orchestra.run.smithi092.stdout:(81/196): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-22T12:48:37.906 INFO:teuthology.orchestra.run.smithi006.stdout:(134/196): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-22T12:48:37.923 INFO:teuthology.orchestra.run.smithi092.stdout:(82/196): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-22T12:48:37.957 INFO:teuthology.orchestra.run.smithi092.stdout:(83/196): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-22T12:48:38.115 INFO:teuthology.orchestra.run.smithi006.stdout:(135/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-22T12:48:38.140 INFO:teuthology.orchestra.run.smithi092.stdout:(84/196): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 5.8 MB 00:00 2024-03-22T12:48:38.174 INFO:teuthology.orchestra.run.smithi092.stdout:(85/196): python3-jsonpatch-1.21-16.el9.noarch. 732 kB/s | 24 kB 00:00 2024-03-22T12:48:38.208 INFO:teuthology.orchestra.run.smithi092.stdout:(86/196): python3-jsonpointer-2.0-4.el9.noarch. 541 kB/s | 18 kB 00:00 2024-03-22T12:48:38.267 INFO:teuthology.orchestra.run.smithi092.stdout:(87/196): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-22T12:48:38.349 INFO:teuthology.orchestra.run.smithi006.stdout:(136/196): python3-asyncssh-2.13.2-5.el9.noarch 2.3 MB/s | 548 kB 00:00 2024-03-22T12:48:38.457 INFO:teuthology.orchestra.run.smithi006.stdout:(137/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 404 kB/s | 43 kB 00:00 2024-03-22T12:48:38.566 INFO:teuthology.orchestra.run.smithi006.stdout:(138/196): python3-cachetools-4.2.4-1.el9.noarc 304 kB/s | 32 kB 00:00 2024-03-22T12:48:38.675 INFO:teuthology.orchestra.run.smithi006.stdout:(139/196): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2024-03-22T12:48:38.734 INFO:teuthology.orchestra.run.smithi092.stdout:(88/196): librbd1-debuginfo-19.0.0-2145.g3be9a5 4.0 MB/s | 52 MB 00:13 2024-03-22T12:48:38.750 INFO:teuthology.orchestra.run.smithi092.stdout:(89/196): python3-markupsafe-1.1.1-12.el9.x86_6 65 kB/s | 32 kB 00:00 2024-03-22T12:48:38.800 INFO:teuthology.orchestra.run.smithi006.stdout:(140/196): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-22T12:48:38.909 INFO:teuthology.orchestra.run.smithi092.stdout:(90/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-22T12:48:38.934 INFO:teuthology.orchestra.run.smithi092.stdout:(91/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-22T12:48:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:(141/196): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-03-22T12:48:38.968 INFO:teuthology.orchestra.run.smithi092.stdout:(92/196): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-22T12:48:38.993 INFO:teuthology.orchestra.run.smithi092.stdout:(93/196): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-22T12:48:39.010 INFO:teuthology.orchestra.run.smithi092.stdout:(94/196): python3-prettytable-0.7.2-27.el9.noar 990 kB/s | 41 kB 00:00 2024-03-22T12:48:39.044 INFO:teuthology.orchestra.run.smithi092.stdout:(95/196): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-22T12:48:39.069 INFO:teuthology.orchestra.run.smithi092.stdout:(96/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-22T12:48:39.084 INFO:teuthology.orchestra.run.smithi006.stdout:(142/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-22T12:48:39.094 INFO:teuthology.orchestra.run.smithi092.stdout:(97/196): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-22T12:48:39.111 INFO:teuthology.orchestra.run.smithi092.stdout:(98/196): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-22T12:48:39.128 INFO:teuthology.orchestra.run.smithi092.stdout:(99/196): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-22T12:48:39.192 INFO:teuthology.orchestra.run.smithi006.stdout:(143/196): python3-colorama-0.4.4-12.el9.noarch 328 kB/s | 35 kB 00:00 2024-03-22T12:48:39.318 INFO:teuthology.orchestra.run.smithi006.stdout:(144/196): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-22T12:48:39.320 INFO:teuthology.orchestra.run.smithi092.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 202 kB/s | 39 kB 00:00 2024-03-22T12:48:39.638 INFO:teuthology.orchestra.run.smithi092.stdout:(101/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-22T12:48:39.643 INFO:teuthology.orchestra.run.smithi006.stdout:(145/196): python3-grpcio-1.46.7-10.el9.x86_64. 6.3 MB/s | 2.0 MB 00:00 2024-03-22T12:48:39.654 INFO:teuthology.orchestra.run.smithi092.stdout:(102/196): python3-wcwidth-0.2.5-8.el9.noarch.r 124 kB/s | 41 kB 00:00 2024-03-22T12:48:39.688 INFO:teuthology.orchestra.run.smithi092.stdout:(103/196): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-22T12:48:39.705 INFO:teuthology.orchestra.run.smithi092.stdout:(104/196): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-22T12:48:39.722 INFO:teuthology.orchestra.run.smithi092.stdout:(105/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-22T12:48:39.739 INFO:teuthology.orchestra.run.smithi092.stdout:(106/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-22T12:48:39.764 INFO:teuthology.orchestra.run.smithi092.stdout:(107/196): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-22T12:48:39.777 INFO:teuthology.orchestra.run.smithi006.stdout:(146/196): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-22T12:48:39.798 INFO:teuthology.orchestra.run.smithi092.stdout:(108/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-22T12:48:39.832 INFO:teuthology.orchestra.run.smithi092.stdout:(109/196): python3-cryptography-36.0.1-4.el9.x8 17 MB/s | 1.1 MB 00:00 2024-03-22T12:48:39.874 INFO:teuthology.orchestra.run.smithi092.stdout:(110/196): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-22T12:48:39.877 INFO:teuthology.orchestra.run.smithi006.stdout:(147/196): python3-jaraco-8.2.1-3.el9.noarch.rp 108 kB/s | 11 kB 00:00 2024-03-22T12:48:39.899 INFO:teuthology.orchestra.run.smithi092.stdout:(111/196): python3-pycparser-2.20-6.el9.noarch. 1.8 MB/s | 124 kB 00:00 2024-03-22T12:48:39.924 INFO:teuthology.orchestra.run.smithi092.stdout:(112/196): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-22T12:48:39.969 INFO:teuthology.orchestra.run.smithi006.stdout:(148/196): python3-jaraco-classes-3.2.1-5.el9.n 195 kB/s | 18 kB 00:00 2024-03-22T12:48:40.086 INFO:teuthology.orchestra.run.smithi006.stdout:(149/196): python3-jaraco-collections-3.0.0-8.e 200 kB/s | 23 kB 00:00 2024-03-22T12:48:40.178 INFO:teuthology.orchestra.run.smithi006.stdout:(150/196): python3-jaraco-functools-3.5.0-2.el9 214 kB/s | 19 kB 00:00 2024-03-22T12:48:40.270 INFO:teuthology.orchestra.run.smithi006.stdout:(151/196): openssl-devel-3.0.7-27.el9.x86_64.rp 643 kB/s | 4.1 MB 00:06 2024-03-22T12:48:40.283 INFO:teuthology.orchestra.run.smithi092.stdout:(113/196): jq-1.6-16.el9.x86_64.rpm 488 kB/s | 187 kB 00:00 2024-03-22T12:48:40.287 INFO:teuthology.orchestra.run.smithi006.stdout:(152/196): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-03-22T12:48:40.379 INFO:teuthology.orchestra.run.smithi006.stdout:(153/196): python3-logutils-0.3.5-21.el9.noarch 508 kB/s | 46 kB 00:00 2024-03-22T12:48:40.488 INFO:teuthology.orchestra.run.smithi006.stdout:(154/196): python3-more-itertools-8.12.0-2.el9. 732 kB/s | 79 kB 00:00 2024-03-22T12:48:40.517 INFO:teuthology.orchestra.run.smithi092.stdout:(114/196): libquadmath-11.4.1-3.el9.x86_64.rpm 837 kB/s | 195 kB 00:00 2024-03-22T12:48:40.596 INFO:teuthology.orchestra.run.smithi006.stdout:(155/196): python3-natsort-7.1.1-5.el9.noarch.r 537 kB/s | 58 kB 00:00 2024-03-22T12:48:40.634 INFO:teuthology.orchestra.run.smithi092.stdout:(115/196): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-22T12:48:40.659 INFO:teuthology.orchestra.run.smithi092.stdout:(116/196): python3-requests-2.25.1-8.el9.noarch 881 kB/s | 125 kB 00:00 2024-03-22T12:48:40.739 INFO:teuthology.orchestra.run.smithi006.stdout:(156/196): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-22T12:48:40.847 INFO:teuthology.orchestra.run.smithi006.stdout:(157/196): python3-portend-3.1.0-2.el9.noarch.r 153 kB/s | 16 kB 00:00 2024-03-22T12:48:40.934 INFO:teuthology.orchestra.run.smithi092.stdout:(117/196): elfutils-libelf-devel-0.190-2.el9.x8 78 kB/s | 21 kB 00:00 2024-03-22T12:48:40.939 INFO:teuthology.orchestra.run.smithi006.stdout:(158/196): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-03-22T12:48:40.959 INFO:teuthology.orchestra.run.smithi092.stdout:(118/196): libuuid-devel-2.37.4-18.el9.x86_64.r 85 kB/s | 25 kB 00:00 2024-03-22T12:48:40.964 INFO:teuthology.orchestra.run.smithi006.stdout:(159/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 775 kB/s | 90 kB 00:00 2024-03-22T12:48:41.048 INFO:teuthology.orchestra.run.smithi006.stdout:(160/196): python3-pyperclip-1.8.2-2.el9.noarch 262 kB/s | 28 kB 00:00 2024-03-22T12:48:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:(161/196): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-03-22T12:48:41.182 INFO:teuthology.orchestra.run.smithi006.stdout:(162/196): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-03-22T12:48:41.240 INFO:teuthology.orchestra.run.smithi006.stdout:(163/196): python3-rsa-4.9-2.el9.noarch.rpm 356 kB/s | 59 kB 00:00 2024-03-22T12:48:41.291 INFO:teuthology.orchestra.run.smithi006.stdout:(164/196): python3-tempora-5.0.0-2.el9.noarch.r 332 kB/s | 36 kB 00:00 2024-03-22T12:48:41.334 INFO:teuthology.orchestra.run.smithi092.stdout:(119/196): python3-devel-3.9.17-2.el9.x86_64.rp 655 kB/s | 245 kB 00:00 2024-03-22T12:48:41.349 INFO:teuthology.orchestra.run.smithi006.stdout:(165/196): python3-typing-extensions-4.4.0-2.el 473 kB/s | 51 kB 00:00 2024-03-22T12:48:41.425 INFO:teuthology.orchestra.run.smithi006.stdout:(166/196): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-22T12:48:41.467 INFO:teuthology.orchestra.run.smithi006.stdout:(167/196): python3-websocket-client-1.2.3-2.el9 771 kB/s | 90 kB 00:00 2024-03-22T12:48:41.560 INFO:teuthology.orchestra.run.smithi092.stdout:(120/196): python3-jinja2-2.11.3-5.el9.noarch.r 1.1 MB/s | 248 kB 00:00 2024-03-22T12:48:41.567 INFO:teuthology.orchestra.run.smithi006.stdout:(168/196): python3-werkzeug-2.0.3-3.el9.1.noarc 3.0 MB/s | 427 kB 00:00 2024-03-22T12:48:41.592 INFO:teuthology.orchestra.run.smithi006.stdout:(169/196): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2024-03-22T12:48:41.693 INFO:teuthology.orchestra.run.smithi006.stdout:(170/196): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-22T12:48:41.693 INFO:teuthology.orchestra.run.smithi092.stdout:(121/196): qatlib-23.11.0-1.el9.x86_64.rpm 1.6 MB/s | 216 kB 00:00 2024-03-22T12:48:41.802 INFO:teuthology.orchestra.run.smithi092.stdout:(122/196): qatlib-service-23.11.0-1.el9.x86_64. 322 kB/s | 35 kB 00:00 2024-03-22T12:48:41.859 INFO:teuthology.orchestra.run.smithi006.stdout:(171/196): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-03-22T12:48:41.869 INFO:teuthology.orchestra.run.smithi092.stdout:(123/196): openssl-devel-3.0.7-27.el9.x86_64.rp 4.4 MB/s | 4.1 MB 00:00 2024-03-22T12:48:41.903 INFO:teuthology.orchestra.run.smithi092.stdout:(124/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 443 kB/s | 44 kB 00:00 2024-03-22T12:48:41.986 INFO:teuthology.orchestra.run.smithi092.stdout:(125/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 801 kB/s | 94 kB 00:00 2024-03-22T12:48:42.003 INFO:teuthology.orchestra.run.smithi092.stdout:(126/196): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-03-22T12:48:42.318 INFO:teuthology.orchestra.run.smithi006.stdout:(172/196): librados2-19.0.0-2145.g3be9a5c9.el9. 5.4 MB/s | 3.4 MB 00:00 2024-03-22T12:48:42.328 INFO:teuthology.orchestra.run.smithi092.stdout:(127/196): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-22T12:48:42.393 INFO:teuthology.orchestra.run.smithi006.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-22T12:48:42.611 INFO:teuthology.orchestra.run.smithi006.stdout:(174/196): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-03-22T12:48:42.645 INFO:teuthology.orchestra.run.smithi092.stdout:(128/196): abseil-cpp-20211102.0-3.el9.x86_64.r 854 kB/s | 548 kB 00:00 2024-03-22T12:48:42.771 INFO:teuthology.orchestra.run.smithi092.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-03-22T12:48:42.921 INFO:teuthology.orchestra.run.smithi092.stdout:(130/196): gperftools-libs-2.9.1-2.el9.x86_64.r 522 kB/s | 309 kB 00:00 2024-03-22T12:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-03-22T12:48:43.136 INFO:teuthology.orchestra.run.smithi006.stdout:(175/196): NetworkManager-1.47.2-1.el9.x86_64.r 3.1 MB/s | 2.3 MB 00:00 2024-03-22T12:48:43.163 INFO:teuthology.orchestra.run.smithi092.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-03-22T12:48:43.244 INFO:teuthology.orchestra.run.smithi006.stdout:(176/196): NetworkManager-team-1.47.2-1.el9.x86 392 kB/s | 42 kB 00:00 2024-03-22T12:48:43.255 INFO:teuthology.orchestra.run.smithi092.stdout:(133/196): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-03-22T12:48:43.289 INFO:teuthology.orchestra.run.smithi092.stdout:(134/196): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-03-22T12:48:43.355 INFO:teuthology.orchestra.run.smithi006.stdout:(177/196): NetworkManager-tui-1.47.2-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-22T12:48:43.389 INFO:teuthology.orchestra.run.smithi092.stdout:(135/196): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-22T12:48:43.447 INFO:teuthology.orchestra.run.smithi006.stdout:(178/196): acl-2.3.1-4.el9.x86_64.rpm 784 kB/s | 71 kB 00:00 2024-03-22T12:48:43.514 INFO:teuthology.orchestra.run.smithi092.stdout:(136/196): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-22T12:48:43.547 INFO:teuthology.orchestra.run.smithi006.stdout:(179/196): elfutils-debuginfod-client-0.190-2.e 368 kB/s | 36 kB 00:00 2024-03-22T12:48:43.573 INFO:teuthology.orchestra.run.smithi092.stdout:(137/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-03-22T12:48:43.606 INFO:teuthology.orchestra.run.smithi006.stdout:(180/196): NetworkManager-libnm-1.47.2-1.el9.x8 1.8 MB/s | 1.8 MB 00:00 2024-03-22T12:48:43.607 INFO:teuthology.orchestra.run.smithi092.stdout:(138/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-22T12:48:43.665 INFO:teuthology.orchestra.run.smithi006.stdout:(181/196): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-22T12:48:43.690 INFO:teuthology.orchestra.run.smithi092.stdout:(139/196): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-22T12:48:43.716 INFO:teuthology.orchestra.run.smithi092.stdout:(140/196): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-22T12:48:43.724 INFO:teuthology.orchestra.run.smithi006.stdout:(182/196): elfutils-libs-0.190-2.el9.x86_64.rpm 2.2 MB/s | 256 kB 00:00 2024-03-22T12:48:43.748 INFO:teuthology.orchestra.run.smithi006.stdout:(183/196): libacl-2.3.1-4.el9.x86_64.rpm 279 kB/s | 23 kB 00:00 2024-03-22T12:48:43.800 INFO:teuthology.orchestra.run.smithi092.stdout:(141/196): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-22T12:48:43.824 INFO:teuthology.orchestra.run.smithi006.stdout:(184/196): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-22T12:48:43.850 INFO:teuthology.orchestra.run.smithi092.stdout:(142/196): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-22T12:48:43.891 INFO:teuthology.orchestra.run.smithi006.stdout:(185/196): libfdisk-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 155 kB 00:00 2024-03-22T12:48:43.925 INFO:teuthology.orchestra.run.smithi006.stdout:(186/196): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-22T12:48:43.933 INFO:teuthology.orchestra.run.smithi092.stdout:(143/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-22T12:48:43.976 INFO:teuthology.orchestra.run.smithi092.stdout:(144/196): python3-colorama-0.4.4-12.el9.noarch 281 kB/s | 35 kB 00:00 2024-03-22T12:48:43.992 INFO:teuthology.orchestra.run.smithi006.stdout:(187/196): libsmartcols-2.37.4-18.el9.x86_64.rp 643 kB/s | 64 kB 00:00 2024-03-22T12:48:44.017 INFO:teuthology.orchestra.run.smithi006.stdout:(188/196): libuuid-2.37.4-18.el9.x86_64.rpm 315 kB/s | 29 kB 00:00 2024-03-22T12:48:44.067 INFO:teuthology.orchestra.run.smithi092.stdout:(145/196): python3-google-auth-2.28.2-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-22T12:48:44.235 INFO:teuthology.orchestra.run.smithi092.stdout:(146/196): python3-grpcio-tools-1.46.7-10.el9.x 866 kB/s | 144 kB 00:00 2024-03-22T12:48:44.251 INFO:teuthology.orchestra.run.smithi006.stdout:(189/196): openssl-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 1.2 MB 00:00 2024-03-22T12:48:44.293 INFO:teuthology.orchestra.run.smithi092.stdout:(147/196): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-03-22T12:48:44.351 INFO:teuthology.orchestra.run.smithi092.stdout:(148/196): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-03-22T12:48:44.410 INFO:teuthology.orchestra.run.smithi092.stdout:(149/196): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-03-22T12:48:44.461 INFO:teuthology.orchestra.run.smithi092.stdout:(150/196): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-22T12:48:44.519 INFO:teuthology.orchestra.run.smithi092.stdout:(151/196): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-22T12:48:44.553 INFO:teuthology.orchestra.run.smithi092.stdout:(152/196): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-22T12:48:44.568 INFO:teuthology.orchestra.run.smithi006.stdout:(190/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.9 MB/s | 2.2 MB 00:00 2024-03-22T12:48:44.634 INFO:teuthology.orchestra.run.smithi006.stdout:(191/196): util-linux-2.37.4-18.el9.x86_64.rpm 6.0 MB/s | 2.3 MB 00:00 2024-03-22T12:48:44.645 INFO:teuthology.orchestra.run.smithi092.stdout:(153/196): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-03-22T12:48:44.678 INFO:teuthology.orchestra.run.smithi092.stdout:(154/196): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-22T12:48:44.710 INFO:teuthology.orchestra.run.smithi006.stdout:(192/196): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-22T12:48:44.735 INFO:teuthology.orchestra.run.smithi006.stdout:(193/196): util-linux-user-2.37.4-18.el9.x86_64 314 kB/s | 31 kB 00:00 2024-03-22T12:48:44.745 INFO:teuthology.orchestra.run.smithi092.stdout:(155/196): python3-more-itertools-8.12.0-2.el9. 786 kB/s | 79 kB 00:00 2024-03-22T12:48:44.779 INFO:teuthology.orchestra.run.smithi092.stdout:(156/196): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-22T12:48:44.852 INFO:teuthology.orchestra.run.smithi006.stdout:(194/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-22T12:48:44.855 INFO:teuthology.orchestra.run.smithi092.stdout:(157/196): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-22T12:48:44.873 INFO:teuthology.orchestra.run.smithi092.stdout:(158/196): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-03-22T12:48:44.960 INFO:teuthology.orchestra.run.smithi006.stdout:(195/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 331 kB/s | 74 kB 00:00 2024-03-22T12:48:44.963 INFO:teuthology.orchestra.run.smithi092.stdout:(159/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 829 kB/s | 90 kB 00:00 2024-03-22T12:48:44.997 INFO:teuthology.orchestra.run.smithi092.stdout:(160/196): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-03-22T12:48:45.080 INFO:teuthology.orchestra.run.smithi092.stdout:(161/196): python3-repoze-lru-0.7-16.el9.noarch 263 kB/s | 31 kB 00:00 2024-03-22T12:48:45.122 INFO:teuthology.orchestra.run.smithi092.stdout:(162/196): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-22T12:48:45.231 INFO:teuthology.orchestra.run.smithi092.stdout:(163/196): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-03-22T12:48:45.256 INFO:teuthology.orchestra.run.smithi092.stdout:(164/196): python3-tempora-5.0.0-2.el9.noarch.r 267 kB/s | 36 kB 00:00 2024-03-22T12:48:45.357 INFO:teuthology.orchestra.run.smithi092.stdout:(165/196): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-03-22T12:48:45.390 INFO:teuthology.orchestra.run.smithi092.stdout:(166/196): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-22T12:48:45.458 INFO:teuthology.orchestra.run.smithi092.stdout:(167/196): python3-websocket-client-1.2.3-2.el9 893 kB/s | 90 kB 00:00 2024-03-22T12:48:45.508 INFO:teuthology.orchestra.run.smithi092.stdout:(168/196): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-22T12:48:45.541 INFO:teuthology.orchestra.run.smithi092.stdout:(169/196): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-22T12:48:45.608 INFO:teuthology.orchestra.run.smithi092.stdout:(170/196): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-22T12:48:45.700 INFO:teuthology.orchestra.run.smithi092.stdout:(171/196): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-22T12:48:47.092 INFO:teuthology.orchestra.run.smithi092.stdout:(172/196): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 2.3 MB/s | 3.1 MB 00:01 2024-03-22T12:48:47.376 INFO:teuthology.orchestra.run.smithi092.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 1.1 MB/s | 321 kB 00:00 2024-03-22T12:48:48.309 INFO:teuthology.orchestra.run.smithi092.stdout:(174/196): NetworkManager-1.47.2-1.el9.x86_64.r 2.5 MB/s | 2.3 MB 00:00 2024-03-22T12:48:48.693 INFO:teuthology.orchestra.run.smithi092.stdout:(175/196): NetworkManager-libnm-1.47.2-1.el9.x8 4.8 MB/s | 1.8 MB 00:00 2024-03-22T12:48:48.743 INFO:teuthology.orchestra.run.smithi092.stdout:(176/196): librados2-19.0.0-2145.g3be9a5c9.el9. 1.1 MB/s | 3.4 MB 00:03 2024-03-22T12:48:48.768 INFO:teuthology.orchestra.run.smithi092.stdout:(177/196): NetworkManager-team-1.47.2-1.el9.x86 562 kB/s | 42 kB 00:00 2024-03-22T12:48:48.835 INFO:teuthology.orchestra.run.smithi092.stdout:(178/196): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-22T12:48:48.894 INFO:teuthology.orchestra.run.smithi092.stdout:(179/196): elfutils-debuginfod-client-0.190-2.e 625 kB/s | 36 kB 00:00 2024-03-22T12:48:48.961 INFO:teuthology.orchestra.run.smithi092.stdout:(180/196): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-22T12:48:48.986 INFO:teuthology.orchestra.run.smithi092.stdout:(181/196): NetworkManager-tui-1.47.2-1.el9.x86_ 1.0 MB/s | 248 kB 00:00 2024-03-22T12:48:49.028 INFO:teuthology.orchestra.run.smithi092.stdout:(182/196): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-22T12:48:49.056 INFO:teuthology.orchestra.run.smithi092.stdout:(183/196): libacl-2.3.1-4.el9.x86_64.rpm 333 kB/s | 23 kB 00:00 2024-03-22T12:48:49.089 INFO:teuthology.orchestra.run.smithi092.stdout:(184/196): libblkid-2.37.4-18.el9.x86_64.rpm 1.7 MB/s | 108 kB 00:00 2024-03-22T12:48:49.148 INFO:teuthology.orchestra.run.smithi092.stdout:(185/196): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-22T12:48:49.173 INFO:teuthology.orchestra.run.smithi092.stdout:(186/196): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-03-22T12:48:49.215 INFO:teuthology.orchestra.run.smithi092.stdout:(187/196): libsmartcols-2.37.4-18.el9.x86_64.rp 953 kB/s | 64 kB 00:00 2024-03-22T12:48:49.240 INFO:teuthology.orchestra.run.smithi092.stdout:(188/196): libuuid-2.37.4-18.el9.x86_64.rpm 429 kB/s | 29 kB 00:00 2024-03-22T12:48:49.474 INFO:teuthology.orchestra.run.smithi092.stdout:(189/196): openssl-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 1.2 MB 00:00 2024-03-22T12:48:49.799 INFO:teuthology.orchestra.run.smithi092.stdout:(190/196): util-linux-2.37.4-18.el9.x86_64.rpm 7.0 MB/s | 2.3 MB 00:00 2024-03-22T12:48:49.908 INFO:teuthology.orchestra.run.smithi092.stdout:(191/196): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-22T12:48:49.983 INFO:teuthology.orchestra.run.smithi092.stdout:(192/196): util-linux-user-2.37.4-18.el9.x86_64 417 kB/s | 31 kB 00:00 2024-03-22T12:48:50.025 INFO:teuthology.orchestra.run.smithi092.stdout:(193/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.7 MB/s | 2.2 MB 00:00 2024-03-22T12:48:50.209 INFO:teuthology.orchestra.run.smithi092.stdout:(194/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.8 MB/s | 1.1 MB 00:00 2024-03-22T12:48:50.476 INFO:teuthology.orchestra.run.smithi092.stdout:(195/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 165 kB/s | 74 kB 00:00 2024-03-22T12:49:21.170 INFO:teuthology.orchestra.run.smithi092.stdout:(196/196): ceph-test-debuginfo-19.0.0-2145.g3be 10 MB/s | 679 MB 01:07 2024-03-22T12:49:21.175 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-22T12:49:21.175 INFO:teuthology.orchestra.run.smithi092.stdout:Total 14 MB/s | 2.2 GB 02:40 2024-03-22T12:49:21.515 INFO:teuthology.orchestra.run.smithi006.stdout:(196/196): ceph-test-debuginfo-19.0.0-2145.g3be 10 MB/s | 679 MB 01:07 2024-03-22T12:49:21.530 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-22T12:49:21.530 INFO:teuthology.orchestra.run.smithi006.stdout:Total 14 MB/s | 2.2 GB 02:35 2024-03-22T12:49:21.623 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-22T12:49:21.902 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-22T12:49:21.903 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-22T12:49:23.005 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-22T12:49:23.405 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-22T12:49:23.405 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-22T12:49:35.511 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-22T12:49:35.512 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-22T12:49:38.704 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-22T12:49:38.705 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-22T12:49:54.727 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-22T12:49:57.300 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-22T12:49:57.334 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_6 2/220 2024-03-22T12:49:57.368 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-22T12:49:57.399 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-22T12:49:57.407 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-22T12:49:57.448 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-22T12:49:57.501 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-22T12:49:57.535 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-22T12:49:57.561 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-22T12:49:57.603 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-22T12:49:57.926 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-22T12:49:57.987 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/220 2024-03-22T12:49:58.108 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/220 2024-03-22T12:49:58.130 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/220 2024-03-22T12:49:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/220 2024-03-22T12:49:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/220 2024-03-22T12:49:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-22T12:49:58.474 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-22T12:49:58.531 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-22T12:49:58.559 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/220 2024-03-22T12:49:58.662 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/220 2024-03-22T12:49:58.685 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 18/220 2024-03-22T12:49:58.740 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 18/220 2024-03-22T12:49:58.772 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-22T12:49:58.803 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-22T12:49:58.821 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-22T12:49:58.994 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-22T12:49:59.026 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-22T12:49:59.058 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-22T12:49:59.165 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-22T12:49:59.260 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-22T12:49:59.310 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-22T12:49:59.320 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-22T12:49:59.400 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-22T12:49:59.438 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-22T12:49:59.651 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-22T12:49:59.760 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-22T12:49:59.862 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-22T12:49:59.892 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-22T12:49:59.919 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-22T12:49:59.955 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-22T12:50:00.028 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-22T12:50:00.082 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-22T12:50:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-22T12:50:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-22T12:50:00.236 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-22T12:50:00.280 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-22T12:50:00.312 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-22T12:50:00.340 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-22T12:50:00.409 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-22T12:50:00.445 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 40/220 2024-03-22T12:50:00.489 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 41/220 2024-03-22T12:50:00.517 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 42/220 2024-03-22T12:50:00.586 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-22T12:50:00.632 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-22T12:50:00.705 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-22T12:50:00.840 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-22T12:50:00.884 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-22T12:50:01.400 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-22T12:50:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-22T12:50:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-22T12:50:01.615 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-22T12:50:01.622 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-22T12:50:01.655 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-22T12:50:01.696 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-22T12:50:01.733 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 54/220 2024-03-22T12:50:01.763 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-22T12:50:01.996 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-22T12:50:02.191 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-22T12:50:02.240 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-22T12:50:02.242 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-22T12:50:02.284 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-22T12:50:02.302 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_6 2/220 2024-03-22T12:50:02.350 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-22T12:50:02.407 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-22T12:50:02.418 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-22T12:50:02.448 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-22T12:50:02.479 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-22T12:50:02.530 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-22T12:50:02.533 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-22T12:50:02.598 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-22T12:50:02.630 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-22T12:50:02.666 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-22T12:50:02.713 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-22T12:50:02.749 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-22T12:50:02.772 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-22T12:50:02.791 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-22T12:50:02.832 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-22T12:50:02.833 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-22T12:50:02.865 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-22T12:50:02.968 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-22T12:50:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-22T12:50:03.014 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-22T12:50:03.064 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-22T12:50:03.111 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-22T12:50:03.205 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-22T12:50:03.244 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 75/220 2024-03-22T12:50:03.284 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-22T12:50:03.310 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-22T12:50:03.334 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-22T12:50:03.384 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-22T12:50:03.406 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-22T12:50:03.428 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/220 2024-03-22T12:50:03.535 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-22T12:50:03.615 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/220 2024-03-22T12:50:03.634 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-22T12:50:03.676 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-22T12:50:03.725 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-22T12:50:03.763 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/220 2024-03-22T12:50:03.782 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-22T12:50:03.899 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/220 2024-03-22T12:50:03.948 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/220 2024-03-22T12:50:03.965 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-22T12:50:04.013 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-22T12:50:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-22T12:50:04.150 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-22T12:50:04.186 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 88/220 2024-03-22T12:50:04.218 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-22T12:50:04.249 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-22T12:50:04.277 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-22T12:50:04.303 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-22T12:50:04.342 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-22T12:50:04.357 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-22T12:50:04.374 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-22T12:50:04.404 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-22T12:50:04.410 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-22T12:50:04.435 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/220 2024-03-22T12:50:04.479 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-22T12:50:04.514 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-22T12:50:04.547 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-22T12:50:04.588 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-22T12:50:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/220 2024-03-22T12:50:04.615 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-22T12:50:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 18/220 2024-03-22T12:50:04.703 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-22T12:50:04.735 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 18/220 2024-03-22T12:50:04.782 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-22T12:50:04.801 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-22T12:50:04.813 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-22T12:50:04.854 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-22T12:50:04.885 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-22T12:50:04.886 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-22T12:50:04.933 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-22T12:50:05.030 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-22T12:50:05.078 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-22T12:50:05.205 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-22T12:50:05.279 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-22T12:50:05.339 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-22T12:50:05.740 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-22T12:50:05.800 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-22T12:50:05.842 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-22T12:50:05.876 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-22T12:50:05.893 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-22T12:50:05.923 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-22T12:50:05.986 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-22T12:50:06.003 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-22T12:50:06.035 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-22T12:50:06.177 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-22T12:50:06.205 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-22T12:50:06.234 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-22T12:50:06.439 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-22T12:50:06.545 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-22T12:50:06.610 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-22T12:50:06.743 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-22T12:50:06.876 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-22T12:50:06.915 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-22T12:50:06.944 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-22T12:50:07.003 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-22T12:50:07.139 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-22T12:50:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-22T12:50:07.239 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-22T12:50:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-22T12:50:07.354 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-22T12:50:07.374 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-22T12:50:07.422 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-22T12:50:07.487 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-22T12:50:07.535 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-22T12:50:07.566 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-22T12:50:07.687 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-22T12:50:07.729 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 40/220 2024-03-22T12:50:07.793 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 41/220 2024-03-22T12:50:07.827 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 42/220 2024-03-22T12:50:07.953 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-22T12:50:08.012 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-22T12:50:08.113 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-22T12:50:08.290 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-22T12:50:08.350 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-22T12:50:08.557 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-22T12:50:08.640 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-22T12:50:08.672 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-22T12:50:08.695 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-22T12:50:08.713 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-22T12:50:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-22T12:50:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-22T12:50:09.060 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/220 2024-03-22T12:50:09.140 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-22T12:50:09.151 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/220 2024-03-22T12:50:09.181 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 126/220 2024-03-22T12:50:09.217 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-22T12:50:09.253 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-22T12:50:09.313 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-22T12:50:09.340 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-22T12:50:09.423 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-22T12:50:09.506 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-22T12:50:09.571 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 54/220 2024-03-22T12:50:09.614 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-22T12:50:09.681 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-22T12:50:09.713 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/220 2024-03-22T12:50:09.792 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/220 2024-03-22T12:50:09.815 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/220 2024-03-22T12:50:10.039 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-22T12:50:10.237 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-22T12:50:10.334 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-22T12:50:10.400 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-22T12:50:10.617 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-22T12:50:10.664 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-22T12:50:10.740 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-22T12:50:10.837 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-22T12:50:10.880 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-22T12:50:10.922 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-22T12:50:10.992 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-22T12:50:11.037 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-22T12:50:11.121 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-22T12:50:11.198 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-22T12:50:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-22T12:50:11.459 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-22T12:50:11.498 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-22T12:50:11.558 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-22T12:50:11.702 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-22T12:50:11.737 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 75/220 2024-03-22T12:50:11.774 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-22T12:50:11.811 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-22T12:50:11.852 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-22T12:50:12.017 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-22T12:50:12.053 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/220 2024-03-22T12:50:12.086 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/220 2024-03-22T12:50:12.201 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-22T12:50:12.350 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-22T12:50:12.396 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-22T12:50:12.460 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-22T12:50:12.535 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-22T12:50:12.622 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/220 2024-03-22T12:50:12.664 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/220 2024-03-22T12:50:12.745 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/220 2024-03-22T12:50:12.745 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T12:50:12.745 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T12:50:12.746 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:12.774 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 133/220 2024-03-22T12:50:12.787 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-22T12:50:12.851 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-22T12:50:13.021 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-22T12:50:13.063 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 88/220 2024-03-22T12:50:13.103 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-22T12:50:13.133 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-22T12:50:13.173 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-22T12:50:13.218 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-22T12:50:13.276 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-22T12:50:13.330 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-22T12:50:13.405 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-22T12:50:13.552 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-22T12:50:13.592 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-22T12:50:13.628 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-22T12:50:13.693 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-22T12:50:13.740 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-22T12:50:13.890 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-22T12:50:14.002 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-22T12:50:14.052 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-22T12:50:14.088 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-22T12:50:14.136 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-22T12:50:14.277 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-22T12:50:14.345 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-22T12:50:15.353 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-22T12:50:15.411 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-22T12:50:15.445 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-22T12:50:15.503 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-22T12:50:15.608 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-22T12:50:15.653 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-22T12:50:15.926 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-22T12:50:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-22T12:50:16.059 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-22T12:50:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-22T12:50:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-22T12:50:18.726 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-22T12:50:18.813 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-22T12:50:18.850 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-22T12:50:18.881 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-22T12:50:18.922 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-22T12:50:19.320 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-22T12:50:19.361 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/220 2024-03-22T12:50:19.484 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/220 2024-03-22T12:50:19.518 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 126/220 2024-03-22T12:50:19.575 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-22T12:50:20.126 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-22T12:50:20.164 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/220 2024-03-22T12:50:20.264 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/220 2024-03-22T12:50:20.333 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/220 2024-03-22T12:50:22.977 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/220 2024-03-22T12:50:23.013 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/220 2024-03-22T12:50:23.676 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/220 2024-03-22T12:50:23.724 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/220 2024-03-22T12:50:23.897 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/220 2024-03-22T12:50:23.897 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T12:50:23.897 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T12:50:23.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:23.958 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 133/220 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 133/220 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2024-03-22T12:50:25.166 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2024-03-22T12:50:25.167 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 134/220 2024-03-22T12:50:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 134/220 2024-03-22T12:50:25.340 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:25.340 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T12:50:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:26.820 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/220 2024-03-22T12:50:26.846 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/220 2024-03-22T12:50:26.846 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:26.846 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T12:50:26.846 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T12:50:26.846 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T12:50:26.847 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:27.072 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/220 2024-03-22T12:50:27.094 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/220 2024-03-22T12:50:27.094 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:27.094 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T12:50:27.094 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T12:50:27.094 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T12:50:27.094 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:27.539 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 137/220 2024-03-22T12:50:27.561 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 137/220 2024-03-22T12:50:27.561 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:27.561 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T12:50:27.562 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T12:50:27.562 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T12:50:27.562 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:27.657 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-22T12:50:28.085 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 139/220 2024-03-22T12:50:28.116 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 139/220 2024-03-22T12:50:28.148 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 140/220 2024-03-22T12:50:28.208 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 141/220 2024-03-22T12:50:28.239 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 142/220 2024-03-22T12:50:28.471 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 142/220 2024-03-22T12:50:28.498 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 143/220 2024-03-22T12:50:29.459 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 143/220 2024-03-22T12:50:29.490 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 144/220 2024-03-22T12:50:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 144/220 2024-03-22T12:50:29.753 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 145/220 2024-03-22T12:50:29.784 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 146/220 2024-03-22T12:50:29.806 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 146/220 2024-03-22T12:50:29.806 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:29.806 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T12:50:29.806 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T12:50:29.806 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T12:50:29.806 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:29.851 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 147/220 2024-03-22T12:50:29.861 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 147/220 2024-03-22T12:50:29.907 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 148/220 2024-03-22T12:50:30.781 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-22T12:50:30.811 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 150/220 2024-03-22T12:50:30.834 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 150/220 2024-03-22T12:50:30.834 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:30.834 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T12:50:30.834 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T12:50:30.834 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T12:50:30.834 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:30.879 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 151/220 2024-03-22T12:50:30.900 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 151/220 2024-03-22T12:50:30.900 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:30.900 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T12:50:30.900 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:30.946 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/220 2024-03-22T12:50:30.968 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/220 2024-03-22T12:50:30.968 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:30.968 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T12:50:30.968 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T12:50:30.969 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T12:50:30.969 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:31.272 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/220 2024-03-22T12:50:31.293 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/220 2024-03-22T12:50:31.294 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:31.294 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T12:50:31.294 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T12:50:31.294 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T12:50:31.294 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:50:35.734 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/220 2024-03-22T12:50:35.775 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/220 2024-03-22T12:50:35.814 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 156/220 2024-03-22T12:50:35.841 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-22T12:50:36.019 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 158/220 2024-03-22T12:50:36.132 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-22T12:50:36.172 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 160/220 2024-03-22T12:50:36.211 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-22T12:50:36.264 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-22T12:50:36.298 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-22T12:50:36.323 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 164/220 2024-03-22T12:50:36.355 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-22T12:50:36.385 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 166/220 2024-03-22T12:50:36.449 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 167/220 2024-03-22T12:50:36.508 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-22T12:50:36.550 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-22T12:50:37.154 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-22T12:50:37.198 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-22T12:50:37.217 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-22T12:50:37.253 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 133/220 2024-03-22T12:50:37.253 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-03-22T12:50:37.253 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-03-22T12:50:37.253 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-03-22T12:50:37.253 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-03-22T12:50:37.253 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-03-22T12:50:37.253 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-03-22T12:50:37.254 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-03-22T12:50:37.254 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:37.488 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 134/220 2024-03-22T12:50:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 134/220 2024-03-22T12:50:37.541 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:37.541 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T12:50:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:39.226 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/220 2024-03-22T12:50:39.313 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/220 2024-03-22T12:50:39.313 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:39.313 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T12:50:39.313 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T12:50:39.314 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T12:50:39.314 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:39.609 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/220 2024-03-22T12:50:39.710 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/220 2024-03-22T12:50:39.710 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:39.710 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T12:50:39.710 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T12:50:39.710 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T12:50:39.710 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 137/220 2024-03-22T12:50:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 137/220 2024-03-22T12:50:40.329 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:40.330 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T12:50:40.330 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T12:50:40.330 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T12:50:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:40.462 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-22T12:50:41.073 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 139/220 2024-03-22T12:50:41.104 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 139/220 2024-03-22T12:50:41.148 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 140/220 2024-03-22T12:50:41.233 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 141/220 2024-03-22T12:50:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 142/220 2024-03-22T12:50:41.635 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 142/220 2024-03-22T12:50:41.676 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 143/220 2024-03-22T12:50:42.858 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 143/220 2024-03-22T12:50:42.893 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 144/220 2024-03-22T12:50:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 144/220 2024-03-22T12:50:43.231 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 145/220 2024-03-22T12:50:43.261 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 146/220 2024-03-22T12:50:43.338 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 146/220 2024-03-22T12:50:43.338 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:43.338 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T12:50:43.338 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T12:50:43.338 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T12:50:43.339 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:43.387 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 147/220 2024-03-22T12:50:43.428 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 147/220 2024-03-22T12:50:43.517 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 148/220 2024-03-22T12:50:44.360 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 171/220 2024-03-22T12:50:44.545 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-22T12:50:44.581 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 150/220 2024-03-22T12:50:44.657 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 150/220 2024-03-22T12:50:44.657 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:44.657 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T12:50:44.657 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T12:50:44.658 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T12:50:44.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:44.706 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 151/220 2024-03-22T12:50:44.794 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 151/220 2024-03-22T12:50:44.794 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:44.794 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T12:50:44.794 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:44.874 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/220 2024-03-22T12:50:44.958 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/220 2024-03-22T12:50:44.958 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:44.958 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T12:50:44.958 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T12:50:44.958 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T12:50:44.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:45.349 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/220 2024-03-22T12:50:45.426 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/220 2024-03-22T12:50:45.426 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T12:50:45.426 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T12:50:45.426 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T12:50:45.426 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T12:50:45.426 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:50:50.566 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/220 2024-03-22T12:50:50.622 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/220 2024-03-22T12:50:50.664 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 156/220 2024-03-22T12:50:50.703 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-22T12:50:51.011 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 158/220 2024-03-22T12:50:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-22T12:50:51.232 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 160/220 2024-03-22T12:50:51.293 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-22T12:50:51.363 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-22T12:50:51.417 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-22T12:50:51.452 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 164/220 2024-03-22T12:50:51.497 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-22T12:50:51.549 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 166/220 2024-03-22T12:50:51.636 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 167/220 2024-03-22T12:50:51.772 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-22T12:50:51.813 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-22T12:50:52.757 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-22T12:50:52.819 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-22T12:50:52.881 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-22T12:51:01.037 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 171/220 2024-03-22T12:51:26.172 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 172/220 2024-03-22T12:51:26.819 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2145.g3be9a5c9.el 173/220 2024-03-22T12:51:26.983 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 174/220 2024-03-22T12:51:31.829 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 175/220 2024-03-22T12:51:34.118 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 176/220 2024-03-22T12:51:43.845 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 177/220 2024-03-22T12:51:48.533 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 172/220 2024-03-22T12:51:49.248 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2145.g3be9a5c9.el 173/220 2024-03-22T12:51:49.461 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 174/220 2024-03-22T12:51:54.885 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 175/220 2024-03-22T12:51:57.237 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 176/220 2024-03-22T12:52:08.170 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 177/220 2024-03-22T12:52:11.732 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 178/220 2024-03-22T12:52:28.974 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 179/220 2024-03-22T12:52:39.993 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 178/220 2024-03-22T12:52:59.693 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 179/220 2024-03-22T12:53:36.482 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2145.g3be9a5c9.e 180/220 2024-03-22T12:53:37.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 181/220 2024-03-22T12:53:37.167 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 182/220 2024-03-22T12:53:41.968 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2145.g3be9a5c9 183/220 2024-03-22T12:53:47.711 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 184/220 2024-03-22T12:53:48.031 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librbd1-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 185/220 2024-03-22T12:53:48.354 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2145.g3be9a5c9 186/220 2024-03-22T12:53:48.707 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2145.g3be9a5c9. 187/220 2024-03-22T12:53:48.830 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el 188/220 2024-03-22T12:53:48.919 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el 189/220 2024-03-22T12:53:54.466 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 190/220 2024-03-22T12:53:54.861 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 191/220 2024-03-22T12:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 192/220 2024-03-22T12:53:56.312 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-22T12:53:56.371 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-22T12:53:56.461 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-22T12:53:56.498 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-22T12:53:56.532 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-22T12:53:56.557 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-22T12:53:56.584 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-22T12:53:56.584 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-22T12:53:56.663 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-22T12:53:56.693 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-22T12:53:56.693 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-22T12:53:56.761 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-22T12:53:56.787 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-22T12:53:56.829 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-22T12:53:56.854 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-22T12:53:56.879 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-22T12:53:56.912 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-22T12:53:56.951 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-22T12:53:56.984 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-22T12:53:57.009 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-22T12:53:57.034 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-22T12:53:57.059 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-22T12:53:57.085 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-22T12:53:57.091 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-22T12:53:57.091 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-22T12:53:57.271 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-22T12:53:57.271 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-22T12:53:57.354 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-22T12:53:57.379 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-22T12:53:57.410 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-22T12:53:57.438 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-22T12:53:57.471 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-22T12:53:57.536 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 3/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2145.g3be9a5c9.el 5/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_6 6/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 8/220 2024-03-22T12:54:06.867 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 9/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 10/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 12/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 14/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 15/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 16/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 18/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 19/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2145.g3be9a5c9.e 20/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 21/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 22/220 2024-03-22T12:54:06.868 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 23/220 2024-03-22T12:54:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/220 2024-03-22T12:54:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/220 2024-03-22T12:54:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 26/220 2024-03-22T12:54:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/220 2024-03-22T12:54:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 28/220 2024-03-22T12:54:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/220 2024-03-22T12:54:06.869 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 30/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2145.g3be9a5c9 31/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 32/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 33/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 34/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 35/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 36/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 37/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 38/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2145.g3be9a5c9 39/220 2024-03-22T12:54:06.886 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 40/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2145.g3be9a5c9. 41/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 42/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el 43/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 44/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el 45/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 46/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 47/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 48/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 49/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 50/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 51/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 52/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 53/220 2024-03-22T12:54:06.887 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 54/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 55/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 56/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 57/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 58/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 59/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 60/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 61/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-22T12:54:06.888 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-22T12:54:06.889 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-22T12:54:06.890 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-22T12:54:06.891 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-22T12:54:06.892 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 138/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-22T12:54:06.893 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 146/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-22T12:54:06.894 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-22T12:54:06.895 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 173/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 175/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 179/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 181/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 183/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 185/220 2024-03-22T12:54:06.896 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-22T12:54:06.897 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-22T12:54:06.898 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-22T12:54:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-22T12:54:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-22T12:54:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-22T12:54:06.899 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T12:54:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T12:54:09.920 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.921 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.922 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.923 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-16.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T12:54:09.924 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.925 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T12:54:09.926 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.927 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T12:54:09.928 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T12:54:09.929 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T12:54:09.930 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T12:54:09.931 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.932 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-22T12:54:10.608 DEBUG:teuthology.parallel:result is None 2024-03-22T12:54:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2145.g3be9a5c9.e 180/220 2024-03-22T12:54:17.689 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 181/220 2024-03-22T12:54:17.759 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 182/220 2024-03-22T12:54:23.399 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2145.g3be9a5c9 183/220 2024-03-22T12:54:31.342 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 184/220 2024-03-22T12:54:32.341 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librbd1-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 185/220 2024-03-22T12:54:32.466 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2145.g3be9a5c9 186/220 2024-03-22T12:54:32.621 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2145.g3be9a5c9. 187/220 2024-03-22T12:54:32.691 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el 188/220 2024-03-22T12:54:32.794 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el 189/220 2024-03-22T12:54:39.151 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 190/220 2024-03-22T12:54:39.325 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 191/220 2024-03-22T12:54:40.689 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 192/220 2024-03-22T12:54:40.811 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-22T12:54:40.873 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-22T12:54:41.010 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-22T12:54:41.050 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-22T12:54:41.083 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-22T12:54:41.117 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-22T12:54:41.167 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-22T12:54:41.167 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-22T12:54:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-22T12:54:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-22T12:54:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-22T12:54:41.518 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-22T12:54:41.583 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-22T12:54:41.626 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-22T12:54:41.666 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-22T12:54:41.699 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-22T12:54:41.785 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-22T12:54:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-22T12:54:41.941 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-22T12:54:41.978 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-22T12:54:42.012 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-22T12:54:42.045 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-22T12:54:42.078 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-22T12:54:42.116 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-22T12:54:42.116 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-22T12:54:44.364 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-22T12:54:44.364 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-22T12:54:44.510 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-22T12:54:44.549 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-22T12:54:44.604 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-22T12:54:44.643 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-22T12:54:44.692 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-22T12:54:44.818 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-22T12:54:45.078 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-22T12:54:45.078 INFO:teuthology.orchestra.run.smithi006.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-22T12:54:45.078 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/220 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/220 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 3/220 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/220 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2145.g3be9a5c9.el 5/220 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_6 6/220 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/220 2024-03-22T12:54:52.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 8/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 9/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 10/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 12/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 14/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 15/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 16/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 18/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 19/220 2024-03-22T12:54:52.077 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2145.g3be9a5c9.e 20/220 2024-03-22T12:54:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 21/220 2024-03-22T12:54:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 22/220 2024-03-22T12:54:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 23/220 2024-03-22T12:54:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/220 2024-03-22T12:54:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/220 2024-03-22T12:54:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 26/220 2024-03-22T12:54:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 28/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 30/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2145.g3be9a5c9 31/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9. 32/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 33/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 34/220 2024-03-22T12:54:52.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 35/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 36/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 37/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 38/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2145.g3be9a5c9 39/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 40/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2145.g3be9a5c9. 41/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 42/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el 43/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 44/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el 45/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 46/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x 47/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 48/220 2024-03-22T12:54:52.097 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2145.g3be9a5c9.el9 49/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 50/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x8 51/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 52/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 53/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 54/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 55/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 56/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 57/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 58/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 59/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 60/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 61/220 2024-03-22T12:54:52.098 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-22T12:54:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-22T12:54:52.100 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-22T12:54:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-22T12:54:52.102 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-22T12:54:52.103 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 138/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-22T12:54:52.104 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 146/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-22T12:54:52.105 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 173/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-22T12:54:52.106 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 175/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 179/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 181/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 183/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 185/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-22T12:54:52.107 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-22T12:54:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-22T12:54:52.109 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-22T12:54:52.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-22T12:54:52.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-22T12:54:52.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-22T12:54:52.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-22T12:54:52.110 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-22T12:54:55.120 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-22T12:54:55.120 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:54:55.120 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-22T12:54:55.120 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:55.120 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:55.120 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.121 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T12:54:55.122 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.123 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T12:54:55.125 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.126 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T12:54:55.127 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T12:54:55.128 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T12:54:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T12:54:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T12:54:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T12:54:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T12:54:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T12:54:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T12:54:55.129 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T12:54:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.130 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T12:54:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T12:54:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T12:54:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T12:54:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T12:54:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T12:54:55.131 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T12:54:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T12:54:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T12:54:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T12:54:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T12:54:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T12:54:55.132 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T12:54:55.133 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T12:54:55.134 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.135 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T12:54:55.136 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T12:54:55.137 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T12:54:55.137 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T12:54:55.137 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T12:54:55.137 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T12:54:55.137 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.138 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.138 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.138 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.138 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.138 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-debuginfo-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-22T12:54:55.138 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T12:54:55.139 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T12:54:55.139 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T12:54:55.139 INFO:teuthology.orchestra.run.smithi006.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T12:54:55.139 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T12:54:55.139 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T12:54:55.140 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T12:54:55.140 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:54:55.140 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-22T12:54:56.571 DEBUG:teuthology.parallel:result is None 2024-03-22T12:54:56.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-22T12:54:56.792 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T12:54:56.849 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-22T12:54:56.850 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-22T12:54:56.850 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-22T12:54:56.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-22T12:54:56.991 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T12:54:57.016 INFO:teuthology.orchestra.run.smithi092.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-22T12:54:57.016 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-22T12:54:57.016 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-22T12:54:57.018 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-22T12:54:57.018 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:54:57.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T12:54:57.121 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:54:57.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T12:54:57.153 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-22T12:54:57.153 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:54:57.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T12:54:57.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T12:54:57.349 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:54:57.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T12:54:57.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T12:54:57.447 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-22T12:54:57.448 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:54:57.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T12:54:57.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T12:54:57.632 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:54:57.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T12:54:57.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T12:54:57.729 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-22T12:54:57.730 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:54:57.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T12:54:57.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T12:54:57.915 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:54:57.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T12:54:57.944 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T12:54:58.013 INFO:teuthology.run_tasks:Running task ceph... 2024-03-22T12:54:58.097 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-22T12:54:58.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-03-22T12:54:58.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2024-03-22T12:54:58.204 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-22T12:54:58.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-22T12:54:58.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-22T12:54:58.307 INFO:tasks.ceph:Creating extra log directories... 2024-03-22T12:54:58.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-22T12:54:58.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-22T12:54:58.413 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-22T12:54:58.413 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-22T12:54:58.413 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-22T12:54:58.413 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:54:58.413 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-22T12:54:58.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-22T12:54:58.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-22T12:54:58.851 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:54:58.851 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-22T12:54:58.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-22T12:54:58.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-22T12:54:59.025 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-22T12:54:59.025 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-22T12:54:59.025 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325', 'branch': 'wip-mchangir-testing-main-20240318.032620', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7616325', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-testing-main-20240318.032620'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'client.0': {'valgrind': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-testing-main-20240318.032620', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'owner': 'mchangir', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3550, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'sleep_before_teardown': 0, 'subset': '613/1024', 'suite': 'fs', 'suite_branch': 'wip-mchangir-testing-main-20240318.032620', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6BhfP/x4l8nDPTZnbV3xnODLJoELpLhMe8jZfm+FBBZzddHhUHg7GKzQLOknR/zo4xfve+LI+p66Lb2RTroU0=', 'smithi092.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4IWQLXvvgXJLm0v+K1UZa29kuyXyIouGFDfsGfDWmjnNHWLCxxQMQGuVH9JaD0cFOC6fhXVNlYmFKQo0WSwsQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}, 'cluster': 'ceph'}}, {'kclient': None}, {'workunit': {'clients': {'all': ['suites/dbench.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-22_09:49:57', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-22T12:54:59.026 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-22T12:54:59.033 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-22T12:54:59.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-03-22T12:54:59.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2024-03-22T12:54:59.190 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:54:59.191 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T12:54:59.246 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T12:54:59.247 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-22 12:54:45.523654674 +0000 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-22 12:45:44.653340929 +0000 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-22 12:45:44.653340929 +0000 2024-03-22T12:54:59.297 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-22 12:45:44.653340929 +0000 2024-03-22T12:54:59.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T12:54:59.384 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-22T12:54:59.384 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-22T12:54:59.384 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000360502 s, 1.4 MB/s 2024-03-22T12:54:59.389 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T12:54:59.447 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-22 12:54:45.524654663 +0000 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-22 12:45:45.205334660 +0000 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-22 12:45:45.205334660 +0000 2024-03-22T12:54:59.500 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-22 12:45:45.205334660 +0000 2024-03-22T12:54:59.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T12:54:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-22T12:54:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-22T12:54:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000308994 s, 1.7 MB/s 2024-03-22T12:54:59.598 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T12:54:59.657 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-22 12:54:45.525654652 +0000 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-22 12:45:45.786328062 +0000 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-22 12:45:45.786328062 +0000 2024-03-22T12:54:59.709 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-22 12:45:45.786328062 +0000 2024-03-22T12:54:59.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T12:54:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-22T12:54:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-22T12:54:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000336705 s, 1.5 MB/s 2024-03-22T12:54:59.812 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T12:54:59.873 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-03-22T12:54:59.926 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-22 12:54:45.525654652 +0000 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-22 12:45:46.370321429 +0000 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-22 12:45:46.370321429 +0000 2024-03-22T12:54:59.927 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-22 12:45:46.370321429 +0000 2024-03-22T12:54:59.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T12:55:00.021 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-22T12:55:00.021 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-22T12:55:00.021 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000373298 s, 1.4 MB/s 2024-03-22T12:55:00.026 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T12:55:00.085 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:00.085 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:00.085 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T12:55:00.102 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T12:55:00.102 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-22 12:53:57.869870116 +0000 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-22 12:44:22.270477888 +0000 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-22 12:44:22.270477888 +0000 2024-03-22T12:55:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-22 12:44:22.270477888 +0000 2024-03-22T12:55:00.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T12:55:00.229 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-22T12:55:00.229 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-22T12:55:00.229 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000301178 s, 1.7 MB/s 2024-03-22T12:55:00.230 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T12:55:00.287 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-03-22T12:55:00.344 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T12:55:00.344 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:55:00.345 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-22T12:55:00.345 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:55:00.345 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:55:00.345 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-22 12:53:57.869870116 +0000 2024-03-22T12:55:00.345 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-22 12:44:22.791462560 +0000 2024-03-22T12:55:00.345 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-22 12:44:22.791462560 +0000 2024-03-22T12:55:00.345 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-22 12:44:22.791462560 +0000 2024-03-22T12:55:00.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T12:55:00.412 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-22T12:55:00.412 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-22T12:55:00.412 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000235874 s, 2.2 MB/s 2024-03-22T12:55:00.413 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T12:55:00.470 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-03-22T12:55:00.526 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T12:55:00.526 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:55:00.526 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-22T12:55:00.527 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:55:00.527 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:55:00.527 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-22 12:53:57.869870116 +0000 2024-03-22T12:55:00.527 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-22 12:44:23.138452351 +0000 2024-03-22T12:55:00.527 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-22 12:44:23.138452351 +0000 2024-03-22T12:55:00.527 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-22 12:44:23.138452351 +0000 2024-03-22T12:55:00.527 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T12:55:00.593 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-22T12:55:00.593 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-22T12:55:00.593 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000166793 s, 3.1 MB/s 2024-03-22T12:55:00.594 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T12:55:00.649 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-22 12:53:57.870870087 +0000 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-22 12:44:23.478442348 +0000 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-22 12:44:23.478442348 +0000 2024-03-22T12:55:00.705 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-22 12:44:23.478442348 +0000 2024-03-22T12:55:00.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T12:55:00.770 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-22T12:55:00.770 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-22T12:55:00.770 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000199732 s, 2.6 MB/s 2024-03-22T12:55:00.771 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T12:55:00.827 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:00.827 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-22T12:55:00.828 INFO:tasks.ceph:Generating config... 2024-03-22T12:55:00.829 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-22T12:55:00.829 INFO:tasks.ceph:[client] debug client = 20 2024-03-22T12:55:00.829 INFO:tasks.ceph:[client] debug ms = 1 2024-03-22T12:55:00.829 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-22T12:55:00.829 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-22T12:55:00.829 INFO:tasks.ceph:[global] mds beacon grace = 60 2024-03-22T12:55:00.829 INFO:tasks.ceph:[global] mds heartbeat grace = 60 2024-03-22T12:55:00.829 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-22T12:55:00.829 INFO:tasks.ceph:[global] osd heartbeat grace = 60 2024-03-22T12:55:00.829 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-22T12:55:00.829 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] mds valgrind exit = True 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-22T12:55:00.830 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-22T12:55:00.831 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-22T12:55:00.831 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-22T12:55:00.832 INFO:tasks.ceph:Setting up mon.a... 2024-03-22T12:55:00.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-22T12:55:00.991 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-03-22T12:55:01.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-22T12:55:01.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-22T12:55:01.243 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.b', '172.21.15.92'), ('mon.c', '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]')] 2024-03-22T12:55:01.244 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]', 'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-22T12:55:01.244 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:01.245 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-22T12:55:01.302 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --add b 172.21.15.92 --addv c '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 1722e499-f2bc-49ac-a1c9-7ab0d7863db9 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 1722e499-f2bc-49ac-a1c9-7ab0d7863db9 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-03-22T12:55:01.450251+0000 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-03-22T12:55:01.450251+0000 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-03-22T12:55:01.449 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-22T12:55:01.452 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-22T12:55:01.504 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1722e499-f2bc-49ac-a1c9-7ab0d7863db9... 2024-03-22T12:55:01.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-22T12:55:01.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-22T12:55:01.565 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.566 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2024-03-22T12:55:01.567 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: mon_allow_pool_size_one = true 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2024-03-22T12:55:01.568 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.6,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: mds beacon grace = 60 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: mds heartbeat grace = 60 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on skipped message = False 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: osd heartbeat grace = 60 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = 1722e499-f2bc-49ac-a1c9-7ab0d7863db9 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2024-03-22T12:55:01.569 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock profile = high_recovery_ops 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug trim objects = true 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2024-03-22T12:55:01.570 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore block size = 96636764160 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore compression mode = aggressive 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore fsck on mount = True 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluefs = 20 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluestore = 20 2024-03-22T12:55:01.571 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: debug rocksdb = 10 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd backfillfull_ratio = 0.85 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd full ratio = 0.9 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd nearfull ratio = 0.8 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: osd failsafe full ratio = 0.95 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: osd fast shutdown = False 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = bluestore 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-03-22T12:55:01.572 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2024-03-22T12:55:01.573 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: # 1m isn't quite enough 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: mon_down_mkfs_grace = 2m 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_filestore_osds = false 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: mon op complaint time = 120 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd crush smoke test = False 2024-03-22T12:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: client mount timeout = 600 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2024-03-22T12:55:01.575 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.e] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.f] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout:[mds] 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds = 20 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds balancer = 20 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug frag = True 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug scatterstat = True 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: mds op complaint time = 180 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: mds valgrind exit = True 2024-03-22T12:55:01.576 INFO:teuthology.orchestra.run.smithi092.stdout: mds verify scatter = True 2024-03-22T12:55:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2024-03-22T12:55:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2024-03-22T12:55:01.577 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2024-03-22T12:55:01.684 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-22T12:55:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-22T12:55:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-22T12:55:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-22T12:55:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.684 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.685 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-22T12:55:01.686 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: mds beacon grace = 60 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: mds heartbeat grace = 60 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: osd heartbeat grace = 60 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 1722e499-f2bc-49ac-a1c9-7ab0d7863db9 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-22T12:55:01.687 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.688 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-22T12:55:01.689 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: osd fast shutdown = False 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-22T12:55:01.690 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-22T12:55:01.691 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-22T12:55:01.692 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd crush smoke test = False 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-22T12:55:01.693 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.e] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.f] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-22T12:55:01.694 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-22T12:55:01.695 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-22T12:55:01.695 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-22T12:55:01.695 INFO:teuthology.orchestra.run.smithi006.stdout: mds valgrind exit = True 2024-03-22T12:55:01.695 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-22T12:55:01.695 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-22T12:55:01.695 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-22T12:55:01.695 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-22T12:55:01.734 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-22T12:55:01.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-22T12:55:01.867 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-22T12:55:01.867 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:01.867 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-22T12:55:01.922 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:01.922 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-22T12:55:01.974 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T12:55:01.975 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:01.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-22T12:55:01.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-22T12:55:02.134 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:02.135 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:02.234 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T12:55:02.234 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:02.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-22T12:55:02.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-22T12:55:02.278 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:02.278 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:02.334 INFO:tasks.ceph:Setting up mon nodes... 2024-03-22T12:55:02.334 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-22T12:55:02.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-22T12:55:02.512 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-22T12:55:02.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-22T12:55:02.596 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-22T12:55:02.599 INFO:tasks.ceph:Setting up mds nodes... 2024-03-22T12:55:02.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-22T12:55:02.791 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-22T12:55:02.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-22T12:55:02.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-22T12:55:03.099 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-22T12:55:03.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-22T12:55:03.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-22T12:55:03.395 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-22T12:55:03.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-22T12:55:03.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-22T12:55:03.563 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-22T12:55:03.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-22T12:55:03.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-22T12:55:03.687 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-22T12:55:03.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-22T12:55:03.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-22T12:55:03.809 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-22T12:55:03.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-22T12:55:03.838 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-22T12:55:03.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-22T12:55:03.978 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-22T12:55:04.035 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-22T12:55:04.035 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-22T12:55:04.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-22T12:55:04.130 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:04.130 INFO:tasks.ceph:role: osd.0 2024-03-22T12:55:04.130 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T12:55:04.130 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:04.231 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:04.726 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-22T12:55:04.733 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-22T12:55:04.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-22T12:55:04.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-22T12:55:05.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-22T12:55:05.107 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:05.107 INFO:tasks.ceph:role: osd.1 2024-03-22T12:55:05.107 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T12:55:05.107 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:05.207 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:05.674 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-22T12:55:05.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-22T12:55:05.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-22T12:55:05.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-22T12:55:05.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-22T12:55:05.978 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:05.978 INFO:tasks.ceph:role: osd.2 2024-03-22T12:55:05.978 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T12:55:05.978 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-22T12:55:06.071 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:06.071 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:06.071 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:06.071 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:06.071 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:06.072 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:06.072 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:06.072 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:06.072 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:06.072 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:06.597 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-22T12:55:06.605 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-22T12:55:06.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-22T12:55:06.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-22T12:55:06.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-22T12:55:06.981 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:06.981 INFO:tasks.ceph:role: osd.3 2024-03-22T12:55:06.981 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T12:55:06.981 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-22T12:55:07.084 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:07.085 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:07.539 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-22T12:55:07.546 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-22T12:55:07.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-22T12:55:07.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-22T12:55:07.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:07.976+0000 7eff7be0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-22T12:55:07.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:07.982+0000 7eff7be0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-22T12:55:07.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:07.982+0000 7eff7be0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-22T12:55:07.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:07.982+0000 7eff7be0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-22T12:55:07.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:07.982+0000 7eff7be0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-22T12:55:07.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:07.992+0000 7eff7be0c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-22T12:55:11.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T12:55:11.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:11.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:11.430+0000 7f57d67df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-22T12:55:11.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:11.430+0000 7f57d67df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-22T12:55:11.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:11.431+0000 7f57d67df640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-22T12:55:11.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:11.431+0000 7f57d67df640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-22T12:55:11.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:11.431+0000 7f57d67df640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-22T12:55:11.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:11.433+0000 7f57d67df640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-22T12:55:14.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T12:55:14.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:14.854+0000 7f6cfa147640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-22T12:55:14.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:14.855+0000 7f6cfa147640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-22T12:55:14.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:14.855+0000 7f6cfa147640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-22T12:55:14.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:14.855+0000 7f6cfa147640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-22T12:55:14.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:14.855+0000 7f6cfa147640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-22T12:55:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:14.858+0000 7f6cfa147640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-22T12:55:18.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T12:55:18.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:18.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:18.279+0000 7f0d4ffbb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-22T12:55:18.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:18.280+0000 7f0d4ffbb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-22T12:55:18.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:18.280+0000 7f0d4ffbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-22T12:55:18.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:18.280+0000 7f0d4ffbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-22T12:55:18.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:18.280+0000 7f0d4ffbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-22T12:55:18.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:18.282+0000 7f0d4ffbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-22T12:55:21.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T12:55:21.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-22T12:55:21.610 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:21.610 INFO:tasks.ceph:role: osd.4 2024-03-22T12:55:21.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T12:55:21.610 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:21.688 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:22.197 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-22T12:55:22.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-22T12:55:22.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-22T12:55:22.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-22T12:55:22.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-22T12:55:22.452 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:22.452 INFO:tasks.ceph:role: osd.5 2024-03-22T12:55:22.452 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T12:55:22.453 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-22T12:55:22.529 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:22.530 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:22.978 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-22T12:55:22.980 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-22T12:55:22.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-22T12:55:23.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-22T12:55:23.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-22T12:55:23.179 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:23.179 INFO:tasks.ceph:role: osd.6 2024-03-22T12:55:23.179 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T12:55:23.179 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:23.257 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:23.744 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-22T12:55:23.746 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-22T12:55:23.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-22T12:55:23.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-22T12:55:23.928 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-22T12:55:23.999 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-22T12:55:23.999 INFO:tasks.ceph:role: osd.7 2024-03-22T12:55:23.999 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T12:55:23.999 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-22T12:55:24.077 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-22T12:55:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-22T12:55:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-22T12:55:24.077 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-22T12:55:24.077 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-22T12:55:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2024-03-22T12:55:24.078 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-22T12:55:24.078 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-22T12:55:24.078 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-22T12:55:24.078 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-22T12:55:24.587 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2024-03-22T12:55:24.590 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2024-03-22T12:55:24.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-22T12:55:24.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-22T12:55:24.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:24.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:24.889+0000 7f66954f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-22T12:55:24.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:24.889+0000 7f66954f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-22T12:55:24.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:24.889+0000 7f66954f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-22T12:55:24.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:24.889+0000 7f66954f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-22T12:55:24.889 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:24.889+0000 7f66954f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-22T12:55:24.890 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:24.890+0000 7f66954f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-22T12:55:27.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T12:55:27.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:27.879 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:27.880+0000 7f8be3dbd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-22T12:55:27.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:27.880+0000 7f8be3dbd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-22T12:55:27.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:27.880+0000 7f8be3dbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-22T12:55:27.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:27.880+0000 7f8be3dbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-22T12:55:27.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:27.880+0000 7f8be3dbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-22T12:55:27.880 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:27.881+0000 7f8be3dbd640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-22T12:55:29.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:55:29.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:55:30.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T12:55:30.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:30.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:30.847+0000 7f0fff20d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-22T12:55:30.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:30.847+0000 7f0fff20d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-22T12:55:30.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:30.847+0000 7f0fff20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-22T12:55:30.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:30.847+0000 7f0fff20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-22T12:55:30.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:30.847+0000 7f0fff20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-22T12:55:30.847 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:30.848+0000 7f0fff20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-22T12:55:33.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-22T12:55:33.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:33.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:33.829+0000 7f53efc1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-22T12:55:33.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:33.829+0000 7f53efc1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-22T12:55:33.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:33.830+0000 7f53efc1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-22T12:55:33.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:33.830+0000 7f53efc1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-22T12:55:33.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:33.830+0000 7f53efc1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-22T12:55:33.829 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:33.830+0000 7f53efc1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-22T12:55:36.677 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-22T12:55:36.709 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-22T12:55:36.709 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:36.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-22T12:55:36.817 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:36.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-22T12:55:36.920 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:36.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-22T12:55:37.020 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:37.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-22T12:55:37.124 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:37.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-22T12:55:37.216 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:37.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-22T12:55:37.315 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:37.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-22T12:55:37.418 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:37.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-22T12:55:37.516 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.516 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-22T12:55:37.548 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-22T12:55:37.617 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-22T12:55:37.686 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-22T12:55:37.755 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-22T12:55:37.822 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-22T12:55:37.889 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-22T12:55:37.956 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:37.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-22T12:55:38.022 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:38.022 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-22T12:55:38.083 INFO:tasks.ceph:Adding keys to all mons... 2024-03-22T12:55:38.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-22T12:55:38.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-22T12:55:38.112 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCmf/1l+bChHhAA2wNsC9qxYw33qEsHVSsvtQ== 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCmf/1lfEo6LxAAkPBy68/F1s1dJ127pnBpNQ== 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCnf/1lmsz8BRAAIKzxDcEphv3Xs03ZE8ytWA== 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.e] 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCnf/1l7WWgFxAA6BFr4eC5sJDJfEkULxgquQ== 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCrf/1lbdI9OhAAv7ZVe+wmp8iXj+vmVS8WZw== 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2024-03-22T12:55:38.113 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCvf/1lsnO2GRAAxCnO91MCn2+wO5yF3s0Lwg== 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCyf/1lblkEMxAAJpn40jkjU8LaerQgLqgZbA== 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC2f/1lYL6yEBAAofT68U22cUkAoXqku5QCtw== 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCmf/1lPuCfIxAArQVUJboWTHPYAXn1Xfi/ZQ== 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCnf/1l6KGtIRAAaoQurV7/j2i08Uw7WflolQ== 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCnf/1lkw4YKRAAeJnJpWXUy1QgASm7VX+QKg== 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.f] 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCnf/1l5gxcMBAAzD7GEi/KfQZiepT5yEKcig== 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2024-03-22T12:55:38.114 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC8f/1lzWoLNRAA+fgVvHs5DPjgoKUw6Hiy/A== 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQC/f/1lySeBNBAAWQ+ndGWvBvyfkRBXyrzn2Q== 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDCf/1lRAqNMhAAtQEgxL7+i9z1PjTsun0wFw== 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDFf/1lQId7MRAAvYJY5wr94CD1iYJ1yFhw2g== 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-03-22T12:55:38.115 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCnf/1lY1BkOhAAKW5EpLpMH/5M3S9V2KbDWg== 2024-03-22T12:55:38.176 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-03-22T12:55:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCmf/1l+bChHhAA2wNsC9qxYw33qEsHVSsvtQ== 2024-03-22T12:55:38.176 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-22T12:55:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCmf/1lfEo6LxAAkPBy68/F1s1dJ127pnBpNQ== 2024-03-22T12:55:38.176 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCnf/1lmsz8BRAAIKzxDcEphv3Xs03ZE8ytWA== 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.e] 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCnf/1l7WWgFxAA6BFr4eC5sJDJfEkULxgquQ== 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCrf/1lbdI9OhAAv7ZVe+wmp8iXj+vmVS8WZw== 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCvf/1lsnO2GRAAxCnO91MCn2+wO5yF3s0Lwg== 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCyf/1lblkEMxAAJpn40jkjU8LaerQgLqgZbA== 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC2f/1lYL6yEBAAofT68U22cUkAoXqku5QCtw== 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-03-22T12:55:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCmf/1lPuCfIxAArQVUJboWTHPYAXn1Xfi/ZQ== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCnf/1l6KGtIRAAaoQurV7/j2i08Uw7WflolQ== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCnf/1lkw4YKRAAeJnJpWXUy1QgASm7VX+QKg== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.f] 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCnf/1l5gxcMBAAzD7GEi/KfQZiepT5yEKcig== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC8f/1lzWoLNRAA+fgVvHs5DPjgoKUw6Hiy/A== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC/f/1lySeBNBAAWQ+ndGWvBvyfkRBXyrzn2Q== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDCf/1lRAqNMhAAtQEgxL7+i9z1PjTsun0wFw== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDFf/1lQId7MRAAvYJY5wr94CD1iYJ1yFhw2g== 2024-03-22T12:55:38.178 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-22T12:55:38.179 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCnf/1lY1BkOhAAKW5EpLpMH/5M3S9V2KbDWg== 2024-03-22T12:55:38.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-22T12:55:38.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-22T12:55:38.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:38.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:38.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:38.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:38.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:38.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:38.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:38.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:38.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:38.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:39.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:39.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:39.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:39.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:39.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-22T12:55:39.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-22T12:55:39.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:39.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:39.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:39.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:39.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:39.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-22T12:55:39.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:39.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:40.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:40.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:40.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:40.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:40.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:40.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-22T12:55:40.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-22T12:55:40.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-22T12:55:40.695 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-22T12:55:40.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-22T12:55:40.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-22T12:55:40.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-22T12:55:41.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-22T12:55:41.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-22T12:55:41.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-22T12:55:41.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-22T12:55:41.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-22T12:55:41.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-22T12:55:41.394 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:41.402 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-22T12:55:41.454 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-22T12:55:41.454 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-22T12:55:41.454 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-22T12:55:41.454 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-22T12:55:41.460 INFO:tasks.ceph.mon.a:Started 2024-03-22T12:55:41.460 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-22T12:55:41.460 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-22T12:55:41.460 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-22T12:55:41.490 INFO:tasks.ceph.mon.b:Started 2024-03-22T12:55:41.490 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-22T12:55:41.491 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-22T12:55:41.491 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-22T12:55:41.493 INFO:tasks.ceph.mon.c:Started 2024-03-22T12:55:41.493 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-22T12:55:41.493 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-22T12:55:41.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-22T12:55:41.498 INFO:tasks.ceph.mgr.y:Started 2024-03-22T12:55:41.498 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-22T12:55:41.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-22T12:55:41.500 INFO:tasks.ceph.mgr.x:Started 2024-03-22T12:55:41.500 INFO:tasks.ceph:Setting crush tunables to default 2024-03-22T12:55:41.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-03-22T12:55:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.682+0000 7f118e800640 1 Processor -- start 2024-03-22T12:55:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.682+0000 7f118e800640 1 -- start start 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f118e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f118815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f118e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 0x7f11880ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f118e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880c5470 con 0x7f1188067550 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f118e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880cd050 con 0x7f11880cc700 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f118e800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880c4ef0 con 0x7f11881548b0 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f1186a00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f11880cb340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f118815b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f1187e00640 1 -- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 msgr2=0x7f11880ccb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.6:3300/0 2024-03-22T12:55:41.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f118815b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:41.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.684+0000 7f1187e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 0x7f11880ccb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:41.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.885+0000 7f1187e00640 1 -- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 msgr2=0x7f11880ccb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.6:3300/0 2024-03-22T12:55:41.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.885+0000 7f1187e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 0x7f11880ccb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:41.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.885+0000 7f1186a00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f11880cb340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:41.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.885+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f118815b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:41.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:41.885+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f118815b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:42.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:42.286+0000 7f1187e00640 1 -- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 msgr2=0x7f11880ccb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.6:3300/0 2024-03-22T12:55:42.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:42.286+0000 7f1187e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 0x7f11880ccb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:42.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:42.286+0000 7f1186a00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f11880cb340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:42.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:42.286+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f118815b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:42.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:42.286+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f118815b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:43.087+0000 7f1187e00640 1 -- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 msgr2=0x7f11880ccb10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.6:3300/0 2024-03-22T12:55:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:43.087+0000 7f1187e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 0x7f11880ccb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:43.087+0000 7f1186a00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f11880cb340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:43.087+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f118815b100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:43.087+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f118815b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:44.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f118815b100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f118815b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 -- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 msgr2=0x7f11880ccb10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11880cc700 0x7f11880ccb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f11880cb340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880cba50 con 0x7f1168003690 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1184c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11880cd050 con 0x7f1168007380 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1187400640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f1168003690 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.683+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.684+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:44.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.884+0000 7f1187400640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f1168003690 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2024-03-22T12:55:44.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.885+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:44.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:44.885+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:45.285+0000 7f1187400640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f1168003690 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2024-03-22T12:55:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:45.286+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:45.286+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:46.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:46.086+0000 7f1187400640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f1168003690 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2024-03-22T12:55:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:46.087+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:46.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:46.087+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.683+0000 7f1184c00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.683+0000 7f1184c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.683+0000 7f1184c00640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f1168003690 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.683+0000 7f1187e00640 1 -- reap_dead start 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1184c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f1168009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1184c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1184c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880c5470 con 0x7f1188067550 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1184c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880cd050 con 0x7f11881548b0 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1184c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11880c4ef0 con 0x7f1168007380 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f1168009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f1168009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.684+0000 7f1187e00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f116800a500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:47.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.885+0000 7f1187e00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f116800a500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:47.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.885+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f1168009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:47.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.885+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:47.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.885+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f1168009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:47.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:47.885+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:48.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:48.286+0000 7f1187e00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f116800a500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:48.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:48.286+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f1168009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:48.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:48.286+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:48.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:48.286+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f1168009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:48.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:48.286+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:49.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:49.087+0000 7f1186a00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3301/0 2024-03-22T12:55:49.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:49.087+0000 7f1186a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:49.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:49.087+0000 7f1187400640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f1168009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2024-03-22T12:55:49.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:49.087+0000 7f1187e00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f116800a500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-03-22T12:55:49.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:49.087+0000 7f1187400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f1168009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:49.118 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:00:07.509 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T12:55:49.119 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:00:07.509 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T12:55:49.439 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:00:07.829 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T12:55:49.440 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:00:07.829 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T12:55:49.701 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:00:08.036 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T12:55:50.039 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:00:08.373 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T12:55:50.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 -- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 msgr2=0x7f1168009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168007380 0x7f1168009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 msgr2=0x7f1168009f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11881548b0 0x7f1168009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f116800a500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880cba50 con 0x7f11680062e0 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.684+0000 7f1184c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880cd050 con 0x7f11680066c0 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.685+0000 7f1186a00640 1 -- >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.6:3300/0 2024-03-22T12:55:50.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.685+0000 7f1186a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:50.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.886+0000 7f1186a00640 1 -- >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.6:3300/0 2024-03-22T12:55:50.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.886+0000 7f1186a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:50.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.909+0000 7f1187400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f11680062e0 0x7f1168005b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:32932/0 (socket says 172.21.15.6:32932) 2024-03-22T12:55:50.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:50.909+0000 7f1187400640 1 -- 172.21.15.6:0/2905519957 learned_addr learned my addr 172.21.15.6:0/2905519957 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:55:51.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:51.134+0000 7f1187400640 1 -- 172.21.15.6:0/2905519957 >> v1:172.21.15.92:6789/0 conn(0x7f11680062e0 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-22T12:55:51.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:51.134+0000 7f1187400640 1 -- 172.21.15.6:0/2905519957 >> v1:172.21.15.92:6789/0 conn(0x7f11680062e0 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T12:55:51.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:51.134+0000 7f1187400640 1 --1- 172.21.15.6:0/2905519957 >> v1:172.21.15.92:6789/0 conn(0x7f11680062e0 0x7f1168005b10 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-22T12:55:51.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:51.134+0000 7f1187400640 1 --1- 172.21.15.6:0/2905519957 >> v1:172.21.15.92:6789/0 conn(0x7f11680062e0 0x7f1168005b10 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-22T12:55:51.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:51.287+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.6:3300/0 2024-03-22T12:55:51.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:51.287+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:52.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:52.779+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:52.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:52.789+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-22T12:55:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:52.789+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:52.789+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:52.789+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:53.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.684+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:53.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.684+0000 7f1184c00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:53.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.685+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 >> v1:172.21.15.92:6789/0 conn(0x7f11680062e0 legacy=0x7f1168005b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-22T12:55:53.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.685+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 reap_dead start 2024-03-22T12:55:53.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.685+0000 7f1184c00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:53.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.685+0000 7f1184c00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:53.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.685+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880c5470 con 0x7f1188067550 2024-03-22T12:55:53.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.685+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880cd050 con 0x7f11680066c0 2024-03-22T12:55:53.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.685+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880c4ef0 con 0x7f1168009f60 2024-03-22T12:55:53.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.697+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:53.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.698+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-22T12:55:53.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.698+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:53.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.698+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:53.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.698+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:53.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.900+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:53.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.901+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-22T12:55:53.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.901+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:53.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.901+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:53.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:53.901+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:54.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.303+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.303+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-22T12:55:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.304+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.304+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.304+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:54.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.571+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.573+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-22T12:55:54.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.573+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:54.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.573+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:54.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.573+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:55:54.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.836+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.884+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-22T12:55:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.884+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:54.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.884+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:54.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:54.885+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:55:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.106+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.107+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-22T12:55:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.107+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.107+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:55.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.108+0000 7f1187e00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.419+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.424+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-22T12:55:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.424+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.424+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:55.424+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:55:56.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.285+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:56.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.298+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-22T12:55:56.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.298+0000 7f1186a00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-22T12:55:56.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.298+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T12:55:56.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.298+0000 7f1186a00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 msgr2=0x7f11680101d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.92:3300/0 conn(0x7f1168009f60 0x7f11680101d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 msgr2=0x7f1168006ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 --2- 172.21.15.6:0/2905519957 >> v2:172.21.15.6:3300/0 conn(0x7f11680066c0 0x7f1168006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 >> v1:172.21.15.6:6789/0 conn(0x7f1188067550 legacy=0x7f116800a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 --2- 172.21.15.6:0/2905519957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168010790 0x7f1168009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1168007110 con 0x7f1168010cc0 2024-03-22T12:55:56.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.686+0000 7f1184c00640 1 -- 172.21.15.6:0/2905519957 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11880cd050 con 0x7f1168010790 2024-03-22T12:55:56.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.704+0000 7f1187400640 1 --2- 172.21.15.6:0/2905519957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168010790 0x7f1168009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:56.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.718+0000 7f1187400640 1 -- 172.21.15.6:0/2905519957 >> v1:172.21.15.92:6789/0 conn(0x7f1168010cc0 legacy=0x7f1168005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:56.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.718+0000 7f1187400640 1 -- 172.21.15.6:0/2905519957 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168015330 con 0x7f1168010790 2024-03-22T12:55:56.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.737+0000 7f1187400640 1 --2- 172.21.15.6:0/2905519957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168010790 0x7f1168009840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1178009c80 tx=0x7f117802c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b46fd6ce63428977 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:56.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.779+0000 7f1186000640 1 -- 172.21.15.6:0/2905519957 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178003d40 con 0x7f1168010790 2024-03-22T12:55:56.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.796+0000 7f1186000640 1 -- 172.21.15.6:0/2905519957 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1178005070 con 0x7f1168010790 2024-03-22T12:55:56.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.796+0000 7f1186000640 1 -- 172.21.15.6:0/2905519957 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178005460 con 0x7f1168010790 2024-03-22T12:55:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.798+0000 7f118e800640 1 -- 172.21.15.6:0/2905519957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168010790 msgr2=0x7f1168009840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.798+0000 7f118e800640 1 --2- 172.21.15.6:0/2905519957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168010790 0x7f1168009840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1178009c80 tx=0x7f117802c3d0 comp rx=0 tx=0).stop 2024-03-22T12:55:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.798+0000 7f1187e00640 1 -- 172.21.15.6:0/2905519957 reap_dead start 2024-03-22T12:55:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.798+0000 7f118e800640 1 -- 172.21.15.6:0/2905519957 shutdown_connections 2024-03-22T12:55:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.798+0000 7f118e800640 1 -- 172.21.15.6:0/2905519957 >> 172.21.15.6:0/2905519957 conn(0x7f118806cbc0 msgr2=0x7f1188066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:55:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.798+0000 7f118e800640 1 -- 172.21.15.6:0/2905519957 shutdown_connections 2024-03-22T12:55:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.798+0000 7f118e800640 1 -- 172.21.15.6:0/2905519957 wait complete. 2024-03-22T12:55:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.805+0000 7f118e800640 1 Processor -- start 2024-03-22T12:55:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.806+0000 7f118e800640 1 -- start start 2024-03-22T12:55:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.808+0000 7f118e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1188156540 0x7f11881f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.808+0000 7f118e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1188156920 0x7f1188158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.808+0000 7f118e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f118814d790 0x7f11881f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.808+0000 7f118e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880cab40 con 0x7f1188156920 2024-03-22T12:55:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.808+0000 7f118e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880ca860 con 0x7f118814d790 2024-03-22T12:55:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.808+0000 7f118e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11880c5880 con 0x7f1188156540 2024-03-22T12:55:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.809+0000 7f1187400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1188156540 0x7f11881f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.809+0000 7f1187e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f118814d790 0x7f11881f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.809+0000 7f1187400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1188156540 0x7f11881f4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:38902/0 (socket says 172.21.15.6:38902) 2024-03-22T12:55:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.809+0000 7f1187400640 1 -- 172.21.15.6:0/399045893 learned_addr learned my addr 172.21.15.6:0/399045893 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:55:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.813+0000 7f1187400640 1 -- 172.21.15.6:0/399045893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f118814d790 msgr2=0x7f11881f0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.813+0000 7f1187400640 1 --2- 172.21.15.6:0/399045893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f118814d790 0x7f11881f0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.813+0000 7f1187400640 1 -- 172.21.15.6:0/399045893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1188156920 msgr2=0x7f1188158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.813+0000 7f1187400640 1 --2- 172.21.15.6:0/399045893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1188156920 0x7f1188158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.813+0000 7f1187400640 1 -- 172.21.15.6:0/399045893 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1188156ee0 con 0x7f1188156540 2024-03-22T12:55:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.815+0000 7f1187400640 1 --2- 172.21.15.6:0/399045893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1188156540 0x7f11881f4380 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f117802c9e0 tx=0x7f1178000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.819+0000 7f117f400640 1 -- 172.21.15.6:0/399045893 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178005840 con 0x7f1188156540 2024-03-22T12:55:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.819+0000 7f117f400640 1 -- 172.21.15.6:0/399045893 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1178044070 con 0x7f1188156540 2024-03-22T12:55:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.819+0000 7f117f400640 1 -- 172.21.15.6:0/399045893 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178034630 con 0x7f1188156540 2024-03-22T12:55:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.819+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11881571c0 con 0x7f1188156540 2024-03-22T12:55:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.820+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11881f38a0 con 0x7f1188156540 2024-03-22T12:55:56.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.822+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118806b740 con 0x7f1188156540 2024-03-22T12:55:56.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.825+0000 7f117f400640 1 -- 172.21.15.6:0/399045893 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f11780347d0 con 0x7f1188156540 2024-03-22T12:55:56.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:56.825+0000 7f117f400640 1 -- 172.21.15.6:0/399045893 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1178049770 con 0x7f1188156540 2024-03-22T12:55:57.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.013+0000 7f117f400640 1 -- 172.21.15.6:0/399045893 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1178034ac0 con 0x7f1188156540 2024-03-22T12:55:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.101+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f11880757d0 con 0x7f1188156540 2024-03-22T12:55:57.307 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:57.307+0000 7f0efea30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T12:55:57.367 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:57.368+0000 7f0efea30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T12:55:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.633+0000 7f117f400640 1 -- 172.21.15.6:0/399045893 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1178034ca0 con 0x7f1188156540 2024-03-22T12:55:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1188156540 msgr2=0x7f11881f4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 --2- 172.21.15.6:0/399045893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1188156540 0x7f11881f4380 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f117802c9e0 tx=0x7f1178000bc0 comp rx=0 tx=0).stop 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 shutdown_connections 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 --2- 172.21.15.6:0/399045893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f118814d790 0x7f11881f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 --2- 172.21.15.6:0/399045893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1188156920 0x7f1188158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 --2- 172.21.15.6:0/399045893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1188156540 0x7f11881f4380 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 >> 172.21.15.6:0/399045893 conn(0x7f118806cbc0 msgr2=0x7f11880d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 shutdown_connections 2024-03-22T12:55:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:55:57.635+0000 7f118e800640 1 -- 172.21.15.6:0/399045893 wait complete. 2024-03-22T12:55:57.658 INFO:tasks.ceph:check_enable_crimson: False 2024-03-22T12:55:57.658 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-22T12:55:57.658 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:57.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-22T12:55:57.749 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:57.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-22T12:55:57.875 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:57.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-22T12:55:57.907 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:57.908+0000 7f0efea30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T12:55:57.963 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:57.964+0000 7f0efea30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T12:55:57.996 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:55:57.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-22T12:55:58.009 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.009+0000 7f0efea30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T12:55:58.062 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.063+0000 7f0efea30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T12:55:58.106 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.107+0000 7f0efea30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T12:55:58.120 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:58.120+0000 7f1d3c417040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T12:55:58.131 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:58.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-22T12:55:58.158 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:58.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-22T12:55:58.192 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:58.192+0000 7f1d3c417040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T12:55:58.224 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:58.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-22T12:55:58.228 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.228+0000 7f0efea30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T12:55:58.253 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-22T12:55:58.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-22T12:55:58.300 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.301+0000 7f0efea30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T12:55:58.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 256bea72-2eca-4d53-a283-029ac721281b 0 2024-03-22T12:55:58.348 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.349+0000 7f0efea30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T12:55:58.404 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.404+0000 7f0efea30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T12:55:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.449+0000 7f9cf21ab640 1 Processor -- start 2024-03-22T12:55:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.449+0000 7f9cf21ab640 1 -- start start 2024-03-22T12:55:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.450+0000 7f9cf21ab640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cec15adb0 0x7f9cec165cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.450+0000 7f9cf21ab640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec06c950 con 0x7f9cec1545b0 2024-03-22T12:55:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.450+0000 7f9cf21ab640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec06c1e0 con 0x7f9cec0c7150 2024-03-22T12:55:58.449 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.450+0000 7f9cf21ab640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cec15b310 con 0x7f9cec15adb0 2024-03-22T12:55:58.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.451+0000 7f9cebfff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cec15adb0 0x7f9cec165cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:58.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.451+0000 7f9ce2ffd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9cec0c7150 0x7f9cec071f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:54428/0 (socket says 172.21.15.92:54428) 2024-03-22T12:55:58.450 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.451+0000 7f9ce2ffd640 1 -- 172.21.15.92:0/2018830863 learned_addr learned my addr 172.21.15.92:0/2018830863 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-22T12:55:58.453 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.454+0000 7f0efea30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T12:55:58.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.455+0000 7f9cebfff640 1 -- 172.21.15.92:0/2018830863 >> v1:172.21.15.92:6789/0 conn(0x7f9cec0c7150 legacy=0x7f9cec071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:58.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.455+0000 7f9cebfff640 1 -- 172.21.15.92:0/2018830863 >> v1:172.21.15.6:6789/0 conn(0x7f9cec1545b0 legacy=0x7f9cec06bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:58.455 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.455+0000 7f9cebfff640 1 -- 172.21.15.92:0/2018830863 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec067020 con 0x7f9cec15adb0 2024-03-22T12:55:58.459 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.460+0000 7f9cebfff640 1 --2- 172.21.15.92:0/2018830863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec15adb0 0x7f9cec165cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009d90 tx=0x7f9ce402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ad4a327b1212ab server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:58.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.467+0000 7f9ceaffd640 1 -- 172.21.15.92:0/2018830863 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4039070 con 0x7f9cec15adb0 2024-03-22T12:55:58.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.467+0000 7f9ceaffd640 1 -- 172.21.15.92:0/2018830863 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce402cc60 con 0x7f9cec15adb0 2024-03-22T12:55:58.466 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.467+0000 7f9ceaffd640 1 -- 172.21.15.92:0/2018830863 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4034660 con 0x7f9cec15adb0 2024-03-22T12:55:58.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.468+0000 7f9cf21ab640 1 -- 172.21.15.92:0/2018830863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec15adb0 msgr2=0x7f9cec165cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:58.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.468+0000 7f9cf21ab640 1 --2- 172.21.15.92:0/2018830863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec15adb0 0x7f9cec165cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009d90 tx=0x7f9ce402beb0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.468+0000 7f9cf21ab640 1 -- 172.21.15.92:0/2018830863 shutdown_connections 2024-03-22T12:55:58.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.468+0000 7f9cf21ab640 1 --2- 172.21.15.92:0/2018830863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec15adb0 0x7f9cec165cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.468+0000 7f9cf21ab640 1 -- 172.21.15.92:0/2018830863 >> 172.21.15.92:0/2018830863 conn(0x7f9cec0c8bf0 msgr2=0x7f9cec0c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:55:58.467 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.468+0000 7f9cf21ab640 1 -- 172.21.15.92:0/2018830863 shutdown_connections 2024-03-22T12:55:58.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.468+0000 7f9cf21ab640 1 -- 172.21.15.92:0/2018830863 wait complete. 2024-03-22T12:55:58.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.469+0000 7f9cf21ab640 1 Processor -- start 2024-03-22T12:55:58.468 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.469+0000 7f9cf21ab640 1 -- start start 2024-03-22T12:55:58.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9cf21ab640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec0c7150 0x7f9cec1eff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:58.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9cf21ab640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cec1545b0 0x7f9cec0ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:58.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9cf21ab640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cec15adb0 0x7f9cec14d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:58.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9cf21ab640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec06b2d0 con 0x7f9cec15adb0 2024-03-22T12:55:58.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9cf21ab640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec06b080 con 0x7f9cec0c7150 2024-03-22T12:55:58.469 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9cf21ab640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec06b5b0 con 0x7f9cec1545b0 2024-03-22T12:55:58.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9ce2ffd640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cec1545b0 0x7f9cec0ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:58.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.470+0000 7f9ceb7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec0c7150 0x7f9cec1eff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:58.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.471+0000 7f9ceb7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec0c7150 0x7f9cec1eff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:38968/0 (socket says 172.21.15.92:38968) 2024-03-22T12:55:58.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.471+0000 7f9ceb7fe640 1 -- 172.21.15.92:0/870063169 learned_addr learned my addr 172.21.15.92:0/870063169 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:55:58.470 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.471+0000 7f9cebfff640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cec15adb0 0x7f9cec14d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:58.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.474+0000 7f9ceb7fe640 1 -- 172.21.15.92:0/870063169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cec1545b0 msgr2=0x7f9cec0ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:58.473 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.474+0000 7f9ceb7fe640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cec1545b0 0x7f9cec0ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.474+0000 7f9ceb7fe640 1 -- 172.21.15.92:0/870063169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cec15adb0 msgr2=0x7f9cec14d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:58.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.474+0000 7f9ceb7fe640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cec15adb0 0x7f9cec14d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.474 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.474+0000 7f9ceb7fe640 1 -- 172.21.15.92:0/870063169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec157770 con 0x7f9cec0c7150 2024-03-22T12:55:58.476 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.477+0000 7f9ceb7fe640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec0c7150 0x7f9cec1eff70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c430 tx=0x7f9cdc00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:58.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.479+0000 7f9ce8ff9640 1 -- 172.21.15.92:0/870063169 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0052d0 con 0x7f9cec0c7150 2024-03-22T12:55:58.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.479+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec150770 con 0x7f9cec0c7150 2024-03-22T12:55:58.478 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.479+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec157a10 con 0x7f9cec0c7150 2024-03-22T12:55:58.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.480+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec0757d0 con 0x7f9cec0c7150 2024-03-22T12:55:58.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.480+0000 7f9ce8ff9640 1 -- 172.21.15.92:0/870063169 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc005470 con 0x7f9cec0c7150 2024-03-22T12:55:58.479 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.480+0000 7f9ce8ff9640 1 -- 172.21.15.92:0/870063169 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc020060 con 0x7f9cec0c7150 2024-03-22T12:55:58.657 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.658+0000 7f9ce8ff9640 1 -- 172.21.15.92:0/870063169 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9cdc007350 con 0x7f9cec0c7150 2024-03-22T12:55:58.658 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.658+0000 7f9ce8ff9640 1 -- 172.21.15.92:0/870063169 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9cdc003960 con 0x7f9cec0c7150 2024-03-22T12:55:58.658 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.658+0000 7f9ce8ff9640 1 -- 172.21.15.92:0/870063169 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9cdc003bd0 con 0x7f9cec0c7150 2024-03-22T12:55:58.734 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.734+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "256bea72-2eca-4d53-a283-029ac721281b", "id": 0} v 0) v1 -- 0x7f9cec0c64a0 con 0x7f9cec0c7150 2024-03-22T12:55:58.832 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:58.833+0000 7f1d3c417040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T12:55:58.840 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.840+0000 7f0efea30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T12:55:58.900 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:58.900+0000 7f1d3c417040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T12:55:58.956 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:58.956+0000 7f1d3c417040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T12:55:58.974 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:58.975+0000 7f0efea30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T12:55:58.991 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.992+0000 7f9ce8ff9640 1 -- 172.21.15.92:0/870063169 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "256bea72-2eca-4d53-a283-029ac721281b", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9cdc003db0 con 0x7f9cec0c7150 2024-03-22T12:55:58.992 INFO:teuthology.orchestra.run.smithi092.stdout:0 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec0c7150 msgr2=0x7f9cec1eff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec0c7150 0x7f9cec1eff70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c430 tx=0x7f9cdc00c8f0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 shutdown_connections 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cec15adb0 0x7f9cec14d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cec1545b0 0x7f9cec0ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 --2- 172.21.15.92:0/870063169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cec0c7150 0x7f9cec1eff70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 >> 172.21.15.92:0/870063169 conn(0x7f9cec0c8bf0 msgr2=0x7f9cec0c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 shutdown_connections 2024-03-22T12:55:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:58.993+0000 7f9cf21ab640 1 -- 172.21.15.92:0/870063169 wait complete. 2024-03-22T12:55:59.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 1dad1564-b377-49fa-91e8-ddbe1febfeb8 1 2024-03-22T12:55:59.022 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.023+0000 7f1d3c417040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T12:55:59.075 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.075+0000 7f1d3c417040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T12:55:59.106 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:59.107+0000 7f0efea30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.109+0000 7f1a9c337640 1 Processor -- start 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.109+0000 7f1a9c337640 1 -- start start 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.110+0000 7f1a9c337640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a941417f0 0x7f1a940c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.110+0000 7f1a9c337640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a9406b080 0x7f1a94070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.110+0000 7f1a9c337640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a9406b4a0 0x7f1a940c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.110+0000 7f1a9c337640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9406b930 con 0x7f1a9406b080 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.110+0000 7f1a9c337640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9406bc10 con 0x7f1a9406b4a0 2024-03-22T12:55:59.110 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.110+0000 7f1a9c337640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a94158500 con 0x7f1a941417f0 2024-03-22T12:55:59.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.111+0000 7f1a9a8d4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a9406b4a0 0x7f1a940c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.111+0000 7f1a9a8d4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a9406b4a0 0x7f1a940c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:38982/0 (socket says 172.21.15.92:38982) 2024-03-22T12:55:59.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.111+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/429768271 learned_addr learned my addr 172.21.15.92:0/429768271 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:55:59.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.112+0000 7f1a9a0d3640 1 --2- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a941417f0 0x7f1a940c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.111 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.112+0000 7f1a998d2640 1 --2- 172.21.15.92:0/429768271 >> v2:172.21.15.6:3300/0 conn(0x7f1a9406b080 0x7f1a94070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.114+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a941417f0 msgr2=0x7f1a940c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:55:59.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.114+0000 7f1a9a8d4640 1 --2- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a941417f0 0x7f1a940c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.114+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/429768271 >> v2:172.21.15.6:3300/0 conn(0x7f1a9406b080 msgr2=0x7f1a94070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:55:59.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.114+0000 7f1a9a8d4640 1 --2- 172.21.15.92:0/429768271 >> v2:172.21.15.6:3300/0 conn(0x7f1a9406b080 0x7f1a94070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.114+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/429768271 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a941587e0 con 0x7f1a9406b4a0 2024-03-22T12:55:59.115 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.116+0000 7f1a9a8d4640 1 --2- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9406b4a0 0x7f1a940c60c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a94073b40 tx=0x7f1a8c02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c97a729de64015 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:59.119 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.119+0000 7f1a990d1640 1 -- 172.21.15.92:0/429768271 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c02c9e0 con 0x7f1a9406b4a0 2024-03-22T12:55:59.119 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a990d1640 1 -- 172.21.15.92:0/429768271 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8c02cb80 con 0x7f1a9406b4a0 2024-03-22T12:55:59.119 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a990d1640 1 -- 172.21.15.92:0/429768271 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0343f0 con 0x7f1a9406b4a0 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 -- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9406b4a0 msgr2=0x7f1a940c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 --2- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9406b4a0 0x7f1a940c60c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a94073b40 tx=0x7f1a8c02be40 comp rx=0 tx=0).stop 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 -- 172.21.15.92:0/429768271 shutdown_connections 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 --2- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9406b4a0 0x7f1a940c60c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 --2- 172.21.15.92:0/429768271 >> v2:172.21.15.6:3300/0 conn(0x7f1a9406b080 0x7f1a94070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 --2- 172.21.15.92:0/429768271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a941417f0 0x7f1a940c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 -- 172.21.15.92:0/429768271 >> 172.21.15.92:0/429768271 conn(0x7f1a94072790 msgr2=0x7f1a94072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:55:59.120 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 -- 172.21.15.92:0/429768271 shutdown_connections 2024-03-22T12:55:59.121 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.120+0000 7f1a9c337640 1 -- 172.21.15.92:0/429768271 wait complete. 2024-03-22T12:55:59.121 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.122+0000 7f1a9c337640 1 Processor -- start 2024-03-22T12:55:59.121 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.122+0000 7f1a9c337640 1 -- start start 2024-03-22T12:55:59.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.122+0000 7f1a9c337640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9406b080 0x7f1a9413db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.122+0000 7f1a9c337640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a941417f0 0x7f1a9413a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.122+0000 7f1a9c337640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9413a7c0 0x7f1a94153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.123+0000 7f1a9c337640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940707a0 con 0x7f1a941417f0 2024-03-22T12:55:59.122 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.123+0000 7f1a9c337640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a941581f0 con 0x7f1a9413a7c0 2024-03-22T12:55:59.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.123+0000 7f1a9c337640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9413ac40 con 0x7f1a9406b080 2024-03-22T12:55:59.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.123+0000 7f1a9a8d4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9413a7c0 0x7f1a94153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.123+0000 7f1a9a0d3640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9406b080 0x7f1a9413db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.123+0000 7f1a9a8d4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9413a7c0 0x7f1a94153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:38984/0 (socket says 172.21.15.92:38984) 2024-03-22T12:55:59.123 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.123+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/2336656709 learned_addr learned my addr 172.21.15.92:0/2336656709 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:55:59.124 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.124+0000 7f1a998d2640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a941417f0 0x7f1a9413a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.127+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9406b080 msgr2=0x7f1a9413db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:59.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.127+0000 7f1a9a8d4640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9406b080 0x7f1a9413db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.127+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/2336656709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a941417f0 msgr2=0x7f1a9413a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:59.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.127+0000 7f1a9a8d4640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a941417f0 0x7f1a9413a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.127+0000 7f1a9a8d4640 1 -- 172.21.15.92:0/2336656709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a94155460 con 0x7f1a9413a7c0 2024-03-22T12:55:59.126 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.127+0000 7f1a9a0d3640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9406b080 0x7f1a9413db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:55:59.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.129+0000 7f1a9a8d4640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9413a7c0 0x7f1a94153da0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009f40 tx=0x7f1a8c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:59.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.132+0000 7f1a8affd640 1 -- 172.21.15.92:0/2336656709 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c043020 con 0x7f1a9413a7c0 2024-03-22T12:55:59.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.132+0000 7f1a8affd640 1 -- 172.21.15.92:0/2336656709 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8c03e450 con 0x7f1a9413a7c0 2024-03-22T12:55:59.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.132+0000 7f1a8affd640 1 -- 172.21.15.92:0/2336656709 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c009510 con 0x7f1a9413a7c0 2024-03-22T12:55:59.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.132+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a94155740 con 0x7f1a9413a7c0 2024-03-22T12:55:59.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.132+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9413be90 con 0x7f1a9413a7c0 2024-03-22T12:55:59.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.133+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a940757d0 con 0x7f1a9413a7c0 2024-03-22T12:55:59.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.135+0000 7f1a8affd640 1 -- 172.21.15.92:0/2336656709 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1a8c005d10 con 0x7f1a9413a7c0 2024-03-22T12:55:59.135 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.135+0000 7f1a8affd640 1 -- 172.21.15.92:0/2336656709 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f1a8c03e910 con 0x7f1a9413a7c0 2024-03-22T12:55:59.157 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:59.158+0000 7f0efea30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T12:55:59.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:55:59.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:55:59.213 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.213+0000 7f1d3c417040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T12:55:59.235 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.236+0000 7f1a8affd640 1 -- 172.21.15.92:0/2336656709 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1a8c04d020 con 0x7f1a9413a7c0 2024-03-22T12:55:59.274 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:59.275+0000 7f0efea30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T12:55:59.298 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.298+0000 7f1d3c417040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T12:55:59.308 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.309+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1dad1564-b377-49fa-91e8-ddbe1febfeb8", "id": 1} v 0) v1 -- 0x7f1a940c6dd0 con 0x7f1a9413a7c0 2024-03-22T12:55:59.355 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.355+0000 7f1d3c417040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T12:55:59.370 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:59.371+0000 7f0efea30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T12:55:59.419 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.419+0000 7f1d3c417040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T12:55:59.477 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.477+0000 7f1d3c417040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T12:55:59.510 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.510+0000 7f1a8affd640 1 -- 172.21.15.92:0/2336656709 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1dad1564-b377-49fa-91e8-ddbe1febfeb8", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1a8c02c9f0 con 0x7f1a9413a7c0 2024-03-22T12:55:59.510 INFO:teuthology.orchestra.run.smithi092.stdout:1 2024-03-22T12:55:59.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9413a7c0 msgr2=0x7f1a94153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:59.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9413a7c0 0x7f1a94153da0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009f40 tx=0x7f1a8c004a40 comp rx=0 tx=0).stop 2024-03-22T12:55:59.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 shutdown_connections 2024-03-22T12:55:59.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9413a7c0 0x7f1a94153da0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a941417f0 0x7f1a9413a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 --2- 172.21.15.92:0/2336656709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9406b080 0x7f1a9413db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.511 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 >> 172.21.15.92:0/2336656709 conn(0x7f1a94072790 msgr2=0x7f1a940cc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:55:59.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 shutdown_connections 2024-03-22T12:55:59.512 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.511+0000 7f1a9c337640 1 -- 172.21.15.92:0/2336656709 wait complete. 2024-03-22T12:55:59.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4 2 2024-03-22T12:55:59.531 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:59.531+0000 7f0efea30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T12:55:59.576 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:59.577+0000 7f0efea30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T12:55:59.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.630+0000 7f67a8edc640 1 Processor -- start 2024-03-22T12:55:59.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.630+0000 7f67a8edc640 1 -- start start 2024-03-22T12:55:59.630 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.631+0000 7f67a8edc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a414dc10 0x7f67a40c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.631+0000 7f67a8edc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f67a40c7680 0x7f67a41575a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.631+0000 7f67a8edc640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67a4157ae0 0x7f67a40cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.631+0000 7f67a8edc640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a406e4e0 con 0x7f67a4157ae0 2024-03-22T12:55:59.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.631+0000 7f67a8edc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a406ecf0 con 0x7f67a40c7680 2024-03-22T12:55:59.631 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.631+0000 7f67a8edc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a406f4b0 con 0x7f67a414dc10 2024-03-22T12:55:59.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.632+0000 7f67a259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a414dc10 0x7f67a40c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.632+0000 7f67a1d9b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f67a40c7680 0x7f67a41575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.632+0000 7f67a259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a414dc10 0x7f67a40c1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:40930/0 (socket says 172.21.15.92:40930) 2024-03-22T12:55:59.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.632+0000 7f67a259c640 1 -- 172.21.15.92:0/1135765966 learned_addr learned my addr 172.21.15.92:0/1135765966 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:55:59.632 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.633+0000 7f67a2d9d640 1 --2- 172.21.15.92:0/1135765966 >> v2:172.21.15.6:3300/0 conn(0x7f67a4157ae0 0x7f67a40cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.634+0000 7f67a259c640 1 -- 172.21.15.92:0/1135765966 >> v2:172.21.15.92:3300/0 conn(0x7f67a40c7680 msgr2=0x7f67a41575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:55:59.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.634+0000 7f67a259c640 1 --2- 172.21.15.92:0/1135765966 >> v2:172.21.15.92:3300/0 conn(0x7f67a40c7680 0x7f67a41575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.634+0000 7f67a259c640 1 -- 172.21.15.92:0/1135765966 >> v2:172.21.15.6:3300/0 conn(0x7f67a4157ae0 msgr2=0x7f67a40cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:55:59.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.634+0000 7f67a259c640 1 --2- 172.21.15.92:0/1135765966 >> v2:172.21.15.6:3300/0 conn(0x7f67a4157ae0 0x7f67a40cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.634+0000 7f67a259c640 1 -- 172.21.15.92:0/1135765966 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a406e730 con 0x7f67a414dc10 2024-03-22T12:55:59.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.634+0000 7f67a2d9d640 1 --2- 172.21.15.92:0/1135765966 >> v2:172.21.15.6:3300/0 conn(0x7f67a4157ae0 0x7f67a40cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T12:55:59.635 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.636+0000 7f67a259c640 1 --2- 172.21.15.92:0/1135765966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a414dc10 0x7f67a40c1be0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67a406d220 tx=0x7f679402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88a10c11e7bd0ae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:59.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.640+0000 7f67a159a640 1 -- 172.21.15.92:0/1135765966 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794035020 con 0x7f67a414dc10 2024-03-22T12:55:59.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.640+0000 7f67a159a640 1 -- 172.21.15.92:0/1135765966 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6794031870 con 0x7f67a414dc10 2024-03-22T12:55:59.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.640+0000 7f67a159a640 1 -- 172.21.15.92:0/1135765966 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794031c40 con 0x7f67a414dc10 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.640+0000 7f67a8edc640 1 -- 172.21.15.92:0/1135765966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a414dc10 msgr2=0x7f67a40c1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.640+0000 7f67a8edc640 1 --2- 172.21.15.92:0/1135765966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a414dc10 0x7f67a40c1be0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f67a406d220 tx=0x7f679402b910 comp rx=0 tx=0).stop 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.641+0000 7f67a8edc640 1 -- 172.21.15.92:0/1135765966 shutdown_connections 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.641+0000 7f67a8edc640 1 --2- 172.21.15.92:0/1135765966 >> v2:172.21.15.6:3300/0 conn(0x7f67a4157ae0 0x7f67a40cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.641+0000 7f67a8edc640 1 --2- 172.21.15.92:0/1135765966 >> v2:172.21.15.92:3300/0 conn(0x7f67a40c7680 0x7f67a41575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.641+0000 7f67a8edc640 1 --2- 172.21.15.92:0/1135765966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a414dc10 0x7f67a40c1be0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.641+0000 7f67a8edc640 1 -- 172.21.15.92:0/1135765966 >> 172.21.15.92:0/1135765966 conn(0x7f67a4070720 msgr2=0x7f67a4070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:55:59.640 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.641+0000 7f67a8edc640 1 -- 172.21.15.92:0/1135765966 shutdown_connections 2024-03-22T12:55:59.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.641+0000 7f67a8edc640 1 -- 172.21.15.92:0/1135765966 wait complete. 2024-03-22T12:55:59.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.642+0000 7f67a8edc640 1 Processor -- start 2024-03-22T12:55:59.641 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.642+0000 7f67a8edc640 1 -- start start 2024-03-22T12:55:59.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.643+0000 7f67a8edc640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a40c7680 0x7f67a4153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.643+0000 7f67a8edc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a414dc10 0x7f67a40c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.643+0000 7f67a8edc640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a4157ae0 0x7f67a40c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:55:59.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.643+0000 7f67a8edc640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a4073600 con 0x7f67a4157ae0 2024-03-22T12:55:59.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.643+0000 7f67a8edc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40c3080 con 0x7f67a414dc10 2024-03-22T12:55:59.642 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.643+0000 7f67a8edc640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a406f100 con 0x7f67a40c7680 2024-03-22T12:55:59.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.643+0000 7f67a259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a40c7680 0x7f67a4153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.644+0000 7f67a1d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a414dc10 0x7f67a40c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.644+0000 7f67a259c640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a40c7680 0x7f67a4153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:40946/0 (socket says 172.21.15.92:40946) 2024-03-22T12:55:59.643 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.644+0000 7f67a259c640 1 -- 172.21.15.92:0/785817120 learned_addr learned my addr 172.21.15.92:0/785817120 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:55:59.644 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.645+0000 7f67a2d9d640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a4157ae0 0x7f67a40c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:55:59.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.647+0000 7f67a259c640 1 -- 172.21.15.92:0/785817120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a414dc10 msgr2=0x7f67a40c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:59.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.647+0000 7f67a259c640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a414dc10 0x7f67a40c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.647+0000 7f67a259c640 1 -- 172.21.15.92:0/785817120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a4157ae0 msgr2=0x7f67a40c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:55:59.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.647+0000 7f67a259c640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a4157ae0 0x7f67a40c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:55:59.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.647+0000 7f67a259c640 1 -- 172.21.15.92:0/785817120 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a40ca110 con 0x7f67a40c7680 2024-03-22T12:55:59.648 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.649+0000 7f67a259c640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a40c7680 0x7f67a4153880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f67940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:55:59.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.653+0000 7f6792ffd640 1 -- 172.21.15.92:0/785817120 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794035040 con 0x7f67a40c7680 2024-03-22T12:55:59.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.653+0000 7f6792ffd640 1 -- 172.21.15.92:0/785817120 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6794030d00 con 0x7f67a40c7680 2024-03-22T12:55:59.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.653+0000 7f6792ffd640 1 -- 172.21.15.92:0/785817120 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940456a0 con 0x7f67a40c7680 2024-03-22T12:55:59.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.653+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a40ca3f0 con 0x7f67a40c7680 2024-03-22T12:55:59.652 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.653+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a414c390 con 0x7f67a40c7680 2024-03-22T12:55:59.653 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.654+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a40ca5c0 con 0x7f67a40c7680 2024-03-22T12:55:59.738 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:55:59.739+0000 7f0efea30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T12:55:59.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.756+0000 7f6792ffd640 1 -- 172.21.15.92:0/785817120 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6794038020 con 0x7f67a40c7680 2024-03-22T12:55:59.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.756+0000 7f6792ffd640 1 -- 172.21.15.92:0/785817120 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f679403a840 con 0x7f67a40c7680 2024-03-22T12:55:59.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.756+0000 7f6792ffd640 1 -- 172.21.15.92:0/785817120 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f679403aad0 con 0x7f67a40c7680 2024-03-22T12:55:59.850 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:55:59.851+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4", "id": 2} v 0) v1 -- 0x7f67a40757d0 con 0x7f67a40c7680 2024-03-22T12:55:59.953 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:55:59.953+0000 7f1d3c417040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T12:56:00.016 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.017+0000 7f6792ffd640 1 -- 172.21.15.92:0/785817120 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f679403acb0 con 0x7f67a40c7680 2024-03-22T12:56:00.016 INFO:teuthology.orchestra.run.smithi092.stdout:2 2024-03-22T12:56:00.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a40c7680 msgr2=0x7f67a4153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.017 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a40c7680 0x7f67a4153880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f67940046c0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 shutdown_connections 2024-03-22T12:56:00.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67a4157ae0 0x7f67a40c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a414dc10 0x7f67a40c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 --2- 172.21.15.92:0/785817120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67a40c7680 0x7f67a4153880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 >> 172.21.15.92:0/785817120 conn(0x7f67a4070720 msgr2=0x7f67a40cd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:00.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 shutdown_connections 2024-03-22T12:56:00.018 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.018+0000 7f67a8edc640 1 -- 172.21.15.92:0/785817120 wait complete. 2024-03-22T12:56:00.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 9ca78a23-26be-4a7a-8b71-0b17aece4c80 3 2024-03-22T12:56:00.123 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:00.123+0000 7f1d3c417040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T12:56:00.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.130+0000 7f09c253b640 1 Processor -- start 2024-03-22T12:56:00.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.130+0000 7f09c253b640 1 -- start start 2024-03-22T12:56:00.130 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.131+0000 7f09c253b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.131+0000 7f09c253b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09bc073280 0x7f09bc0cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.131+0000 7f09c253b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc154960 con 0x7f09bc154580 2024-03-22T12:56:00.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.131+0000 7f09c253b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc15d890 con 0x7f09bc073280 2024-03-22T12:56:00.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.131+0000 7f09c253b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc06f6f0 con 0x7f09bc15e5c0 2024-03-22T12:56:00.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.132+0000 7f09c0ad8640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09bc073280 0x7f09bc0cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.132+0000 7f09bb7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.132+0000 7f09bb7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc0cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:40952/0 (socket says 172.21.15.92:40952) 2024-03-22T12:56:00.132 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.132+0000 7f09bb7fe640 1 -- 172.21.15.92:0/1302519772 learned_addr learned my addr 172.21.15.92:0/1302519772 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:56:00.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.134+0000 7f09bb7fe640 1 -- 172.21.15.92:0/1302519772 >> v2:172.21.15.92:3300/0 conn(0x7f09bc073280 msgr2=0x7f09bc0cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:00.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.134+0000 7f09bb7fe640 1 --2- 172.21.15.92:0/1302519772 >> v2:172.21.15.92:3300/0 conn(0x7f09bc073280 0x7f09bc0cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.134+0000 7f09bb7fe640 1 -- 172.21.15.92:0/1302519772 >> v1:172.21.15.92:6789/0 conn(0x7f09bc154580 legacy=0x7f09bc15a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.134 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.134+0000 7f09bb7fe640 1 -- 172.21.15.92:0/1302519772 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc15dae0 con 0x7f09bc15e5c0 2024-03-22T12:56:00.136 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.137+0000 7f09bb7fe640 1 --2- 172.21.15.92:0/1302519772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc0cd070 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009870 tx=0x7f09b002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3aa28bd020b431d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:00.138 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.139+0000 7f09baffd640 1 -- 172.21.15.92:0/1302519772 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0034070 con 0x7f09bc15e5c0 2024-03-22T12:56:00.139 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.140+0000 7f09baffd640 1 -- 172.21.15.92:0/1302519772 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b0030840 con 0x7f09bc15e5c0 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09baffd640 1 -- 172.21.15.92:0/1302519772 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0030c10 con 0x7f09bc15e5c0 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 -- 172.21.15.92:0/1302519772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 msgr2=0x7f09bc0cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 --2- 172.21.15.92:0/1302519772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc0cd070 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009870 tx=0x7f09b002b980 comp rx=0 tx=0).stop 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 -- 172.21.15.92:0/1302519772 shutdown_connections 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 --2- 172.21.15.92:0/1302519772 >> v2:172.21.15.92:3300/0 conn(0x7f09bc073280 0x7f09bc0cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 --2- 172.21.15.92:0/1302519772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc0cd070 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 -- 172.21.15.92:0/1302519772 >> 172.21.15.92:0/1302519772 conn(0x7f09bc070720 msgr2=0x7f09bc070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 -- 172.21.15.92:0/1302519772 shutdown_connections 2024-03-22T12:56:00.141 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.141+0000 7f09c253b640 1 -- 172.21.15.92:0/1302519772 wait complete. 2024-03-22T12:56:00.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.143+0000 7f09c253b640 1 Processor -- start 2024-03-22T12:56:00.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.143+0000 7f09c253b640 1 -- start start 2024-03-22T12:56:00.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.143+0000 7f09c253b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09bc073280 0x7f09bc14d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c253b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc154580 0x7f09bc06c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c253b640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c253b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc06e6c0 con 0x7f09bc154580 2024-03-22T12:56:00.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c253b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc15e0a0 con 0x7f09bc073280 2024-03-22T12:56:00.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c253b640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc06e840 con 0x7f09bc15e5c0 2024-03-22T12:56:00.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c0ad8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09bbfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09bc073280 0x7f09bc14d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c0ad8640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:40968/0 (socket says 172.21.15.92:40968) 2024-03-22T12:56:00.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.144+0000 7f09c0ad8640 1 -- 172.21.15.92:0/974590970 learned_addr learned my addr 172.21.15.92:0/974590970 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:56:00.145 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.146+0000 7f09bb7fe640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc154580 0x7f09bc06c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.148+0000 7f09c0ad8640 1 -- 172.21.15.92:0/974590970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09bc073280 msgr2=0x7f09bc14d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.148+0000 7f09c0ad8640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09bc073280 0x7f09bc14d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.148+0000 7f09c0ad8640 1 -- 172.21.15.92:0/974590970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc154580 msgr2=0x7f09bc06c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.148+0000 7f09c0ad8640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc154580 0x7f09bc06c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.148+0000 7f09c0ad8640 1 -- 172.21.15.92:0/974590970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc159a80 con 0x7f09bc15e5c0 2024-03-22T12:56:00.147 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.148+0000 7f09bb7fe640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc154580 0x7f09bc06c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T12:56:00.149 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.150+0000 7f09c0ad8640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc06cbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f09b400efc0 tx=0x7f09b400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:00.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.153+0000 7f09b8ff9640 1 -- 172.21.15.92:0/974590970 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4018020 con 0x7f09bc15e5c0 2024-03-22T12:56:00.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.153+0000 7f09b8ff9640 1 -- 172.21.15.92:0/974590970 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b4014480 con 0x7f09bc15e5c0 2024-03-22T12:56:00.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.153+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc1f3fb0 con 0x7f09bc15e5c0 2024-03-22T12:56:00.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.154+0000 7f09b8ff9640 1 -- 172.21.15.92:0/974590970 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4014870 con 0x7f09bc15e5c0 2024-03-22T12:56:00.153 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.154+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc159350 con 0x7f09bc15e5c0 2024-03-22T12:56:00.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.155+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc0757d0 con 0x7f09bc15e5c0 2024-03-22T12:56:00.154 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.156+0000 7f09b8ff9640 1 -- 172.21.15.92:0/974590970 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f09b4005d10 con 0x7f09bc15e5c0 2024-03-22T12:56:00.155 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.156+0000 7f09b8ff9640 1 -- 172.21.15.92:0/974590970 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f09b4014da0 con 0x7f09bc15e5c0 2024-03-22T12:56:00.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.259+0000 7f09b8ff9640 1 -- 172.21.15.92:0/974590970 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f09b4022030 con 0x7f09bc15e5c0 2024-03-22T12:56:00.273 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:00.273+0000 7f1d3c417040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T12:56:00.329 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:00.329+0000 7f1d3c417040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T12:56:00.332 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.333+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9ca78a23-26be-4a7a-8b71-0b17aece4c80", "id": 3} v 0) v1 -- 0x7f09bc073060 con 0x7f09bc15e5c0 2024-03-22T12:56:00.463 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:00.463+0000 7f1d3c417040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T12:56:00.519 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.520+0000 7f09b8ff9640 1 -- 172.21.15.92:0/974590970 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9ca78a23-26be-4a7a-8b71-0b17aece4c80", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f09b4010040 con 0x7f09bc15e5c0 2024-03-22T12:56:00.520 INFO:teuthology.orchestra.run.smithi092.stdout:3 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 msgr2=0x7f09bc06cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc06cbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f09b400efc0 tx=0x7f09b400c490 comp rx=0 tx=0).stop 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 shutdown_connections 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09bc15e5c0 0x7f09bc06cbe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc154580 0x7f09bc06c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 --2- 172.21.15.92:0/974590970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09bc073280 0x7f09bc14d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 >> 172.21.15.92:0/974590970 conn(0x7f09bc070720 msgr2=0x7f09bc0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 shutdown_connections 2024-03-22T12:56:00.521 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.521+0000 7f09c253b640 1 -- 172.21.15.92:0/974590970 wait complete. 2024-03-22T12:56:00.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new f1ba8f52-e2b1-4dc2-a330-27f2b026e60d 4 2024-03-22T12:56:00.580 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:00.580+0000 7f1d3c417040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T12:56:00.593 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:56:00.594+0000 7f0efea30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T12:56:00.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.635+0000 7fe3723c4640 1 Processor -- start 2024-03-22T12:56:00.634 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.635+0000 7fe3723c4640 1 -- start start 2024-03-22T12:56:00.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.636+0000 7fe3723c4640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe36c0c7550 0x7fe36c070b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.636+0000 7fe3723c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0c2ad0 con 0x7fe36c1511d0 2024-03-22T12:56:00.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.636+0000 7fe3723c4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0c7970 con 0x7fe36c071090 2024-03-22T12:56:00.636 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.636+0000 7fe3723c4640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c156fb0 con 0x7fe36c0c7550 2024-03-22T12:56:00.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.637+0000 7fe370961640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe36c071090 0x7fe36c0ccac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:54458/0 (socket says 172.21.15.92:54458) 2024-03-22T12:56:00.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.637+0000 7fe370961640 1 -- 172.21.15.92:0/3735255924 learned_addr learned my addr 172.21.15.92:0/3735255924 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-22T12:56:00.637 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.638+0000 7fe36b7fe640 1 --2- 172.21.15.92:0/3735255924 >> v2:172.21.15.6:3300/0 conn(0x7fe36c0c7550 0x7fe36c070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.640+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415968054 0 0) 0x7fe36c0c7970 con 0x7fe36c071090 2024-03-22T12:56:00.639 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.640+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe354003520 con 0x7fe36c071090 2024-03-22T12:56:00.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897785908 0 0) 0x7fe354003520 con 0x7fe36c071090 2024-03-22T12:56:00.645 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 >> v2:172.21.15.6:3300/0 conn(0x7fe36c0c7550 msgr2=0x7fe36c070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 --2- 172.21.15.92:0/3735255924 >> v2:172.21.15.6:3300/0 conn(0x7fe36c0c7550 0x7fe36c070b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 >> v1:172.21.15.6:6789/0 conn(0x7fe36c1511d0 legacy=0x7fe36c0c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0c1f50 con 0x7fe36c071090 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7fe3640019f0 con 0x7fe36c071090 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 >> v1:172.21.15.92:6789/0 conn(0x7fe36c071090 legacy=0x7fe36c0ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe354003a00 0x7fe354003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.645+0000 7fe36affd640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe354004800 0x7fe354004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.646+0000 7fe36affd640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3540051c0 0x7fe35400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.646+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c156fb0 con 0x7fe3540051c0 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.646+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35400e2d0 con 0x7fe354004800 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.646+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35400e4d0 con 0x7fe354003a00 2024-03-22T12:56:00.646 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.646+0000 7fe36bfff640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe354003a00 0x7fe354003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.647 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.648+0000 7fe36b7fe640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe354004800 0x7fe354004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.655+0000 7fe36bfff640 1 -- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe354004800 msgr2=0x7fe354004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:00.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.655+0000 7fe36bfff640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe354004800 0x7fe354004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.655+0000 7fe36bfff640 1 -- 172.21.15.92:0/3735255924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3540051c0 msgr2=0x7fe35400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:00.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.655+0000 7fe36bfff640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3540051c0 0x7fe35400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.655+0000 7fe36bfff640 1 -- 172.21.15.92:0/3735255924 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354003520 con 0x7fe354003a00 2024-03-22T12:56:00.655 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.655+0000 7fe370961640 1 -- 172.21.15.92:0/3735255924 reap_dead start 2024-03-22T12:56:00.656 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.657+0000 7fe36bfff640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe354003a00 0x7fe354003e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe360001750 tx=0x7fe354005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dacda0da78da925 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.661+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360001370 con 0x7fe354003a00 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.661+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3600052d0 con 0x7fe354003a00 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.661+0000 7fe36affd640 1 -- 172.21.15.92:0/3735255924 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360003830 con 0x7fe354003a00 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.662+0000 7fe3723c4640 1 -- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe354003a00 msgr2=0x7fe354003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.662+0000 7fe3723c4640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe354003a00 0x7fe354003e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe360001750 tx=0x7fe354005830 comp rx=0 tx=0).stop 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.662+0000 7fe3723c4640 1 -- 172.21.15.92:0/3735255924 shutdown_connections 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.662+0000 7fe3723c4640 1 --2- 172.21.15.92:0/3735255924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe354003a00 0x7fe354003e20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.662+0000 7fe3723c4640 1 -- 172.21.15.92:0/3735255924 >> 172.21.15.92:0/3735255924 conn(0x7fe36c0674d0 msgr2=0x7fe36c072970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.662+0000 7fe3723c4640 1 -- 172.21.15.92:0/3735255924 shutdown_connections 2024-03-22T12:56:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.662+0000 7fe3723c4640 1 -- 172.21.15.92:0/3735255924 wait complete. 2024-03-22T12:56:00.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.663+0000 7fe3723c4640 1 Processor -- start 2024-03-22T12:56:00.662 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.663+0000 7fe3723c4640 1 -- start start 2024-03-22T12:56:00.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.664+0000 7fe3723c4640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe354003a00 0x7fe36c14c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.664+0000 7fe3723c4640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe36c14b130 0x7fe36c0cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.664+0000 7fe3723c4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe36c14c850 0x7fe36c14d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.664+0000 7fe3723c4640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c27f0 con 0x7fe354003a00 2024-03-22T12:56:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.664+0000 7fe3723c4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c2510 con 0x7fe36c14c850 2024-03-22T12:56:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.664+0000 7fe3723c4640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c7c00 con 0x7fe36c14b130 2024-03-22T12:56:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.664+0000 7fe36b7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe36c14b130 0x7fe36c0cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.665+0000 7fe370961640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe36c14c850 0x7fe36c14d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.665+0000 7fe36b7fe640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe36c14b130 0x7fe36c0cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:40988/0 (socket says 172.21.15.92:40988) 2024-03-22T12:56:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.665+0000 7fe36b7fe640 1 -- 172.21.15.92:0/3957662831 learned_addr learned my addr 172.21.15.92:0/3957662831 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:56:00.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.666+0000 7fe36bfff640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe354003a00 0x7fe36c14c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:00.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.668+0000 7fe36b7fe640 1 -- 172.21.15.92:0/3957662831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe36c14c850 msgr2=0x7fe36c14d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.668+0000 7fe36b7fe640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe36c14c850 0x7fe36c14d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.668+0000 7fe36b7fe640 1 -- 172.21.15.92:0/3957662831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe354003a00 msgr2=0x7fe36c14c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:00.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.668+0000 7fe36b7fe640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe354003a00 0x7fe36c14c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:00.667 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.668+0000 7fe36b7fe640 1 -- 172.21.15.92:0/3957662831 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c15a3b0 con 0x7fe36c14b130 2024-03-22T12:56:00.669 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.670+0000 7fe36b7fe640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe36c14b130 0x7fe36c0cb4c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe35c011aa0 tx=0x7fe35c011f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:00.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.673+0000 7fe368ff9640 1 -- 172.21.15.92:0/3957662831 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c00fb20 con 0x7fe36c14b130 2024-03-22T12:56:00.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.673+0000 7fe368ff9640 1 -- 172.21.15.92:0/3957662831 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c00fcc0 con 0x7fe36c14b130 2024-03-22T12:56:00.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.673+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c1f3110 con 0x7fe36c14b130 2024-03-22T12:56:00.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.673+0000 7fe368ff9640 1 -- 172.21.15.92:0/3957662831 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c014660 con 0x7fe36c14b130 2024-03-22T12:56:00.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.673+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c1f36d0 con 0x7fe36c14b130 2024-03-22T12:56:00.673 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.674+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c0757d0 con 0x7fe36c14b130 2024-03-22T12:56:00.765 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:00.765+0000 7f1d3c417040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T12:56:00.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.777+0000 7fe368ff9640 1 -- 172.21.15.92:0/3957662831 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe35c012040 con 0x7fe36c14b130 2024-03-22T12:56:00.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.777+0000 7fe368ff9640 1 -- 172.21.15.92:0/3957662831 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fe35c01dbd0 con 0x7fe36c14b130 2024-03-22T12:56:00.777 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.777+0000 7fe368ff9640 1 -- 172.21.15.92:0/3957662831 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe35c02b020 con 0x7fe36c14b130 2024-03-22T12:56:00.817 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:00.818+0000 7f1d3c417040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T12:56:00.867 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:00.868+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f1ba8f52-e2b1-4dc2-a330-27f2b026e60d", "id": 4} v 0) v1 -- 0x7fe36c1562d0 con 0x7fe36c14b130 2024-03-22T12:56:00.893 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:56:00.893+0000 7f0efea30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T12:56:00.946 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:56:00.947+0000 7f0efea30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T12:56:01.002 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:01.002+0000 7f1d3c417040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T12:56:01.063 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:56:01.064+0000 7f0efea30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T12:56:01.142 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.142+0000 7fe368ff9640 1 -- 172.21.15.92:0/3957662831 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f1ba8f52-e2b1-4dc2-a330-27f2b026e60d", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe35c019030 con 0x7fe36c14b130 2024-03-22T12:56:01.142 INFO:teuthology.orchestra.run.smithi092.stdout:4 2024-03-22T12:56:01.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.143+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe36c14b130 msgr2=0x7fe36c0cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:01.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.143+0000 7fe3723c4640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe36c14b130 0x7fe36c0cb4c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe35c011aa0 tx=0x7fe35c011f60 comp rx=0 tx=0).stop 2024-03-22T12:56:01.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.143+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 shutdown_connections 2024-03-22T12:56:01.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.143+0000 7fe3723c4640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe36c14c850 0x7fe36c14d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.143 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.143+0000 7fe3723c4640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe36c14b130 0x7fe36c0cb4c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.143+0000 7fe3723c4640 1 --2- 172.21.15.92:0/3957662831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe354003a00 0x7fe36c14c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.143+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 >> 172.21.15.92:0/3957662831 conn(0x7fe36c0674d0 msgr2=0x7fe36c0caa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:01.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.144+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 shutdown_connections 2024-03-22T12:56:01.144 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.144+0000 7fe3723c4640 1 -- 172.21.15.92:0/3957662831 wait complete. 2024-03-22T12:56:01.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 69f533c6-b98b-40bd-ac72-fea32148cb6a 5 2024-03-22T12:56:01.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.253+0000 7f271e9ab640 1 Processor -- start 2024-03-22T12:56:01.252 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.253+0000 7f271e9ab640 1 -- start start 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.253+0000 7f271e9ab640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cf660 0x7f27180cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.253+0000 7f271e9ab640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f27180ceae0 0x7f27180cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.253+0000 7f271e9ab640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27180ccf20 con 0x7f27180cdf00 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.253+0000 7f271e9ab640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27180cfa40 con 0x7f27180ceae0 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.253+0000 7f271e9ab640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271815b470 con 0x7f27180cf660 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.254+0000 7f271cf48640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f27180ceae0 0x7f27180cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.254+0000 7f2717fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cf660 0x7f27180cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.254+0000 7f2717fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cf660 0x7f27180cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:41004/0 (socket says 172.21.15.92:41004) 2024-03-22T12:56:01.254 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.254+0000 7f2717fff640 1 -- 172.21.15.92:0/368656357 learned_addr learned my addr 172.21.15.92:0/368656357 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:56:01.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.257+0000 7f2717fff640 1 -- 172.21.15.92:0/368656357 >> v2:172.21.15.92:3300/0 conn(0x7f27180ceae0 msgr2=0x7f27180cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:01.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.257+0000 7f2717fff640 1 --2- 172.21.15.92:0/368656357 >> v2:172.21.15.92:3300/0 conn(0x7f27180ceae0 0x7f27180cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.257+0000 7f2717fff640 1 -- 172.21.15.92:0/368656357 >> v1:172.21.15.92:6789/0 conn(0x7f27180cdf00 legacy=0x7f27180ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:01.256 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.257+0000 7f2717fff640 1 -- 172.21.15.92:0/368656357 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271815a6b0 con 0x7f27180cf660 2024-03-22T12:56:01.258 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.259+0000 7f2717fff640 1 --2- 172.21.15.92:0/368656357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cf660 0x7f27180cd9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f270c009870 tx=0x7f270c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee552a7bf95f8273 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:01.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.262+0000 7f2716ffd640 1 -- 172.21.15.92:0/368656357 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c02f840 con 0x7f27180cf660 2024-03-22T12:56:01.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f2716ffd640 1 -- 172.21.15.92:0/368656357 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c02f9e0 con 0x7f27180cf660 2024-03-22T12:56:01.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f2716ffd640 1 -- 172.21.15.92:0/368656357 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c02fdb0 con 0x7f27180cf660 2024-03-22T12:56:01.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 -- 172.21.15.92:0/368656357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cf660 msgr2=0x7f27180cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:01.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 --2- 172.21.15.92:0/368656357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cf660 0x7f27180cd9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f270c009870 tx=0x7f270c02ba20 comp rx=0 tx=0).stop 2024-03-22T12:56:01.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 -- 172.21.15.92:0/368656357 shutdown_connections 2024-03-22T12:56:01.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 --2- 172.21.15.92:0/368656357 >> v2:172.21.15.92:3300/0 conn(0x7f27180ceae0 0x7f27180cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 --2- 172.21.15.92:0/368656357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cf660 0x7f27180cd9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 -- 172.21.15.92:0/368656357 >> 172.21.15.92:0/368656357 conn(0x7f27180b16d0 msgr2=0x7f27180b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:01.263 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 -- 172.21.15.92:0/368656357 shutdown_connections 2024-03-22T12:56:01.264 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.263+0000 7f271e9ab640 1 -- 172.21.15.92:0/368656357 wait complete. 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.264+0000 7f271e9ab640 1 Processor -- start 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.265+0000 7f271e9ab640 1 -- start start 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.265+0000 7f271e9ab640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cdf00 0x7f27181f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.265+0000 7f271e9ab640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27180ceae0 0x7f27181519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.265+0000 7f271e9ab640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27180cf660 0x7f27180c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.265+0000 7f271e9ab640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271815ac70 con 0x7f27180ceae0 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.265+0000 7f271e9ab640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271815afe0 con 0x7f27180cf660 2024-03-22T12:56:01.265 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.265+0000 7f271e9ab640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27180cac50 con 0x7f27180cdf00 2024-03-22T12:56:01.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.266+0000 7f2717fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cdf00 0x7f27181f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:01.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.266+0000 7f271cf48640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27180cf660 0x7f27180c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:01.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.266+0000 7f2717fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cdf00 0x7f27181f0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:41018/0 (socket says 172.21.15.92:41018) 2024-03-22T12:56:01.266 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.266+0000 7f2717fff640 1 -- 172.21.15.92:0/3319262491 learned_addr learned my addr 172.21.15.92:0/3319262491 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:56:01.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.269+0000 7f271cf48640 1 -- 172.21.15.92:0/3319262491 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cdf00 msgr2=0x7f27181f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:01.268 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.269+0000 7f271cf48640 1 --2- 172.21.15.92:0/3319262491 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cdf00 0x7f27181f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.269+0000 7f271cf48640 1 -- 172.21.15.92:0/3319262491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27180ceae0 msgr2=0x7f27181519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:01.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.269+0000 7f271cf48640 1 --2- 172.21.15.92:0/3319262491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27180ceae0 0x7f27181519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.269 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.269+0000 7f271cf48640 1 -- 172.21.15.92:0/3319262491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27180c6690 con 0x7f27180cf660 2024-03-22T12:56:01.271 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.272+0000 7f271cf48640 1 --2- 172.21.15.92:0/3319262491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27180cf660 0x7f27180c7d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2710009d90 tx=0x7f271000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:01.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.275+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710018020 con 0x7f27180cf660 2024-03-22T12:56:01.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.275+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710014480 con 0x7f27180cf660 2024-03-22T12:56:01.274 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.275+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27180c6950 con 0x7f27180cf660 2024-03-22T12:56:01.275 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.275+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710014870 con 0x7f27180cf660 2024-03-22T12:56:01.275 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.275+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27181f33f0 con 0x7f27180cf660 2024-03-22T12:56:01.275 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.276+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27180757d0 con 0x7f27180cf660 2024-03-22T12:56:01.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.279+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2710005d10 con 0x7f27180cf660 2024-03-22T12:56:01.278 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.279+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f271001d980 con 0x7f27180cf660 2024-03-22T12:56:01.370 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.371+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2710022020 con 0x7f27180cf660 2024-03-22T12:56:01.442 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.443+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "69f533c6-b98b-40bd-ac72-fea32148cb6a", "id": 5} v 0) v1 -- 0x7f27180cb0a0 con 0x7f27180cf660 2024-03-22T12:56:01.550 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.551+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 2) v1 ==== 45093+0+0 (secure 0 0 0) 0x7f271001b020 con 0x7f27180cf660 2024-03-22T12:56:01.944 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.945+0000 7f2714ff9640 1 -- 172.21.15.92:0/3319262491 <== mon.1 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69f533c6-b98b-40bd-ac72-fea32148cb6a", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2710010040 con 0x7f27180cf660 2024-03-22T12:56:01.945 INFO:teuthology.orchestra.run.smithi092.stdout:5 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27180cf660 msgr2=0x7f27180c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 --2- 172.21.15.92:0/3319262491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27180cf660 0x7f27180c7d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2710009d90 tx=0x7f271000c490 comp rx=0 tx=0).stop 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 shutdown_connections 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 --2- 172.21.15.92:0/3319262491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27180cf660 0x7f27180c7d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 --2- 172.21.15.92:0/3319262491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27180ceae0 0x7f27181519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 --2- 172.21.15.92:0/3319262491 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f27180cdf00 0x7f27181f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 >> 172.21.15.92:0/3319262491 conn(0x7f27180b16d0 msgr2=0x7f27180cc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 shutdown_connections 2024-03-22T12:56:01.946 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:01.946+0000 7f271e9ab640 1 -- 172.21.15.92:0/3319262491 wait complete. 2024-03-22T12:56:01.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 5b75948a-a510-4418-b902-87cd6d84f34b 6 2024-03-22T12:56:02.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.091+0000 7fee21e80640 1 Processor -- start 2024-03-22T12:56:02.090 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.091+0000 7fee21e80640 1 -- start start 2024-03-22T12:56:02.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.091+0000 7fee21e80640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:02.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.091+0000 7fee21e80640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c0ccf20 con 0x7fee1c0cdf00 2024-03-22T12:56:02.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.091+0000 7fee21e80640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c15a6b0 con 0x7fee1c0ceae0 2024-03-22T12:56:02.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.091+0000 7fee21e80640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c0cfa40 con 0x7fee1c0cf660 2024-03-22T12:56:02.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.092+0000 7fee1bfff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fee1c0ceae0 0x7fee1c0cc3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:54478/0 (socket says 172.21.15.92:54478) 2024-03-22T12:56:02.091 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.092+0000 7fee1bfff640 1 -- 172.21.15.92:0/2087939058 learned_addr learned my addr 172.21.15.92:0/2087939058 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-22T12:56:02.092 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.092+0000 7fee1b7fe640 1 --2- 172.21.15.92:0/2087939058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:02.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.094+0000 7fee1b7fe640 1 -- 172.21.15.92:0/2087939058 >> v1:172.21.15.92:6789/0 conn(0x7fee1c0ceae0 legacy=0x7fee1c0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:02.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.094+0000 7fee1b7fe640 1 -- 172.21.15.92:0/2087939058 >> v1:172.21.15.6:6789/0 conn(0x7fee1c0cdf00 legacy=0x7fee1c0ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:02.094 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.095+0000 7fee1b7fe640 1 -- 172.21.15.92:0/2087939058 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c15a990 con 0x7fee1c0cf660 2024-03-22T12:56:02.095 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.096+0000 7fee1b7fe640 1 --2- 172.21.15.92:0/2087939058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c0cd9c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0097d0 tx=0x7fee0c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762dc01b12891a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:02.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee1a7fc640 1 -- 172.21.15.92:0/2087939058 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c035020 con 0x7fee1c0cf660 2024-03-22T12:56:02.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee1a7fc640 1 -- 172.21.15.92:0/2087939058 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c031890 con 0x7fee1c0cf660 2024-03-22T12:56:02.100 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee1a7fc640 1 -- 172.21.15.92:0/2087939058 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c031c60 con 0x7fee1c0cf660 2024-03-22T12:56:02.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee21e80640 1 -- 172.21.15.92:0/2087939058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 msgr2=0x7fee1c0cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:02.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee21e80640 1 --2- 172.21.15.92:0/2087939058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c0cd9c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0097d0 tx=0x7fee0c02b930 comp rx=0 tx=0).stop 2024-03-22T12:56:02.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee21e80640 1 -- 172.21.15.92:0/2087939058 shutdown_connections 2024-03-22T12:56:02.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee21e80640 1 --2- 172.21.15.92:0/2087939058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c0cd9c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:02.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.101+0000 7fee21e80640 1 -- 172.21.15.92:0/2087939058 >> 172.21.15.92:0/2087939058 conn(0x7fee1c0b16d0 msgr2=0x7fee1c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:02.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.102+0000 7fee21e80640 1 -- 172.21.15.92:0/2087939058 shutdown_connections 2024-03-22T12:56:02.101 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.102+0000 7fee21e80640 1 -- 172.21.15.92:0/2087939058 wait complete. 2024-03-22T12:56:02.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.103+0000 7fee21e80640 1 Processor -- start 2024-03-22T12:56:02.102 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.103+0000 7fee21e80640 1 -- start start 2024-03-22T12:56:02.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee21e80640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee1c0cdf00 0x7fee1c159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:02.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee21e80640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee1c0ceae0 0x7fee1c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:02.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee21e80640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c1524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:02.103 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee21e80640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c15afe0 con 0x7fee1c0cdf00 2024-03-22T12:56:02.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee21e80640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c15b470 con 0x7fee1c0ceae0 2024-03-22T12:56:02.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee21e80640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c0cac50 con 0x7fee1c0cf660 2024-03-22T12:56:02.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee1affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee1c0ceae0 0x7fee1c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:02.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.104+0000 7fee1bfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c1524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:02.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.105+0000 7fee1affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee1c0ceae0 0x7fee1c151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:39156/0 (socket says 172.21.15.92:39156) 2024-03-22T12:56:02.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.105+0000 7fee1bfff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c1524f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:41074/0 (socket says 172.21.15.92:41074) 2024-03-22T12:56:02.104 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.105+0000 7fee1affd640 1 -- 172.21.15.92:0/2431968384 learned_addr learned my addr 172.21.15.92:0/2431968384 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:56:02.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.108+0000 7fee1bfff640 1 -- 172.21.15.92:0/2431968384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee1c0ceae0 msgr2=0x7fee1c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:02.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.108+0000 7fee1bfff640 1 --2- 172.21.15.92:0/2431968384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee1c0ceae0 0x7fee1c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:02.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.108+0000 7fee1bfff640 1 -- 172.21.15.92:0/2431968384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee1c0cdf00 msgr2=0x7fee1c159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:02.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.108+0000 7fee1bfff640 1 --2- 172.21.15.92:0/2431968384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee1c0cdf00 0x7fee1c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:02.107 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.108+0000 7fee1bfff640 1 -- 172.21.15.92:0/2431968384 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c1f0500 con 0x7fee1c0cf660 2024-03-22T12:56:02.109 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.110+0000 7fee1bfff640 1 --2- 172.21.15.92:0/2431968384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c1524f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fee1400caa0 tx=0x7fee1400cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:02.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.113+0000 7fee03fff640 1 -- 172.21.15.92:0/2431968384 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14017070 con 0x7fee1c0cf660 2024-03-22T12:56:02.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.113+0000 7fee03fff640 1 -- 172.21.15.92:0/2431968384 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1400ab40 con 0x7fee1c0cf660 2024-03-22T12:56:02.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.113+0000 7fee03fff640 1 -- 172.21.15.92:0/2431968384 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14012400 con 0x7fee1c0cf660 2024-03-22T12:56:02.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.113+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c1e6370 con 0x7fee1c0cf660 2024-03-22T12:56:02.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.114+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c0c3f90 con 0x7fee1c0cf660 2024-03-22T12:56:02.113 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.114+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c0757d0 con 0x7fee1c0cf660 2024-03-22T12:56:02.128 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.129+0000 7fee03fff640 1 -- 172.21.15.92:0/2431968384 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 2) v1 ==== 45093+0+0 (secure 0 0 0) 0x7fee1401a070 con 0x7fee1c0cf660 2024-03-22T12:56:02.131 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.132+0000 7fee03fff640 1 -- 172.21.15.92:0/2431968384 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fee14012d00 con 0x7fee1c0cf660 2024-03-22T12:56:02.157 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:02.157+0000 7f1d3c417040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T12:56:02.288 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.289+0000 7fee03fff640 1 -- 172.21.15.92:0/2431968384 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fee140125a0 con 0x7fee1c0cf660 2024-03-22T12:56:02.360 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.360+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5b75948a-a510-4418-b902-87cd6d84f34b", "id": 6} v 0) v1 -- 0x7fee1c0cb0a0 con 0x7fee1c0cf660 2024-03-22T12:56:02.490 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:02.490+0000 7f1d3c417040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T12:56:02.553 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:02.553+0000 7f1d3c417040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T12:56:02.690 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-22T12:56:02.691+0000 7f1d3c417040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T12:56:02.922 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.922+0000 7fee03fff640 1 -- 172.21.15.92:0/2431968384 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5b75948a-a510-4418-b902-87cd6d84f34b", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fee140127e0 con 0x7fee1c0cf660 2024-03-22T12:56:02.922 INFO:teuthology.orchestra.run.smithi092.stdout:6 2024-03-22T12:56:02.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.923+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 msgr2=0x7fee1c1524f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:02.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.923+0000 7fee21e80640 1 --2- 172.21.15.92:0/2431968384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c1524f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fee1400caa0 tx=0x7fee1400cf60 comp rx=0 tx=0).stop 2024-03-22T12:56:02.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.923+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 shutdown_connections 2024-03-22T12:56:02.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.923+0000 7fee21e80640 1 --2- 172.21.15.92:0/2431968384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee1c0cf660 0x7fee1c1524f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:02.923 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.923+0000 7fee21e80640 1 --2- 172.21.15.92:0/2431968384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee1c0ceae0 0x7fee1c151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:02.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.923+0000 7fee21e80640 1 --2- 172.21.15.92:0/2431968384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee1c0cdf00 0x7fee1c159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:02.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.923+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 >> 172.21.15.92:0/2431968384 conn(0x7fee1c0b16d0 msgr2=0x7fee1c1542b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:02.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.924+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 shutdown_connections 2024-03-22T12:56:02.924 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:02.924+0000 7fee21e80640 1 -- 172.21.15.92:0/2431968384 wait complete. 2024-03-22T12:56:02.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new e27c5d29-4343-4274-9e28-9d3cd5c3f32b 7 2024-03-22T12:56:03.034 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.034+0000 7fe9baeac640 1 Processor -- start 2024-03-22T12:56:03.034 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.034+0000 7fe9baeac640 1 -- start start 2024-03-22T12:56:03.034 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.035+0000 7fe9baeac640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe9b4154200 0x7fe9b415ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.034 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.035+0000 7fe9baeac640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b40c4650 con 0x7fe9b40c8860 2024-03-22T12:56:03.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.035+0000 7fe9baeac640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b40c90e0 con 0x7fe9b40c8cb0 2024-03-22T12:56:03.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.035+0000 7fe9baeac640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b406f440 con 0x7fe9b4154200 2024-03-22T12:56:03.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.036+0000 7fe9b9449640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe9b40c8cb0 0x7fe9b406c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:54484/0 (socket says 172.21.15.92:54484) 2024-03-22T12:56:03.035 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.036+0000 7fe9b9449640 1 -- 172.21.15.92:0/2425968470 learned_addr learned my addr 172.21.15.92:0/2425968470 (peer_addr_for_me v1:172.21.15.92:0/0) 2024-03-22T12:56:03.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.038+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379391328 0 0) 0x7fe9b40c90e0 con 0x7fe9b40c8cb0 2024-03-22T12:56:03.037 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.038+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe99c003520 con 0x7fe9b40c8cb0 2024-03-22T12:56:03.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4113177003 0 0) 0x7fe99c003520 con 0x7fe9b40c8cb0 2024-03-22T12:56:03.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 >> v2:172.21.15.6:3300/0 conn(0x7fe9b4154200 msgr2=0x7fe9b415ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:03.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 --2- 172.21.15.92:0/2425968470 >> v2:172.21.15.6:3300/0 conn(0x7fe9b4154200 0x7fe9b415ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 >> v1:172.21.15.6:6789/0 conn(0x7fe9b40c8860 legacy=0x7fe9b40c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:03.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b406f690 con 0x7fe9b40c8cb0 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7fe9ac0019f0 con 0x7fe9b40c8cb0 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 >> v1:172.21.15.92:6789/0 conn(0x7fe9b40c8cb0 legacy=0x7fe9b406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe99c003a00 0x7fe99c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c004800 0x7fe99c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe99c0052e0 0x7fe99c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b406f440 con 0x7fe99c0052e0 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c00c220 con 0x7fe99c004800 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.041+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c00c420 con 0x7fe99c003a00 2024-03-22T12:56:03.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.042+0000 7fe9b8c48640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe99c003a00 0x7fe99c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:03.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.043+0000 7fe9b3fff640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c004800 0x7fe99c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:03.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.045+0000 7fe9b8c48640 1 -- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c004800 msgr2=0x7fe99c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:03.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.045+0000 7fe9b8c48640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c004800 0x7fe99c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.045+0000 7fe9b8c48640 1 -- 172.21.15.92:0/2425968470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe99c0052e0 msgr2=0x7fe99c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:03.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.045+0000 7fe9b8c48640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe99c0052e0 0x7fe99c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.045+0000 7fe9b8c48640 1 -- 172.21.15.92:0/2425968470 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c003520 con 0x7fe99c003a00 2024-03-22T12:56:03.044 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.045+0000 7fe9b9449640 1 -- 172.21.15.92:0/2425968470 reap_dead start 2024-03-22T12:56:03.046 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.047+0000 7fe9b8c48640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe99c003a00 0x7fe99c003e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4006fd0 tx=0x7fe99c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c1d3ed6579db50f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:03.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.050+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4037070 con 0x7fe99c003a00 2024-03-22T12:56:03.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.050+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a4032620 con 0x7fe99c003a00 2024-03-22T12:56:03.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.050+0000 7fe9b37fe640 1 -- 172.21.15.92:0/2425968470 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4009460 con 0x7fe99c003a00 2024-03-22T12:56:03.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.051+0000 7fe9baeac640 1 -- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe99c003a00 msgr2=0x7fe99c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:03.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.051+0000 7fe9baeac640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe99c003a00 0x7fe99c003e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4006fd0 tx=0x7fe99c005830 comp rx=0 tx=0).stop 2024-03-22T12:56:03.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.051+0000 7fe9baeac640 1 -- 172.21.15.92:0/2425968470 shutdown_connections 2024-03-22T12:56:03.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.051+0000 7fe9baeac640 1 --2- 172.21.15.92:0/2425968470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe99c003a00 0x7fe99c003e00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.051+0000 7fe9baeac640 1 -- 172.21.15.92:0/2425968470 >> 172.21.15.92:0/2425968470 conn(0x7fe9b4071410 msgr2=0x7fe9b40717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:03.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.051+0000 7fe9baeac640 1 -- 172.21.15.92:0/2425968470 shutdown_connections 2024-03-22T12:56:03.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.051+0000 7fe9baeac640 1 -- 172.21.15.92:0/2425968470 wait complete. 2024-03-22T12:56:03.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.052+0000 7fe9baeac640 1 Processor -- start 2024-03-22T12:56:03.052 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.052+0000 7fe9baeac640 1 -- start start 2024-03-22T12:56:03.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.053+0000 7fe9baeac640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c003a00 0x7fe9b414cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.053+0000 7fe9baeac640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9b414dc40 0x7fe9b415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.053+0000 7fe9baeac640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9b414ef60 0x7fe9b4148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.053+0000 7fe9baeac640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b406fc50 con 0x7fe9b414ef60 2024-03-22T12:56:03.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.053+0000 7fe9baeac640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b406de00 con 0x7fe99c003a00 2024-03-22T12:56:03.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.053+0000 7fe9baeac640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b406ff30 con 0x7fe9b414dc40 2024-03-22T12:56:03.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.054+0000 7fe9b8c48640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c003a00 0x7fe9b414cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:03.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.054+0000 7fe9b3fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9b414dc40 0x7fe9b415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:03.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.054+0000 7fe9b3fff640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9b414dc40 0x7fe9b415a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:41106/0 (socket says 172.21.15.92:41106) 2024-03-22T12:56:03.054 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.054+0000 7fe9b3fff640 1 -- 172.21.15.92:0/3931757567 learned_addr learned my addr 172.21.15.92:0/3931757567 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-03-22T12:56:03.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.058+0000 7fe9b3fff640 1 -- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c003a00 msgr2=0x7fe9b414cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:03.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.058+0000 7fe9b3fff640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c003a00 0x7fe9b414cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.059+0000 7fe9b3fff640 1 -- 172.21.15.92:0/3931757567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9b414ef60 msgr2=0x7fe9b4148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:03.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.059+0000 7fe9b3fff640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9b414ef60 0x7fe9b4148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.058 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.059+0000 7fe9b3fff640 1 -- 172.21.15.92:0/3931757567 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b4157870 con 0x7fe9b414dc40 2024-03-22T12:56:03.060 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.061+0000 7fe9b3fff640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9b414dc40 0x7fe9b415a520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800e510 tx=0x7fe9a8001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:03.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.064+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a80052e0 con 0x7fe9b414dc40 2024-03-22T12:56:03.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.064+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a8018070 con 0x7fe9b414dc40 2024-03-22T12:56:03.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.064+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8003830 con 0x7fe9b414dc40 2024-03-22T12:56:03.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.064+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b41f3ae0 con 0x7fe9b414dc40 2024-03-22T12:56:03.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.064+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b41f40a0 con 0x7fe9b414dc40 2024-03-22T12:56:03.064 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.065+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b40757d0 con 0x7fe9b414dc40 2024-03-22T12:56:03.075 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.076+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 2) v1 ==== 45093+0+0 (secure 0 0 0) 0x7fe9a80039d0 con 0x7fe9b414dc40 2024-03-22T12:56:03.076 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.077+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fe9a8026a40 con 0x7fe9b414dc40 2024-03-22T12:56:03.174 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.175+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe9a8026ce0 con 0x7fe9b414dc40 2024-03-22T12:56:03.244 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.245+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e27c5d29-4343-4274-9e28-9d3cd5c3f32b", "id": 7} v 0) v1 -- 0x7fe9b40c93a0 con 0x7fe9b414dc40 2024-03-22T12:56:03.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.939+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mgrmap(e 3) v1 ==== 89974+0+0 (secure 0 0 0) 0x7fe9a801d980 con 0x7fe9b414dc40 2024-03-22T12:56:03.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.940+0000 7fe9b17fa640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9a006bec0 0x7fe9a006e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:03.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.940+0000 7fe9b8c48640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9a006bec0 0x7fe9a006e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:03.940 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.940+0000 7fe9b8c48640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9a006bec0 0x7fe9a006e2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4034000 tx=0x7fe9a4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:03.962 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.963+0000 7fe9b17fa640 1 -- 172.21.15.92:0/3931757567 <== mon.2 v2:172.21.15.92:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e27c5d29-4343-4274-9e28-9d3cd5c3f32b", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe9a80475c0 con 0x7fe9b414dc40 2024-03-22T12:56:03.963 INFO:teuthology.orchestra.run.smithi092.stdout:7 2024-03-22T12:56:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9a006bec0 msgr2=0x7fe9a006e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9a006bec0 0x7fe9a006e2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4034000 tx=0x7fe9a4008030 comp rx=0 tx=0).stop 2024-03-22T12:56:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9b414dc40 msgr2=0x7fe9b415a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9b414dc40 0x7fe9b415a520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800e510 tx=0x7fe9a8001e70 comp rx=0 tx=0).stop 2024-03-22T12:56:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 shutdown_connections 2024-03-22T12:56:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9b414ef60 0x7fe9b4148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.964 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9b414dc40 0x7fe9b415a520 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9a006bec0 0x7fe9a006e2b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 --2- 172.21.15.92:0/3931757567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe99c003a00 0x7fe9b414cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:03.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 >> 172.21.15.92:0/3931757567 conn(0x7fe9b4071410 msgr2=0x7fe9b41595d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:03.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 shutdown_connections 2024-03-22T12:56:03.965 INFO:teuthology.orchestra.run.smithi092.stderr:2024-03-22T12:56:03.964+0000 7fe9baeac640 1 -- 172.21.15.92:0/3931757567 wait complete. 2024-03-22T12:56:03.974 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-22T12:56:03.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2024-03-22T12:56:03.983 INFO:tasks.ceph.osd.0:Started 2024-03-22T12:56:03.983 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-22T12:56:03.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2024-03-22T12:56:03.989 INFO:tasks.ceph.osd.1:Started 2024-03-22T12:56:03.990 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-22T12:56:03.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2024-03-22T12:56:03.997 INFO:tasks.ceph.osd.2:Started 2024-03-22T12:56:03.997 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-22T12:56:03.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2024-03-22T12:56:04.003 INFO:tasks.ceph.osd.3:Started 2024-03-22T12:56:04.003 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-22T12:56:04.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2024-03-22T12:56:04.005 INFO:tasks.ceph.osd.4:Started 2024-03-22T12:56:04.006 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-22T12:56:04.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2024-03-22T12:56:04.008 INFO:tasks.ceph.osd.5:Started 2024-03-22T12:56:04.008 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-22T12:56:04.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2024-03-22T12:56:04.010 INFO:tasks.ceph.osd.6:Started 2024-03-22T12:56:04.010 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-22T12:56:04.010 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2024-03-22T12:56:04.013 INFO:tasks.ceph.osd.7:Started 2024-03-22T12:56:04.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:04.014 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:56:04.013+0000 7f0e9ef91640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-22T12:56:04.023 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T12:56:04.024+0000 7f0e9ef91640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-22T12:56:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.261+0000 7fa06be00640 1 Processor -- start 2024-03-22T12:56:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.261+0000 7fa06be00640 1 -- start start 2024-03-22T12:56:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.262+0000 7fa06be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa06c0cc380 0x7fa06c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.262+0000 7fa06be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa06c15ae30 0x7fa06c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.262+0000 7fa06be00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06c15b240 con 0x7fa06c0ce8a0 2024-03-22T12:56:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.262+0000 7fa06be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06c0cee00 con 0x7fa06c15ae30 2024-03-22T12:56:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.262+0000 7fa06be00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06c0cf0c0 con 0x7fa06c0cc380 2024-03-22T12:56:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.263+0000 7fa06aa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa06c0cc380 0x7fa06c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.263+0000 7fa06aa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa06c0cc380 0x7fa06c06c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48002/0 (socket says 172.21.15.6:48002) 2024-03-22T12:56:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.263+0000 7fa06aa00640 1 -- 172.21.15.6:0/3124003920 learned_addr learned my addr 172.21.15.6:0/3124003920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.266+0000 7fa06aa00640 1 -- 172.21.15.6:0/3124003920 >> v2:172.21.15.6:3300/0 conn(0x7fa06c15ae30 msgr2=0x7fa06c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.266+0000 7fa06aa00640 1 --2- 172.21.15.6:0/3124003920 >> v2:172.21.15.6:3300/0 conn(0x7fa06c15ae30 0x7fa06c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.266+0000 7fa06aa00640 1 -- 172.21.15.6:0/3124003920 >> v1:172.21.15.92:6789/0 conn(0x7fa06c0ce8a0 legacy=0x7fa06c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.266+0000 7fa06aa00640 1 -- 172.21.15.6:0/3124003920 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c0cf3a0 con 0x7fa06c0cc380 2024-03-22T12:56:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.270+0000 7fa06aa00640 1 --2- 172.21.15.6:0/3124003920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 0x7fa06c06c230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa06c08d0e0 tx=0x7fa05c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e9f821ff9b364e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:04.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.273+0000 7fa06a000640 1 -- 172.21.15.6:0/3124003920 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c034070 con 0x7fa06c0cc380 2024-03-22T12:56:04.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.273+0000 7fa06a000640 1 -- 172.21.15.6:0/3124003920 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c030860 con 0x7fa06c0cc380 2024-03-22T12:56:04.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.273+0000 7fa06a000640 1 -- 172.21.15.6:0/3124003920 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c030c30 con 0x7fa06c0cc380 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 -- 172.21.15.6:0/3124003920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 msgr2=0x7fa06c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 --2- 172.21.15.6:0/3124003920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 0x7fa06c06c230 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa06c08d0e0 tx=0x7fa05c02b9a0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 -- 172.21.15.6:0/3124003920 shutdown_connections 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 --2- 172.21.15.6:0/3124003920 >> v2:172.21.15.6:3300/0 conn(0x7fa06c15ae30 0x7fa06c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 --2- 172.21.15.6:0/3124003920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 0x7fa06c06c230 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 -- 172.21.15.6:0/3124003920 >> 172.21.15.6:0/3124003920 conn(0x7fa06c0b1480 msgr2=0x7fa06c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 -- 172.21.15.6:0/3124003920 shutdown_connections 2024-03-22T12:56:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.274+0000 7fa06be00640 1 -- 172.21.15.6:0/3124003920 wait complete. 2024-03-22T12:56:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.276+0000 7fa06be00640 1 Processor -- start 2024-03-22T12:56:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.276+0000 7fa06be00640 1 -- start start 2024-03-22T12:56:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.277+0000 7fa06be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 0x7fa06c1438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.277+0000 7fa06be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06c0ce8a0 0x7fa06c0c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.277+0000 7fa06be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa06c15ae30 0x7fa06c0c2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.277+0000 7fa06be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c15a910 con 0x7fa06c15ae30 2024-03-22T12:56:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.277+0000 7fa06be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c15a630 con 0x7fa06c0cc380 2024-03-22T12:56:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.277+0000 7fa06be00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c1549d0 con 0x7fa06c0ce8a0 2024-03-22T12:56:04.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.278+0000 7fa06b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa06c15ae30 0x7fa06c0c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:04.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.278+0000 7fa06b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa06c15ae30 0x7fa06c0c2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51000/0 (socket says 172.21.15.6:51000) 2024-03-22T12:56:04.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.278+0000 7fa06b400640 1 -- 172.21.15.6:0/2145199075 learned_addr learned my addr 172.21.15.6:0/2145199075 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:04.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.281+0000 7fa06b400640 1 -- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06c0ce8a0 msgr2=0x7fa06c0c20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:04.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.281+0000 7fa06b400640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06c0ce8a0 0x7fa06c0c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.281+0000 7fa06b400640 1 -- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 msgr2=0x7fa06c1438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:04.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.281+0000 7fa06b400640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 0x7fa06c1438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.281+0000 7fa06b400640 1 -- 172.21.15.6:0/2145199075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c1f01c0 con 0x7fa06c15ae30 2024-03-22T12:56:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.284+0000 7fa06b400640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa06c15ae30 0x7fa06c0c2630 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa06400b9a0 tx=0x7fa06400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:04.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.289+0000 7fa063400640 1 -- 172.21.15.6:0/2145199075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064019070 con 0x7fa06c15ae30 2024-03-22T12:56:04.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.289+0000 7fa063400640 1 -- 172.21.15.6:0/2145199075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06400cbc0 con 0x7fa06c15ae30 2024-03-22T12:56:04.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.289+0000 7fa063400640 1 -- 172.21.15.6:0/2145199075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064014440 con 0x7fa06c15ae30 2024-03-22T12:56:04.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.289+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06c0c1330 con 0x7fa06c15ae30 2024-03-22T12:56:04.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.289+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06c0c18f0 con 0x7fa06c15ae30 2024-03-22T12:56:04.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.290+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06c075650 con 0x7fa06c15ae30 2024-03-22T12:56:04.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.323+0000 7fa063400640 1 -- 172.21.15.6:0/2145199075 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 3) v1 ==== 89974+0+0 (secure 0 0 0) 0x7fa06401c070 con 0x7fa06c15ae30 2024-03-22T12:56:04.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.324+0000 7fa063400640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa04806ba70 0x7fa04806df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:04.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.325+0000 7fa06aa00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa04806ba70 0x7fa04806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:04.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.325+0000 7fa06aa00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa04806ba70 0x7fa04806df30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa05c036000 tx=0x7fa05c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:04.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.328+0000 7fa063400640 1 -- 172.21.15.6:0/2145199075 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa064057d10 con 0x7fa06c15ae30 2024-03-22T12:56:04.639 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-22T12:56:04.639+0000 7f04fa411640 -1 Falling back to public interface 2024-03-22T12:56:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.642+0000 7fa063400640 1 -- 172.21.15.6:0/2145199075 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa064023020 con 0x7fa06c15ae30 2024-03-22T12:56:04.642 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-22T12:56:04.643+0000 7fa2be51f640 -1 Falling back to public interface 2024-03-22T12:56:04.646 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-22T12:56:04.647+0000 7fb28021f640 -1 Falling back to public interface 2024-03-22T12:56:04.647 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-22T12:56:04.647+0000 7f0037415640 -1 Falling back to public interface 2024-03-22T12:56:04.716 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-22T12:56:04.717+0000 7f7b61c19640 -1 Falling back to public interface 2024-03-22T12:56:04.720 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-22T12:56:04.720+0000 7f4dddd73640 -1 Falling back to public interface 2024-03-22T12:56:04.724 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-22T12:56:04.724+0000 7f5d5a128640 -1 Falling back to public interface 2024-03-22T12:56:04.747 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-22T12:56:04.747+0000 7f6b14e9b640 -1 Falling back to public interface 2024-03-22T12:56:04.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.853+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa06c157e10 con 0x7fa06c15ae30 2024-03-22T12:56:04.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.969+0000 7fa063400640 1 -- 172.21.15.6:0/2145199075 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fa064059780 con 0x7fa06c15ae30 2024-03-22T12:56:04.969 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:04.969 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:03.396760+0000","last_up_change":"0.000000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:04.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa04806ba70 msgr2=0x7fa04806df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:04.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa04806ba70 0x7fa04806df30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa05c036000 tx=0x7fa05c030000 comp rx=0 tx=0).stop 2024-03-22T12:56:04.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa06c15ae30 msgr2=0x7fa06c0c2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:04.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa06c15ae30 0x7fa06c0c2630 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa06400b9a0 tx=0x7fa06400be60 comp rx=0 tx=0).stop 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 shutdown_connections 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa06c15ae30 0x7fa06c0c2630 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa06c0ce8a0 0x7fa06c0c20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa04806ba70 0x7fa04806df30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 --2- 172.21.15.6:0/2145199075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa06c0cc380 0x7fa06c1438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 >> 172.21.15.6:0/2145199075 conn(0x7fa06c0b1480 msgr2=0x7fa06c149e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 shutdown_connections 2024-03-22T12:56:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:04.972+0000 7fa06be00640 1 -- 172.21.15.6:0/2145199075 wait complete. 2024-03-22T12:56:04.991 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-22T12:56:04.992 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-22T12:56:05.797 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:05.966 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.966+0000 7fb0c4e00640 1 Processor -- start 2024-03-22T12:56:05.967 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.967+0000 7fb0c4e00640 1 -- start start 2024-03-22T12:56:05.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.968+0000 7fb0c4e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c0154510 0x7fb0c00c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:05.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.968+0000 7fb0c4e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb0c015b120 0x7fb0c00c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:05.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.968+0000 7fb0c4e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c00c8be0 con 0x7fb0c01548f0 2024-03-22T12:56:05.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.968+0000 7fb0c4e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c015e8e0 con 0x7fb0c015b120 2024-03-22T12:56:05.968 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.968+0000 7fb0c4e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c015d9f0 con 0x7fb0c0154510 2024-03-22T12:56:05.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.969+0000 7fb0bde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c0154510 0x7fb0c00c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:05.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.969+0000 7fb0be800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb0c015b120 0x7fb0c00c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:05.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.969+0000 7fb0be800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb0c015b120 0x7fb0c00c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48028/0 (socket says 172.21.15.6:48028) 2024-03-22T12:56:05.969 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.969+0000 7fb0be800640 1 -- 172.21.15.6:0/332744858 learned_addr learned my addr 172.21.15.6:0/332744858 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.971+0000 7fb0be800640 1 -- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c0154510 msgr2=0x7fb0c00c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.971+0000 7fb0be800640 1 --2- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c0154510 0x7fb0c00c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.971+0000 7fb0be800640 1 -- 172.21.15.6:0/332744858 >> v1:172.21.15.6:6789/0 conn(0x7fb0c01548f0 legacy=0x7fb0c00c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.971+0000 7fb0be800640 1 -- 172.21.15.6:0/332744858 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c015dcd0 con 0x7fb0c015b120 2024-03-22T12:56:05.971 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.971+0000 7fb0bde00640 1 --2- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c0154510 0x7fb0c00c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:05.973 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.973+0000 7fb0be800640 1 --2- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c00c44b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009e50 tx=0x7fb0b802bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d37dd2ec856c546 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.977+0000 7fb0bd400640 1 -- 172.21.15.6:0/332744858 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b802cb90 con 0x7fb0c015b120 2024-03-22T12:56:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.977+0000 7fb0bd400640 1 -- 172.21.15.6:0/332744858 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b802cd30 con 0x7fb0c015b120 2024-03-22T12:56:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.977+0000 7fb0bd400640 1 -- 172.21.15.6:0/332744858 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8034610 con 0x7fb0c015b120 2024-03-22T12:56:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 -- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 msgr2=0x7fb0c00c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:05.977 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c00c44b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009e50 tx=0x7fb0b802bff0 comp rx=0 tx=0).stop 2024-03-22T12:56:05.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 -- 172.21.15.6:0/332744858 shutdown_connections 2024-03-22T12:56:05.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c00c44b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:05.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/332744858 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c0154510 0x7fb0c00c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:05.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 -- 172.21.15.6:0/332744858 >> 172.21.15.6:0/332744858 conn(0x7fb0c006b090 msgr2=0x7fb0c006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:05.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 -- 172.21.15.6:0/332744858 shutdown_connections 2024-03-22T12:56:05.978 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.978+0000 7fb0c4e00640 1 -- 172.21.15.6:0/332744858 wait complete. 2024-03-22T12:56:05.979 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.980+0000 7fb0c4e00640 1 Processor -- start 2024-03-22T12:56:05.980 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.980+0000 7fb0c4e00640 1 -- start start 2024-03-22T12:56:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.981+0000 7fb0c4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0c0154510 0x7fb0c01e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.981+0000 7fb0c4e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c01548f0 0x7fb0c01e6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.981+0000 7fb0c4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c014e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.981+0000 7fb0c4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c015e570 con 0x7fb0c0154510 2024-03-22T12:56:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.981+0000 7fb0c4e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c015e290 con 0x7fb0c015b120 2024-03-22T12:56:05.981 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.981+0000 7fb0c4e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00c7060 con 0x7fb0c01548f0 2024-03-22T12:56:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.982+0000 7fb0be800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.982+0000 7fb0b7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c01548f0 0x7fb0c01e6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.982+0000 7fb0bde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0c0154510 0x7fb0c01e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.982+0000 7fb0be800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c014e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48038/0 (socket says 172.21.15.6:48038) 2024-03-22T12:56:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.982+0000 7fb0be800640 1 -- 172.21.15.6:0/223764899 learned_addr learned my addr 172.21.15.6:0/223764899 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:05.982 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.982+0000 7fb0b7e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c01548f0 0x7fb0c01e6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55450/0 (socket says 172.21.15.6:55450) 2024-03-22T12:56:05.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.986+0000 7fb0bde00640 1 -- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c01548f0 msgr2=0x7fb0c01e6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:05.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.986+0000 7fb0bde00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c01548f0 0x7fb0c01e6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:05.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.986+0000 7fb0bde00640 1 -- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 msgr2=0x7fb0c014e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:05.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.986+0000 7fb0bde00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c014e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:05.986 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.986+0000 7fb0bde00640 1 -- 172.21.15.6:0/223764899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c0153c20 con 0x7fb0c0154510 2024-03-22T12:56:05.988 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.988+0000 7fb0bde00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0c0154510 0x7fb0c01e5e50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000e870 tx=0x7fb0b000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:05.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.993+0000 7fb0b6000640 1 -- 172.21.15.6:0/223764899 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0017070 con 0x7fb0c0154510 2024-03-22T12:56:05.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.993+0000 7fb0b6000640 1 -- 172.21.15.6:0/223764899 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b000c940 con 0x7fb0c0154510 2024-03-22T12:56:05.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.993+0000 7fb0b6000640 1 -- 172.21.15.6:0/223764899 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b000cd10 con 0x7fb0c0154510 2024-03-22T12:56:05.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.993+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c015b660 con 0x7fb0c0154510 2024-03-22T12:56:05.993 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.993+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c015c140 con 0x7fb0c0154510 2024-03-22T12:56:05.994 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:05.995+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c004f470 con 0x7fb0c0154510 2024-03-22T12:56:06.333 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.333+0000 7fb0b6000640 1 -- 172.21.15.6:0/223764899 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7fb0b001a070 con 0x7fb0c0154510 2024-03-22T12:56:06.335 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.335+0000 7fb0b6000640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb09c06bb90 0x7fb09c06e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:06.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.336+0000 7fb0b7e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb09c06bb90 0x7fb09c06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:06.336 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.337+0000 7fb0b7e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb09c06bb90 0x7fb09c06e050 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0153af0 tx=0x7fb0a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:06.369 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.370+0000 7fb0b6000640 1 -- 172.21.15.6:0/223764899 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb0b00580a0 con 0x7fb0c0154510 2024-03-22T12:56:06.370 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.370+0000 7fb0b6000640 1 -- 172.21.15.6:0/223764899 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb0b0059410 con 0x7fb0c0154510 2024-03-22T12:56:06.582 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.582+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0c00c74d0 con 0x7fb0c0154510 2024-03-22T12:56:06.629 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.629+0000 7fb0b6000640 1 -- 172.21.15.6:0/223764899 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb0b005d4b0 con 0x7fb0c0154510 2024-03-22T12:56:06.630 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-22T12:56:06.630 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:03.396760+0000","last_up_change":"0.000000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb09c06bb90 msgr2=0x7fb09c06e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb09c06bb90 0x7fb09c06e050 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0153af0 tx=0x7fb0a8009040 comp rx=0 tx=0).stop 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0c0154510 msgr2=0x7fb0c01e5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0c0154510 0x7fb0c01e5e50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000e870 tx=0x7fb0b000ed30 comp rx=0 tx=0).stop 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 shutdown_connections 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb09c06bb90 0x7fb09c06e050 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0c015b120 0x7fb0c014e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0c01548f0 0x7fb0c01e6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:06.633 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 --2- 172.21.15.6:0/223764899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0c0154510 0x7fb0c01e5e50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:06.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 >> 172.21.15.6:0/223764899 conn(0x7fb0c006b090 msgr2=0x7fb0c01525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:06.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.633+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 shutdown_connections 2024-03-22T12:56:06.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:06.634+0000 7fb0c4e00640 1 -- 172.21.15.6:0/223764899 wait complete. 2024-03-22T12:56:06.649 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-22T12:56:08.087 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-22T12:56:08.087+0000 7f5d5a128640 -1 osd.2 0 log_to_monitors true 2024-03-22T12:56:08.090 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-22T12:56:08.091+0000 7f04fa411640 -1 osd.6 0 log_to_monitors true 2024-03-22T12:56:08.133 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-22T12:56:08.134+0000 7fa2be51f640 -1 osd.5 0 log_to_monitors true 2024-03-22T12:56:08.155 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-22T12:56:08.155+0000 7fb28021f640 -1 osd.7 0 log_to_monitors true 2024-03-22T12:56:08.169 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-22T12:56:08.170+0000 7f0037415640 -1 osd.4 0 log_to_monitors true 2024-03-22T12:56:08.268 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-22T12:56:08.268+0000 7f7b61c19640 -1 osd.0 0 log_to_monitors true 2024-03-22T12:56:08.296 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-22T12:56:08.296+0000 7f4dddd73640 -1 osd.1 0 log_to_monitors true 2024-03-22T12:56:08.323 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-22T12:56:08.324+0000 7f6b14e9b640 -1 osd.3 0 log_to_monitors true 2024-03-22T12:56:10.671 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-22T12:56:10.671+0000 7fb27bc88640 -1 osd.7 0 waiting for initial osdmap 2024-03-22T12:56:10.696 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-22T12:56:10.697+0000 7f04f5e27640 -1 osd.6 0 waiting for initial osdmap 2024-03-22T12:56:10.703 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-22T12:56:10.704+0000 7fb273aa8640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:10.728 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-22T12:56:10.729+0000 7f04edc47640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:10.931 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-22T12:56:10.931+0000 7f5d55a00640 -1 osd.2 0 waiting for initial osdmap 2024-03-22T12:56:10.963 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-22T12:56:10.963+0000 7f5d4b000640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:12.025 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-22T12:56:12.026+0000 7fa2ba4b5640 -1 osd.5 0 waiting for initial osdmap 2024-03-22T12:56:12.052 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-22T12:56:12.053+0000 7fa2b22d5640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:12.054 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-22T12:56:12.055+0000 7f0032e0c640 -1 osd.4 0 waiting for initial osdmap 2024-03-22T12:56:12.075 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-22T12:56:12.076+0000 7f002ac2c640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:12.635 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-22T12:56:12.635+0000 7f6b10800640 -1 osd.3 0 waiting for initial osdmap 2024-03-22T12:56:12.638 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-22T12:56:12.638+0000 7f4dd9600640 -1 osd.1 0 waiting for initial osdmap 2024-03-22T12:56:12.644 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-22T12:56:12.645+0000 7f7b5c600640 -1 osd.0 0 waiting for initial osdmap 2024-03-22T12:56:12.706 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-22T12:56:12.706+0000 7f6b05e00640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:12.715 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-22T12:56:12.715+0000 7f7b52600640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:12.716 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-22T12:56:12.717+0000 7f4dcec00640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T12:56:13.455 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.633+0000 7f3b5b200640 1 Processor -- start 2024-03-22T12:56:13.632 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.633+0000 7f3b5b200640 1 -- start start 2024-03-22T12:56:13.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.634+0000 7f3b5b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b5c1522e0 0x7f3b5c15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:13.634 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.634+0000 7f3b5b200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3b5c06f2b0 0x7f3b5c071450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:13.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.634+0000 7f3b5b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c06f860 con 0x7f3b5c151f00 2024-03-22T12:56:13.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.635+0000 7f3b5b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c0c1c90 con 0x7f3b5c1522e0 2024-03-22T12:56:13.635 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.635+0000 7f3b5b200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c06fe20 con 0x7f3b5c06f2b0 2024-03-22T12:56:13.640 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.641+0000 7f3b5a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3b5c06f2b0 0x7f3b5c071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.641+0000 7f3b5a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3b5c06f2b0 0x7f3b5c071450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47728/0 (socket says 172.21.15.6:47728) 2024-03-22T12:56:13.641 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.641+0000 7f3b5a800640 1 -- 172.21.15.6:0/3391656401 learned_addr learned my addr 172.21.15.6:0/3391656401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:13.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.644+0000 7f3b5a800640 1 -- 172.21.15.6:0/3391656401 >> v2:172.21.15.6:3300/0 conn(0x7f3b5c1522e0 msgr2=0x7f3b5c15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:13.643 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.644+0000 7f3b5a800640 1 --2- 172.21.15.6:0/3391656401 >> v2:172.21.15.6:3300/0 conn(0x7f3b5c1522e0 0x7f3b5c15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:13.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.644+0000 7f3b5a800640 1 -- 172.21.15.6:0/3391656401 >> v1:172.21.15.6:6789/0 conn(0x7f3b5c151f00 legacy=0x7f3b5c0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:13.644 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.644+0000 7f3b5a800640 1 -- 172.21.15.6:0/3391656401 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c06fb40 con 0x7f3b5c06f2b0 2024-03-22T12:56:13.645 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.646+0000 7f3b5a800640 1 --2- 172.21.15.6:0/3391656401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c06f2b0 0x7f3b5c071450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0c1f30 tx=0x7f3b5402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4550b62856c6e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:13.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.649+0000 7f3b59400640 1 -- 172.21.15.6:0/3391656401 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5402cb20 con 0x7f3b5c06f2b0 2024-03-22T12:56:13.649 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b59400640 1 -- 172.21.15.6:0/3391656401 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5402ccc0 con 0x7f3b5c06f2b0 2024-03-22T12:56:13.650 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b59400640 1 -- 172.21.15.6:0/3391656401 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54034610 con 0x7f3b5c06f2b0 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b5b200640 1 -- 172.21.15.6:0/3391656401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c06f2b0 msgr2=0x7f3b5c071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b5b200640 1 --2- 172.21.15.6:0/3391656401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c06f2b0 0x7f3b5c071450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0c1f30 tx=0x7f3b5402bf80 comp rx=0 tx=0).stop 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b5b200640 1 -- 172.21.15.6:0/3391656401 shutdown_connections 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b5b200640 1 --2- 172.21.15.6:0/3391656401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c06f2b0 0x7f3b5c071450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b5b200640 1 --2- 172.21.15.6:0/3391656401 >> v2:172.21.15.6:3300/0 conn(0x7f3b5c1522e0 0x7f3b5c15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.650+0000 7f3b5b200640 1 -- 172.21.15.6:0/3391656401 >> 172.21.15.6:0/3391656401 conn(0x7f3b5c06b090 msgr2=0x7f3b5c06b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.651+0000 7f3b5b200640 1 -- 172.21.15.6:0/3391656401 shutdown_connections 2024-03-22T12:56:13.651 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.651+0000 7f3b5b200640 1 -- 172.21.15.6:0/3391656401 wait complete. 2024-03-22T12:56:13.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.653+0000 7f3b5b200640 1 Processor -- start 2024-03-22T12:56:13.653 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.653+0000 7f3b5b200640 1 -- start start 2024-03-22T12:56:13.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.655+0000 7f3b5b200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5c06f2b0 0x7f3b5c0715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:13.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.655+0000 7f3b5b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c151f00 0x7f3b5c1f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:13.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.655+0000 7f3b5b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b5c1522e0 0x7f3b5c1f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:13.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.655+0000 7f3b5b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c158540 con 0x7f3b5c1522e0 2024-03-22T12:56:13.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.655+0000 7f3b5b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c158260 con 0x7f3b5c151f00 2024-03-22T12:56:13.655 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.655+0000 7f3b5b200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0c53f0 con 0x7f3b5c06f2b0 2024-03-22T12:56:13.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.656+0000 7f3b53e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c151f00 0x7f3b5c1f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:13.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.656+0000 7f3b59e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5c06f2b0 0x7f3b5c0715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:13.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.656+0000 7f3b53e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c151f00 0x7f3b5c1f1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47742/0 (socket says 172.21.15.6:47742) 2024-03-22T12:56:13.656 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.656+0000 7f3b53e00640 1 -- 172.21.15.6:0/1304830916 learned_addr learned my addr 172.21.15.6:0/1304830916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:13.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.659+0000 7f3b53e00640 1 -- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5c06f2b0 msgr2=0x7f3b5c0715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:13.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.659+0000 7f3b53e00640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5c06f2b0 0x7f3b5c0715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:13.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.659+0000 7f3b53e00640 1 -- 172.21.15.6:0/1304830916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b5c1522e0 msgr2=0x7f3b5c1f1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:13.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.659+0000 7f3b53e00640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b5c1522e0 0x7f3b5c1f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:13.659 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.659+0000 7f3b53e00640 1 -- 172.21.15.6:0/1304830916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c154700 con 0x7f3b5c151f00 2024-03-22T12:56:13.661 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.661+0000 7f3b53e00640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c151f00 0x7f3b5c1f1300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400b470 tx=0x7f3b4400b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:13.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.665+0000 7f3b52000640 1 -- 172.21.15.6:0/1304830916 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44015020 con 0x7f3b5c151f00 2024-03-22T12:56:13.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.665+0000 7f3b52000640 1 -- 172.21.15.6:0/1304830916 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b440118b0 con 0x7f3b5c151f00 2024-03-22T12:56:13.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.665+0000 7f3b52000640 1 -- 172.21.15.6:0/1304830916 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44011ca0 con 0x7f3b5c151f00 2024-03-22T12:56:13.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.665+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c154e50 con 0x7f3b5c151f00 2024-03-22T12:56:13.665 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.665+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c1550f0 con 0x7f3b5c151f00 2024-03-22T12:56:13.666 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.666+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5c04f400 con 0x7f3b5c151f00 2024-03-22T12:56:13.686 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.687+0000 7f3b52000640 1 -- 172.21.15.6:0/1304830916 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7f3b44018020 con 0x7f3b5c151f00 2024-03-22T12:56:13.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.687+0000 7f3b52000640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3806bbc0 0x7f3b3806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:13.687 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.688+0000 7f3b59e00640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3806bbc0 0x7f3b3806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:13.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.688+0000 7f3b52000640 1 -- 172.21.15.6:0/1304830916 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3071+0+0 (secure 0 0 0) 0x7f3b44057590 con 0x7f3b5c151f00 2024-03-22T12:56:13.688 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.689+0000 7f3b59e00640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3806bbc0 0x7f3b3806e080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0017f0 tx=0x7f3b4c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:13.945 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:13.945+0000 7f3b52000640 1 -- 172.21.15.6:0/1304830916 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3b4401f020 con 0x7f3b5c151f00 2024-03-22T12:56:14.177 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.178+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b5c068470 con 0x7f3b5c151f00 2024-03-22T12:56:14.229 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.229+0000 7f3b52000640 1 -- 172.21.15.6:0/1304830916 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10882 (secure 0 0 0) 0x7f3b4400d040 con 0x7f3b5c151f00 2024-03-22T12:56:14.229 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-22T12:56:14.230 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":15,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:13.373035+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.234+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3806bbc0 msgr2=0x7f3b3806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.234+0000 7f3b5b200640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3806bbc0 0x7f3b3806e080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0017f0 tx=0x7f3b4c009040 comp rx=0 tx=0).stop 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.234+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c151f00 msgr2=0x7f3b5c1f1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.234+0000 7f3b5b200640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c151f00 0x7f3b5c1f1300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400b470 tx=0x7f3b4400b930 comp rx=0 tx=0).stop 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 shutdown_connections 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b5c1522e0 0x7f3b5c1f1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3806bbc0 0x7f3b3806e080 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b5c151f00 0x7f3b5c1f1300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 --2- 172.21.15.6:0/1304830916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5c06f2b0 0x7f3b5c0715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 >> 172.21.15.6:0/1304830916 conn(0x7f3b5c06b090 msgr2=0x7f3b5c0c93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 shutdown_connections 2024-03-22T12:56:14.235 INFO:teuthology.misc.health.smithi006.stderr:2024-03-22T12:56:14.235+0000 7f3b5b200640 1 -- 172.21.15.6:0/1304830916 wait complete. 2024-03-22T12:56:14.251 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-22T12:56:14.251 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-22T12:56:14.252 DEBUG:tasks.ceph_manager:running mds.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-22T12:56:14.252 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-22T12:56:14.252 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a 2024-03-22T12:56:14.259 INFO:tasks.ceph.mds.a:Started 2024-03-22T12:56:14.259 DEBUG:tasks.ceph_manager:running mds.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-22T12:56:14.259 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-22T12:56:14.259 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c 2024-03-22T12:56:14.266 INFO:tasks.ceph.mds.c:Started 2024-03-22T12:56:14.266 DEBUG:tasks.ceph_manager:running mds.e under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.e.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-22T12:56:14.266 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-22T12:56:14.267 DEBUG:teuthology.orchestra.run.smithi006:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i e 2024-03-22T12:56:14.274 INFO:tasks.ceph.mds.e:Started 2024-03-22T12:56:14.274 DEBUG:tasks.ceph_manager:running mds.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-22T12:56:14.274 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-22T12:56:14.274 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i b 2024-03-22T12:56:14.277 INFO:tasks.ceph.mds.b:Started 2024-03-22T12:56:14.277 DEBUG:tasks.ceph_manager:running mds.d under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.d.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-22T12:56:14.277 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-22T12:56:14.277 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i d 2024-03-22T12:56:14.279 INFO:tasks.ceph.mds.d:Started 2024-03-22T12:56:14.279 DEBUG:tasks.ceph_manager:running mds.f under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.f.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-22T12:56:14.279 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-22T12:56:14.279 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i f 2024-03-22T12:56:14.281 INFO:tasks.ceph.mds.f:Started 2024-03-22T12:56:14.282 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-22T12:56:14.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:14.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.474+0000 7f13f5000640 1 Processor -- start 2024-03-22T12:56:14.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.474+0000 7f13f5000640 1 -- start start 2024-03-22T12:56:14.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.475+0000 7f13f5000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13f0151010 0x7f13f00c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:14.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.476+0000 7f13f5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13f015a910 0x7f13f00725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:14.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.476+0000 7f13f5000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f00c56b0 con 0x7f13f015a4a0 2024-03-22T12:56:14.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.476+0000 7f13f5000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f00c2d90 con 0x7f13f015a910 2024-03-22T12:56:14.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.476+0000 7f13f5000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f00c6d70 con 0x7f13f0151010 2024-03-22T12:56:14.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.476+0000 7f13ef400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13f0151010 0x7f13f00c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:14.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.477+0000 7f13efe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13f015a910 0x7f13f00725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:14.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.477+0000 7f13efe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13f015a910 0x7f13f00725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48672/0 (socket says 172.21.15.6:48672) 2024-03-22T12:56:14.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.477+0000 7f13efe00640 1 -- 172.21.15.6:0/1224041093 learned_addr learned my addr 172.21.15.6:0/1224041093 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:14.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.480+0000 7f13efe00640 1 -- 172.21.15.6:0/1224041093 >> v2:172.21.15.92:3300/0 conn(0x7f13f0151010 msgr2=0x7f13f00c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:14.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.480+0000 7f13efe00640 1 --2- 172.21.15.6:0/1224041093 >> v2:172.21.15.92:3300/0 conn(0x7f13f0151010 0x7f13f00c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.480+0000 7f13efe00640 1 -- 172.21.15.6:0/1224041093 >> v1:172.21.15.92:6789/0 conn(0x7f13f015a4a0 legacy=0x7f13f00c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:14.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.480+0000 7f13efe00640 1 -- 172.21.15.6:0/1224041093 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f00c5940 con 0x7f13f015a910 2024-03-22T12:56:14.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.483+0000 7f13efe00640 1 --2- 172.21.15.6:0/1224041093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f015a910 0x7f13f00725b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f13f0070e90 tx=0x7f13e802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a84b2b8f20b5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:14.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.488+0000 7f13ee000640 1 -- 172.21.15.6:0/1224041093 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e802cb10 con 0x7f13f015a910 2024-03-22T12:56:14.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.488+0000 7f13ee000640 1 -- 172.21.15.6:0/1224041093 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e802ccb0 con 0x7f13f015a910 2024-03-22T12:56:14.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.488+0000 7f13ee000640 1 -- 172.21.15.6:0/1224041093 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e802ce60 con 0x7f13f015a910 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 -- 172.21.15.6:0/1224041093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f015a910 msgr2=0x7f13f00725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 --2- 172.21.15.6:0/1224041093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f015a910 0x7f13f00725b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f13f0070e90 tx=0x7f13e802bf70 comp rx=0 tx=0).stop 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 -- 172.21.15.6:0/1224041093 shutdown_connections 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 --2- 172.21.15.6:0/1224041093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f015a910 0x7f13f00725b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 --2- 172.21.15.6:0/1224041093 >> v2:172.21.15.92:3300/0 conn(0x7f13f0151010 0x7f13f00c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 -- 172.21.15.6:0/1224041093 >> 172.21.15.6:0/1224041093 conn(0x7f13f00cb890 msgr2=0x7f13f00cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 -- 172.21.15.6:0/1224041093 shutdown_connections 2024-03-22T12:56:14.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.489+0000 7f13f5000640 1 -- 172.21.15.6:0/1224041093 wait complete. 2024-03-22T12:56:14.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.492+0000 7f13f5000640 1 Processor -- start 2024-03-22T12:56:14.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.492+0000 7f13f5000640 1 -- start start 2024-03-22T12:56:14.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.494+0000 7f13f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f0151010 0x7f13f014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:14.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.494+0000 7f13f5000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13f015a4a0 0x7f13f01552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:14.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.494+0000 7f13f5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13f015a910 0x7f13f0159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:14.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.494+0000 7f13f5000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f00c32c0 con 0x7f13f0151010 2024-03-22T12:56:14.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.494+0000 7f13f5000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f00c2fe0 con 0x7f13f015a910 2024-03-22T12:56:14.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.494+0000 7f13f5000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f00c35a0 con 0x7f13f015a4a0 2024-03-22T12:56:14.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.495+0000 7f13eea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13f015a4a0 0x7f13f01552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:14.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.495+0000 7f13efe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13f015a910 0x7f13f0159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:14.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.495+0000 7f13eea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13f015a4a0 0x7f13f01552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55166/0 (socket says 172.21.15.6:55166) 2024-03-22T12:56:14.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.495+0000 7f13eea00640 1 -- 172.21.15.6:0/1884956186 learned_addr learned my addr 172.21.15.6:0/1884956186 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:14.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.495+0000 7f13ef400640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f0151010 0x7f13f014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:14.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.500+0000 7f13eea00640 1 -- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13f015a910 msgr2=0x7f13f0159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.500+0000 7f13eea00640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13f015a910 0x7f13f0159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.500+0000 7f13eea00640 1 -- 172.21.15.6:0/1884956186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f0151010 msgr2=0x7f13f014aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.500+0000 7f13eea00640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f0151010 0x7f13f014aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.500+0000 7f13eea00640 1 -- 172.21.15.6:0/1884956186 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f01580b0 con 0x7f13f015a4a0 2024-03-22T12:56:14.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.503+0000 7f13eea00640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13f015a4a0 0x7f13f01552b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00c3c0 tx=0x7f13dc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:14.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.505+0000 7f13e7400640 1 -- 172.21.15.6:0/1884956186 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc0052d0 con 0x7f13f015a4a0 2024-03-22T12:56:14.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.505+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f01623e0 con 0x7f13f015a4a0 2024-03-22T12:56:14.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.506+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f0149d00 con 0x7f13f015a4a0 2024-03-22T12:56:14.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.507+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f0075650 con 0x7f13f015a4a0 2024-03-22T12:56:14.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.713+0000 7f13e7400640 1 -- 172.21.15.6:0/1884956186 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc005470 con 0x7f13f015a4a0 2024-03-22T12:56:14.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.782+0000 7f13e7400640 1 -- 172.21.15.6:0/1884956186 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc020060 con 0x7f13f015a4a0 2024-03-22T12:56:14.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.785+0000 7f13e7400640 1 -- 172.21.15.6:0/1884956186 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7f13dc0072f0 con 0x7f13f015a4a0 2024-03-22T12:56:14.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.786+0000 7f13e7400640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f13d006bb40 0x7f13d006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:14.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.787+0000 7f13e7400640 1 -- 172.21.15.6:0/1884956186 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f13dc089d00 con 0x7f13f015a4a0 2024-03-22T12:56:14.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.787+0000 7f13ef400640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f13d006bb40 0x7f13d006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:14.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.788+0000 7f13ef400640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f13d006bb40 0x7f13d006e000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f13f014f9e0 tx=0x7f13e0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:14.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:14.797+0000 7f13e7400640 1 -- 172.21.15.6:0/1884956186 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f13dc0587e0 con 0x7f13f015a4a0 2024-03-22T12:56:15.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.163+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13f014efc0 con 0x7f13f015a4a0 2024-03-22T12:56:15.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.400+0000 7f13e7400640 1 -- 172.21.15.6:0/1884956186 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12792 (secure 0 0 0) 0x7f13dc05c880 con 0x7f13f015a4a0 2024-03-22T12:56:15.401 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:15.401 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:14.706607+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f13d006bb40 msgr2=0x7f13d006e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f13d006bb40 0x7f13d006e000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f13f014f9e0 tx=0x7f13e0009190 comp rx=0 tx=0).stop 2024-03-22T12:56:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13f015a4a0 msgr2=0x7f13f01552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13f015a4a0 0x7f13f01552b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00c3c0 tx=0x7f13dc00c880 comp rx=0 tx=0).stop 2024-03-22T12:56:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 shutdown_connections 2024-03-22T12:56:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f13d006bb40 0x7f13d006e000 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13f015a910 0x7f13f0159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13f015a4a0 0x7f13f01552b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 --2- 172.21.15.6:0/1884956186 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13f0151010 0x7f13f014aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 >> 172.21.15.6:0/1884956186 conn(0x7f13f00cb890 msgr2=0x7f13f00c2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 shutdown_connections 2024-03-22T12:56:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.403+0000 7f13f5000640 1 -- 172.21.15.6:0/1884956186 wait complete. 2024-03-22T12:56:15.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T12:56:14.196164+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T12:56:15.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T12:56:15.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.614+0000 7fb3fa800640 1 Processor -- start 2024-03-22T12:56:15.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.614+0000 7fb3fa800640 1 -- start start 2024-03-22T12:56:15.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.615+0000 7fb3fa800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3f4151fe0 0x7fb3f40cace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:15.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.615+0000 7fb3fa800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb3f4158110 0x7fb3f40c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:15.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.615+0000 7fb3fa800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f41585c0 con 0x7fb3f40ca720 2024-03-22T12:56:15.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.615+0000 7fb3fa800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f415b930 con 0x7fb3f4158110 2024-03-22T12:56:15.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.615+0000 7fb3fa800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f41588a0 con 0x7fb3f4151fe0 2024-03-22T12:56:15.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.616+0000 7fb3f3400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3f4151fe0 0x7fb3f40cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:15.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.617+0000 7fb3f3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb3f4158110 0x7fb3f40c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:15.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.617+0000 7fb3f3400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3f4151fe0 0x7fb3f40cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47770/0 (socket says 172.21.15.6:47770) 2024-03-22T12:56:15.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.617+0000 7fb3f3400640 1 -- 172.21.15.6:0/1265880365 learned_addr learned my addr 172.21.15.6:0/1265880365 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:15.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.619+0000 7fb3f3e00640 1 -- 172.21.15.6:0/1265880365 >> v2:172.21.15.92:3300/0 conn(0x7fb3f4151fe0 msgr2=0x7fb3f40cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:15.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.619+0000 7fb3f3e00640 1 --2- 172.21.15.6:0/1265880365 >> v2:172.21.15.92:3300/0 conn(0x7fb3f4151fe0 0x7fb3f40cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.620+0000 7fb3f3e00640 1 -- 172.21.15.6:0/1265880365 >> v1:172.21.15.92:6789/0 conn(0x7fb3f40ca720 legacy=0x7fb3f415bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:15.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.620+0000 7fb3f3e00640 1 -- 172.21.15.6:0/1265880365 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4158b80 con 0x7fb3f4158110 2024-03-22T12:56:15.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.622+0000 7fb3f3e00640 1 --2- 172.21.15.6:0/1265880365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f4158110 0x7fb3f40c88d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40cb770 tx=0x7fb3ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905aad1ba0cd7c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:15.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.627+0000 7fb3f2000640 1 -- 172.21.15.6:0/1265880365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec039070 con 0x7fb3f4158110 2024-03-22T12:56:15.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.628+0000 7fb3f2000640 1 -- 172.21.15.6:0/1265880365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec02cc80 con 0x7fb3f4158110 2024-03-22T12:56:15.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.628+0000 7fb3f2000640 1 -- 172.21.15.6:0/1265880365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec034660 con 0x7fb3f4158110 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.628+0000 7fb3fa800640 1 -- 172.21.15.6:0/1265880365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f4158110 msgr2=0x7fb3f40c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.628+0000 7fb3fa800640 1 --2- 172.21.15.6:0/1265880365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f4158110 0x7fb3f40c88d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40cb770 tx=0x7fb3ec02bed0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.628+0000 7fb3fa800640 1 -- 172.21.15.6:0/1265880365 shutdown_connections 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.628+0000 7fb3fa800640 1 --2- 172.21.15.6:0/1265880365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f4158110 0x7fb3f40c88d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.628+0000 7fb3fa800640 1 --2- 172.21.15.6:0/1265880365 >> v2:172.21.15.92:3300/0 conn(0x7fb3f4151fe0 0x7fb3f40cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.629+0000 7fb3fa800640 1 -- 172.21.15.6:0/1265880365 >> 172.21.15.6:0/1265880365 conn(0x7fb3f40714a0 msgr2=0x7fb3f4071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.629+0000 7fb3fa800640 1 -- 172.21.15.6:0/1265880365 shutdown_connections 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.629+0000 7fb3fa800640 1 -- 172.21.15.6:0/1265880365 wait complete. 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.630+0000 7fb3fa800640 1 Processor -- start 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.631+0000 7fb3fa800640 1 -- start start 2024-03-22T12:56:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.631+0000 7fb3fa800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f40ca720 0x7fb3f41efc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.631+0000 7fb3fa800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3f4151fe0 0x7fb3f414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.631+0000 7fb3fa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f4158110 0x7fb3f41594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.631+0000 7fb3fa800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f415c650 con 0x7fb3f40ca720 2024-03-22T12:56:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.631+0000 7fb3fa800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f415c4d0 con 0x7fb3f4158110 2024-03-22T12:56:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.631+0000 7fb3fa800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40cb220 con 0x7fb3f4151fe0 2024-03-22T12:56:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.632+0000 7fb3f3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f4158110 0x7fb3f41594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.632+0000 7fb3f2a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3f4151fe0 0x7fb3f414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.633+0000 7fb3f3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f4158110 0x7fb3f41594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47776/0 (socket says 172.21.15.6:47776) 2024-03-22T12:56:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.633+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2792987049 learned_addr learned my addr 172.21.15.6:0/2792987049 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:15.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.634+0000 7fb3f3400640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f40ca720 0x7fb3f41efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.637+0000 7fb3f2a00640 1 -- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f4158110 msgr2=0x7fb3f41594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.637+0000 7fb3f2a00640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f4158110 0x7fb3f41594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.637+0000 7fb3f2a00640 1 -- 172.21.15.6:0/2792987049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f40ca720 msgr2=0x7fb3f41efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.637+0000 7fb3f2a00640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f40ca720 0x7fb3f41efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.637+0000 7fb3f2a00640 1 -- 172.21.15.6:0/2792987049 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f41ebce0 con 0x7fb3f4151fe0 2024-03-22T12:56:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.640+0000 7fb3f2a00640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3f4151fe0 0x7fb3f414b7c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c370 tx=0x7fb3e000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.644+0000 7fb3eb400640 1 -- 172.21.15.6:0/2792987049 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e00052d0 con 0x7fb3f4151fe0 2024-03-22T12:56:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.644+0000 7fb3eb400640 1 -- 172.21.15.6:0/2792987049 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e0005490 con 0x7fb3f4151fe0 2024-03-22T12:56:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.644+0000 7fb3eb400640 1 -- 172.21.15.6:0/2792987049 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0007260 con 0x7fb3f4151fe0 2024-03-22T12:56:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.644+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f41ebfa0 con 0x7fb3f4151fe0 2024-03-22T12:56:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.644+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f41ec560 con 0x7fb3f4151fe0 2024-03-22T12:56:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.645+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f40757d0 con 0x7fb3f4151fe0 2024-03-22T12:56:15.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.665+0000 7fb3eb400640 1 -- 172.21.15.6:0/2792987049 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7fb3e0007400 con 0x7fb3f4151fe0 2024-03-22T12:56:15.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.666+0000 7fb3eb400640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d406bb90 0x7fb3d406e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:15.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.667+0000 7fb3f3400640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d406bb90 0x7fb3d406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:15.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.668+0000 7fb3f3400640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d406bb90 0x7fb3d406e050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009bd0 tx=0x7fb3e4009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:15.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.910+0000 7fb3eb400640 1 -- 172.21.15.6:0/2792987049 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4846+0+0 (secure 0 0 0) 0x7fb3e005ea80 con 0x7fb3f4151fe0 2024-03-22T12:56:15.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:15.928+0000 7fb3eb400640 1 -- 172.21.15.6:0/2792987049 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb3e0058770 con 0x7fb3f4151fe0 2024-03-22T12:56:16.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.132+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3f41ecc10 con 0x7fb3f4151fe0 2024-03-22T12:56:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.345+0000 7fb3eb400640 1 -- 172.21.15.6:0/2792987049 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3e005c810 con 0x7fb3f4151fe0 2024-03-22T12:56:16.345 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-22T12:56:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d406bb90 msgr2=0x7fb3d406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d406bb90 0x7fb3d406e050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009bd0 tx=0x7fb3e4009240 comp rx=0 tx=0).stop 2024-03-22T12:56:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3f4151fe0 msgr2=0x7fb3f414b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3f4151fe0 0x7fb3f414b7c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c370 tx=0x7fb3e000c830 comp rx=0 tx=0).stop 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 shutdown_connections 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d406bb90 0x7fb3d406e050 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f4158110 0x7fb3f41594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3f4151fe0 0x7fb3f414b7c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 --2- 172.21.15.6:0/2792987049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3f40ca720 0x7fb3f41efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.348+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 >> 172.21.15.6:0/2792987049 conn(0x7fb3f40714a0 msgr2=0x7fb3f4156540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.349+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 shutdown_connections 2024-03-22T12:56:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.349+0000 7fb3fa800640 1 -- 172.21.15.6:0/2792987049 wait complete. 2024-03-22T12:56:16.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:56:16.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.567+0000 7efdfa000640 1 Processor -- start 2024-03-22T12:56:16.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.567+0000 7efdfa000640 1 -- start start 2024-03-22T12:56:16.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.569+0000 7efdfa000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406fea0 0x7efdf406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:16.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.569+0000 7efdfa000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdf40c2ea0 0x7efdf40c3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:16.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.569+0000 7efdfa000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf40c3d30 con 0x7efdf406b080 2024-03-22T12:56:16.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.569+0000 7efdfa000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf40c3a50 con 0x7efdf40c2ea0 2024-03-22T12:56:16.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.569+0000 7efdfa000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf415a450 con 0x7efdf406fea0 2024-03-22T12:56:16.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.570+0000 7efdf2a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406fea0 0x7efdf406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:16.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.570+0000 7efdf3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdf40c2ea0 0x7efdf40c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:16.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.570+0000 7efdf2a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406fea0 0x7efdf406b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55176/0 (socket says 172.21.15.6:55176) 2024-03-22T12:56:16.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.570+0000 7efdf2a00640 1 -- 172.21.15.6:0/1330247762 learned_addr learned my addr 172.21.15.6:0/1330247762 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:16.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.574+0000 7efdf2a00640 1 -- 172.21.15.6:0/1330247762 >> v2:172.21.15.6:3300/0 conn(0x7efdf40c2ea0 msgr2=0x7efdf40c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:16.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.574+0000 7efdf2a00640 1 --2- 172.21.15.6:0/1330247762 >> v2:172.21.15.6:3300/0 conn(0x7efdf40c2ea0 0x7efdf40c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.574+0000 7efdf2a00640 1 -- 172.21.15.6:0/1330247762 >> v1:172.21.15.6:6789/0 conn(0x7efdf406b080 legacy=0x7efdf415d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:16.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.574+0000 7efdf2a00640 1 -- 172.21.15.6:0/1330247762 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf415a730 con 0x7efdf406fea0 2024-03-22T12:56:16.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.577+0000 7efdf2a00640 1 --2- 172.21.15.6:0/1330247762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406fea0 0x7efdf406b460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efde0009870 tx=0x7efde002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4512639b604a6940 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:16.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.582+0000 7efdf2000640 1 -- 172.21.15.6:0/1330247762 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde002f830 con 0x7efdf406fea0 2024-03-22T12:56:16.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.582+0000 7efdf2000640 1 -- 172.21.15.6:0/1330247762 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde002f9d0 con 0x7efdf406fea0 2024-03-22T12:56:16.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.582+0000 7efdf2000640 1 -- 172.21.15.6:0/1330247762 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde002fda0 con 0x7efdf406fea0 2024-03-22T12:56:16.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.582+0000 7efdfa000640 1 -- 172.21.15.6:0/1330247762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406fea0 msgr2=0x7efdf406b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:16.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.582+0000 7efdfa000640 1 --2- 172.21.15.6:0/1330247762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406fea0 0x7efdf406b460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efde0009870 tx=0x7efde002ba10 comp rx=0 tx=0).stop 2024-03-22T12:56:16.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.583+0000 7efdfa000640 1 -- 172.21.15.6:0/1330247762 shutdown_connections 2024-03-22T12:56:16.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.583+0000 7efdfa000640 1 --2- 172.21.15.6:0/1330247762 >> v2:172.21.15.6:3300/0 conn(0x7efdf40c2ea0 0x7efdf40c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.583+0000 7efdfa000640 1 --2- 172.21.15.6:0/1330247762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406fea0 0x7efdf406b460 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.583+0000 7efdfa000640 1 -- 172.21.15.6:0/1330247762 >> 172.21.15.6:0/1330247762 conn(0x7efdf40ccb50 msgr2=0x7efdf40ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:16.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.583+0000 7efdfa000640 1 -- 172.21.15.6:0/1330247762 shutdown_connections 2024-03-22T12:56:16.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.583+0000 7efdfa000640 1 -- 172.21.15.6:0/1330247762 wait complete. 2024-03-22T12:56:16.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.585+0000 7efdfa000640 1 Processor -- start 2024-03-22T12:56:16.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.586+0000 7efdfa000640 1 -- start start 2024-03-22T12:56:16.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.587+0000 7efdfa000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406b080 0x7efdf415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:16.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.587+0000 7efdfa000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf406fea0 0x7efdf414a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:16.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.587+0000 7efdfa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdf40c2ea0 0x7efdf4159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:16.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.587+0000 7efdfa000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf415e000 con 0x7efdf40c2ea0 2024-03-22T12:56:16.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.587+0000 7efdfa000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf415acf0 con 0x7efdf406fea0 2024-03-22T12:56:16.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.587+0000 7efdfa000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf415b210 con 0x7efdf406b080 2024-03-22T12:56:16.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.588+0000 7efdf3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406b080 0x7efdf415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:16.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.588+0000 7efdf2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf406fea0 0x7efdf414a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:16.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.588+0000 7efdf3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406b080 0x7efdf415b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55178/0 (socket says 172.21.15.6:55178) 2024-03-22T12:56:16.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.588+0000 7efdf3400640 1 -- 172.21.15.6:0/3207424322 learned_addr learned my addr 172.21.15.6:0/3207424322 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:16.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.588+0000 7efdf3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdf40c2ea0 0x7efdf4159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:16.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.591+0000 7efdf3400640 1 -- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf406fea0 msgr2=0x7efdf414a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:16.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.591+0000 7efdf3400640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf406fea0 0x7efdf414a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.592+0000 7efdf3400640 1 -- 172.21.15.6:0/3207424322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdf40c2ea0 msgr2=0x7efdf4159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:16.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.592+0000 7efdf3400640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdf40c2ea0 0x7efdf4159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:16.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.592+0000 7efdf3400640 1 -- 172.21.15.6:0/3207424322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf40c3f50 con 0x7efdf406b080 2024-03-22T12:56:16.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.594+0000 7efdf3400640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406b080 0x7efdf415b4c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efde400c430 tx=0x7efde400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.598+0000 7efdeb400640 1 -- 172.21.15.6:0/3207424322 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde40052d0 con 0x7efdf406b080 2024-03-22T12:56:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.598+0000 7efdeb400640 1 -- 172.21.15.6:0/3207424322 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4005490 con 0x7efdf406b080 2024-03-22T12:56:16.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.598+0000 7efdeb400640 1 -- 172.21.15.6:0/3207424322 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4007260 con 0x7efdf406b080 2024-03-22T12:56:16.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.598+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf40cc850 con 0x7efdf406b080 2024-03-22T12:56:16.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.599+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf40c44f0 con 0x7efdf406b080 2024-03-22T12:56:16.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.600+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf4152c80 con 0x7efdf406b080 2024-03-22T12:56:16.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.858+0000 7efdeb400640 1 -- 172.21.15.6:0/3207424322 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7efde4007400 con 0x7efdf406b080 2024-03-22T12:56:16.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.859+0000 7efdeb400640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efdd406bb90 0x7efdd406e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:16.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.859+0000 7efdeb400640 1 -- 172.21.15.6:0/3207424322 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efde4089ea0 con 0x7efdf406b080 2024-03-22T12:56:16.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.860+0000 7efdf2a00640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efdd406bb90 0x7efdd406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.861+0000 7efdf2a00640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efdd406bb90 0x7efdd406e050 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efde0036000 tx=0x7efde0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:16.870+0000 7efdeb400640 1 -- 172.21.15.6:0/3207424322 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efdf4152c80 con 0x7efdf406b080 2024-03-22T12:56:17.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.162+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf40756c0 con 0x7efdf406b080 2024-03-22T12:56:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.181+0000 7efdeb400640 1 -- 172.21.15.6:0/3207424322 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7efde4058850 con 0x7efdf406b080 2024-03-22T12:56:17.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:17.181 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-22T12:56:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 1 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efdd406bb90 msgr2=0x7efdd406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efdd406bb90 0x7efdd406e050 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efde0036000 tx=0x7efde0008030 comp rx=0 tx=0).stop 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406b080 msgr2=0x7efdf415b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406b080 0x7efdf415b4c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efde400c430 tx=0x7efde400c8f0 comp rx=0 tx=0).stop 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 shutdown_connections 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdf40c2ea0 0x7efdf4159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efdd406bb90 0x7efdd406e050 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf406fea0 0x7efdf414a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 --2- 172.21.15.6:0/3207424322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf406b080 0x7efdf415b4c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 >> 172.21.15.6:0/3207424322 conn(0x7efdf40ccb50 msgr2=0x7efdf414e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 shutdown_connections 2024-03-22T12:56:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:17.184+0000 7efdfa000640 1 -- 172.21.15.6:0/3207424322 wait complete. 2024-03-22T12:56:19.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:56:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.386+0000 7f9f1ec00640 1 Processor -- start 2024-03-22T12:56:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.386+0000 7f9f1ec00640 1 -- start start 2024-03-22T12:56:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.387+0000 7f9f1ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f180cc380 0x7f9f1806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.387+0000 7f9f1ec00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180ce8a0 0x7f9f1815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.388+0000 7f9f1ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180cee00 con 0x7f9f180cc760 2024-03-22T12:56:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.388+0000 7f9f1ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f180cf0c0 con 0x7f9f180cc380 2024-03-22T12:56:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.388+0000 7f9f1ec00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cf3a0 con 0x7f9f180ce8a0 2024-03-22T12:56:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.388+0000 7f9f1ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180ce8a0 0x7f9f1815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.389+0000 7f9f1d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f180cc380 0x7f9f1806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.389+0000 7f9f1ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180ce8a0 0x7f9f1815ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55190/0 (socket says 172.21.15.6:55190) 2024-03-22T12:56:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.389+0000 7f9f1ce00640 1 -- 172.21.15.6:0/1240111682 learned_addr learned my addr 172.21.15.6:0/1240111682 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:19.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.391+0000 7f9f1ce00640 1 -- 172.21.15.6:0/1240111682 >> v2:172.21.15.6:3300/0 conn(0x7f9f180cc380 msgr2=0x7f9f1806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:19.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.391+0000 7f9f1ce00640 1 --2- 172.21.15.6:0/1240111682 >> v2:172.21.15.6:3300/0 conn(0x7f9f180cc380 0x7f9f1806c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:19.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.391+0000 7f9f1ce00640 1 -- 172.21.15.6:0/1240111682 >> v1:172.21.15.6:6789/0 conn(0x7f9f180cc760 legacy=0x7f9f180cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:19.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.391+0000 7f9f1ce00640 1 -- 172.21.15.6:0/1240111682 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180cf680 con 0x7f9f180ce8a0 2024-03-22T12:56:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.393+0000 7f9f1ce00640 1 --2- 172.21.15.6:0/1240111682 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180ce8a0 0x7f9f1815ae30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009870 tx=0x7f9f0402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ace8abd50b424f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.398+0000 7f9f17e00640 1 -- 172.21.15.6:0/1240111682 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04035020 con 0x7f9f180ce8a0 2024-03-22T12:56:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.398+0000 7f9f17e00640 1 -- 172.21.15.6:0/1240111682 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f04031870 con 0x7f9f180ce8a0 2024-03-22T12:56:19.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.398+0000 7f9f17e00640 1 -- 172.21.15.6:0/1240111682 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04031c80 con 0x7f9f180ce8a0 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1240111682 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180ce8a0 msgr2=0x7f9f1815ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1240111682 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180ce8a0 0x7f9f1815ae30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009870 tx=0x7f9f0402b910 comp rx=0 tx=0).stop 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1240111682 shutdown_connections 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1240111682 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180ce8a0 0x7f9f1815ae30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1240111682 >> v2:172.21.15.6:3300/0 conn(0x7f9f180cc380 0x7f9f1806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1240111682 >> 172.21.15.6:0/1240111682 conn(0x7f9f180b1480 msgr2=0x7f9f180b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1240111682 shutdown_connections 2024-03-22T12:56:19.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.399+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1240111682 wait complete. 2024-03-22T12:56:19.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.401+0000 7f9f1ec00640 1 Processor -- start 2024-03-22T12:56:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.401+0000 7f9f1ec00640 1 -- start start 2024-03-22T12:56:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.402+0000 7f9f1ec00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180cc380 0x7f9f180c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.402+0000 7f9f1ec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f180cc760 0x7f9f180c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.402+0000 7f9f1ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f180ce8a0 0x7f9f181e59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.402+0000 7f9f1ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181549d0 con 0x7f9f180ce8a0 2024-03-22T12:56:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.402+0000 7f9f1ec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1815a910 con 0x7f9f180cc760 2024-03-22T12:56:19.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.402+0000 7f9f1ec00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180ccb90 con 0x7f9f180cc380 2024-03-22T12:56:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.403+0000 7f9f1d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180cc380 0x7f9f180c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.403+0000 7f9f1ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f180cc760 0x7f9f180c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.403+0000 7f9f1d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180cc380 0x7f9f180c81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55204/0 (socket says 172.21.15.6:55204) 2024-03-22T12:56:19.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.403+0000 7f9f1d800640 1 -- 172.21.15.6:0/1029463658 learned_addr learned my addr 172.21.15.6:0/1029463658 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:19.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.403+0000 7f9f1e200640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f180ce8a0 0x7f9f181e59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:19.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.407+0000 7f9f1d800640 1 -- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f180cc760 msgr2=0x7f9f180c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:19.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.407+0000 7f9f1d800640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f180cc760 0x7f9f180c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:19.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.407+0000 7f9f1d800640 1 -- 172.21.15.6:0/1029463658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f180ce8a0 msgr2=0x7f9f181e59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:19.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.407+0000 7f9f1d800640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f180ce8a0 0x7f9f181e59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:19.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.407+0000 7f9f1d800640 1 -- 172.21.15.6:0/1029463658 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180c93d0 con 0x7f9f180cc380 2024-03-22T12:56:19.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.409+0000 7f9f1d800640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180cc380 0x7f9f180c81d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00efc0 tx=0x7f9f0c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.414+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c018020 con 0x7f9f180cc380 2024-03-22T12:56:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.414+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c014480 con 0x7f9f180cc380 2024-03-22T12:56:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.414+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f180c8bb0 con 0x7f9f180cc380 2024-03-22T12:56:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.414+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c014850 con 0x7f9f180cc380 2024-03-22T12:56:19.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.414+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f180c99d0 con 0x7f9f180cc380 2024-03-22T12:56:19.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.415+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f18075650 con 0x7f9f180cc380 2024-03-22T12:56:19.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.439+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7f9f0c01b020 con 0x7f9f180cc380 2024-03-22T12:56:19.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.440+0000 7f9f15600640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ef806bb90 0x7f9ef806e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:19.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.440+0000 7f9f1ce00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ef806bb90 0x7f9ef806e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:19.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.441+0000 7f9f1ce00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ef806bb90 0x7f9ef806e050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9f04036000 tx=0x7f9f0403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:19.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.441+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9f0c05ffc0 con 0x7f9f180cc380 2024-03-22T12:56:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.713+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9f0c022030 con 0x7f9f180cc380 2024-03-22T12:56:19.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:19.997+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f180ce620 con 0x7f9f180cc380 2024-03-22T12:56:20.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.378+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f0c005d10 con 0x7f9f180cc380 2024-03-22T12:56:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.418+0000 7f9f15600640 1 -- 172.21.15.6:0/1029463658 <== mon.2 v2:172.21.15.92:3301/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f9f0c010040 con 0x7f9f180cc380 2024-03-22T12:56:20.418 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:20.419 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-22T12:56:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 1 2024-03-22T12:56:20.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ef806bb90 msgr2=0x7f9ef806e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:20.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ef806bb90 0x7f9ef806e050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9f04036000 tx=0x7f9f0403c040 comp rx=0 tx=0).stop 2024-03-22T12:56:20.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180cc380 msgr2=0x7f9f180c81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:20.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180cc380 0x7f9f180c81d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00efc0 tx=0x7f9f0c00c490 comp rx=0 tx=0).stop 2024-03-22T12:56:20.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 shutdown_connections 2024-03-22T12:56:20.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f180ce8a0 0x7f9f181e59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:20.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ef806bb90 0x7f9ef806e050 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:20.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f180cc760 0x7f9f180c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:20.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 --2- 172.21.15.6:0/1029463658 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f180cc380 0x7f9f180c81d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:20.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 >> 172.21.15.6:0/1029463658 conn(0x7f9f180b1480 msgr2=0x7f9f180af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:20.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 shutdown_connections 2024-03-22T12:56:20.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:20.421+0000 7f9f1ec00640 1 -- 172.21.15.6:0/1029463658 wait complete. 2024-03-22T12:56:20.820 INFO:tasks.ceph.mds.f.smithi092.stdout:starting mds.f at 2024-03-22T12:56:21.058 INFO:tasks.ceph.mds.d.smithi092.stdout:starting mds.d at 2024-03-22T12:56:21.107 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2024-03-22T12:56:21.627 INFO:tasks.ceph.mds.e.smithi006.stdout:starting mds.e at 2024-03-22T12:56:21.708 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-22T12:56:21.917 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-22T12:56:22.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:56:22.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.665+0000 7fbda4400640 1 Processor -- start 2024-03-22T12:56:22.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.665+0000 7fbda4400640 1 -- start start 2024-03-22T12:56:22.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.666+0000 7fbda4400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbd9c0c7640 0x7fbd9c15f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:22.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.666+0000 7fbda4400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c071dc0 0x7fbd9c06b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:22.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.666+0000 7fbda4400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c158dd0 con 0x7fbd9c1589f0 2024-03-22T12:56:22.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.666+0000 7fbda4400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c161c10 con 0x7fbd9c0c7640 2024-03-22T12:56:22.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.666+0000 7fbda4400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0c7b40 con 0x7fbd9c071dc0 2024-03-22T12:56:22.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.708+0000 7fbda2400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c071dc0 0x7fbd9c06b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:22.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.708+0000 7fbda2400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c071dc0 0x7fbd9c06b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55240/0 (socket says 172.21.15.6:55240) 2024-03-22T12:56:22.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.708+0000 7fbda2400640 1 -- 172.21.15.6:0/3445019238 learned_addr learned my addr 172.21.15.6:0/3445019238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:22.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.711+0000 7fbda2400640 1 -- 172.21.15.6:0/3445019238 >> v2:172.21.15.92:3300/0 conn(0x7fbd9c0c7640 msgr2=0x7fbd9c15f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:22.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.711+0000 7fbda2400640 1 --2- 172.21.15.6:0/3445019238 >> v2:172.21.15.92:3300/0 conn(0x7fbd9c0c7640 0x7fbd9c15f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:22.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.711+0000 7fbda2400640 1 -- 172.21.15.6:0/3445019238 >> v1:172.21.15.6:6789/0 conn(0x7fbd9c1589f0 legacy=0x7fbd9c162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:22.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.711+0000 7fbda2400640 1 -- 172.21.15.6:0/3445019238 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c0c6b10 con 0x7fbd9c071dc0 2024-03-22T12:56:22.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.714+0000 7fbda2400640 1 --2- 172.21.15.6:0/3445019238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c071dc0 0x7fbd9c06b080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009e50 tx=0x7fbd9402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a964b43981f340 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:22.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.718+0000 7fbd9be00640 1 -- 172.21.15.6:0/3445019238 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9402cb00 con 0x7fbd9c071dc0 2024-03-22T12:56:22.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.718+0000 7fbd9be00640 1 -- 172.21.15.6:0/3445019238 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9402cca0 con 0x7fbd9c071dc0 2024-03-22T12:56:22.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.718+0000 7fbd9be00640 1 -- 172.21.15.6:0/3445019238 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94034610 con 0x7fbd9c071dc0 2024-03-22T12:56:22.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.719+0000 7fbda4400640 1 -- 172.21.15.6:0/3445019238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c071dc0 msgr2=0x7fbd9c06b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:22.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.719+0000 7fbda4400640 1 --2- 172.21.15.6:0/3445019238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c071dc0 0x7fbd9c06b080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009e50 tx=0x7fbd9402bf60 comp rx=0 tx=0).stop 2024-03-22T12:56:22.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.719+0000 7fbda4400640 1 -- 172.21.15.6:0/3445019238 shutdown_connections 2024-03-22T12:56:22.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.719+0000 7fbda4400640 1 --2- 172.21.15.6:0/3445019238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c071dc0 0x7fbd9c06b080 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:22.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.719+0000 7fbda4400640 1 --2- 172.21.15.6:0/3445019238 >> v2:172.21.15.92:3300/0 conn(0x7fbd9c0c7640 0x7fbd9c15f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:22.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.719+0000 7fbda4400640 1 -- 172.21.15.6:0/3445019238 >> 172.21.15.6:0/3445019238 conn(0x7fbd9c0cb930 msgr2=0x7fbd9c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:22.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.720+0000 7fbda4400640 1 -- 172.21.15.6:0/3445019238 shutdown_connections 2024-03-22T12:56:22.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.720+0000 7fbda4400640 1 -- 172.21.15.6:0/3445019238 wait complete. 2024-03-22T12:56:22.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.722+0000 7fbda4400640 1 Processor -- start 2024-03-22T12:56:22.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.722+0000 7fbda4400640 1 -- start start 2024-03-22T12:56:22.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.723+0000 7fbda4400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd9c071dc0 0x7fbd9c0ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:22.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.723+0000 7fbda4400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd9c0c7640 0x7fbd9c15b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:22.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.723+0000 7fbda4400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c1589f0 0x7fbd9c15a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:22.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.723+0000 7fbda4400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c161e60 con 0x7fbd9c071dc0 2024-03-22T12:56:22.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.723+0000 7fbda4400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0c70d0 con 0x7fbd9c0c7640 2024-03-22T12:56:22.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.723+0000 7fbda4400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c162a90 con 0x7fbd9c1589f0 2024-03-22T12:56:22.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.723+0000 7fbda2400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c1589f0 0x7fbd9c15a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:22.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.724+0000 7fbda2400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c1589f0 0x7fbd9c15a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55250/0 (socket says 172.21.15.6:55250) 2024-03-22T12:56:22.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.724+0000 7fbda2400640 1 -- 172.21.15.6:0/4263655515 learned_addr learned my addr 172.21.15.6:0/4263655515 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:22.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.724+0000 7fbda1000640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd9c0c7640 0x7fbd9c15b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:22.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.728+0000 7fbda2400640 1 -- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd9c0c7640 msgr2=0x7fbd9c15b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:22.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.728+0000 7fbda2400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd9c0c7640 0x7fbd9c15b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:22.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.728+0000 7fbda2400640 1 -- 172.21.15.6:0/4263655515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd9c071dc0 msgr2=0x7fbd9c0ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:22.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.728+0000 7fbda2400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd9c071dc0 0x7fbd9c0ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:22.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.728+0000 7fbda2400640 1 -- 172.21.15.6:0/4263655515 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c0c50c0 con 0x7fbd9c1589f0 2024-03-22T12:56:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.731+0000 7fbda2400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c1589f0 0x7fbd9c15a2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009e50 tx=0x7fbd94004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:22.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.739+0000 7fbd99600640 1 -- 172.21.15.6:0/4263655515 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94043020 con 0x7fbd9c1589f0 2024-03-22T12:56:22.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.739+0000 7fbd99600640 1 -- 172.21.15.6:0/4263655515 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9402ce50 con 0x7fbd9c1589f0 2024-03-22T12:56:22.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.739+0000 7fbd99600640 1 -- 172.21.15.6:0/4263655515 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9403e670 con 0x7fbd9c1589f0 2024-03-22T12:56:22.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.739+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c0c53a0 con 0x7fbd9c1589f0 2024-03-22T12:56:22.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.739+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c159dd0 con 0x7fbd9c1589f0 2024-03-22T12:56:22.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.741+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c0c5570 con 0x7fbd9c1589f0 2024-03-22T12:56:22.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.762+0000 7fbd99600640 1 -- 172.21.15.6:0/4263655515 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbd94046020 con 0x7fbd9c1589f0 2024-03-22T12:56:22.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.764+0000 7fbd99600640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd7c06bc20 0x7fbd7c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:22.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.764+0000 7fbd99600640 1 -- 172.21.15.6:0/4263655515 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbd940ad1d0 con 0x7fbd9c1589f0 2024-03-22T12:56:22.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.764+0000 7fbda1a00640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd7c06bc20 0x7fbd7c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:22.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:22.765+0000 7fbda1a00640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd7c06bc20 0x7fbd7c06e0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbd90001560 tx=0x7fbd90007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:23.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.006+0000 7fbd99600640 1 -- 172.21.15.6:0/4263655515 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbd9404d030 con 0x7fbd9c1589f0 2024-03-22T12:56:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.260+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd9c0756c0 con 0x7fbd9c1589f0 2024-03-22T12:56:23.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.405+0000 7fbd99600640 1 -- 172.21.15.6:0/4263655515 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5262 (secure 0 0 0) 0x7fbd9c158dd0 con 0x7fbd9c1589f0 2024-03-22T12:56:23.406 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:23.406 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2825293724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2825293724},{"type":"v1","addr":"172.21.15.92:6837","nonce":2825293724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-22T12:56:23.406 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 3 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.409+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd7c06bc20 msgr2=0x7fbd7c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.409+0000 7fbda4400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd7c06bc20 0x7fbd7c06e0e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbd90001560 tx=0x7fbd90007990 comp rx=0 tx=0).stop 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c1589f0 msgr2=0x7fbd9c15a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c1589f0 0x7fbd9c15a2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009e50 tx=0x7fbd94004a40 comp rx=0 tx=0).stop 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 shutdown_connections 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd9c1589f0 0x7fbd9c15a2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd7c06bc20 0x7fbd7c06e0e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd9c0c7640 0x7fbd9c15b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 --2- 172.21.15.6:0/4263655515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd9c071dc0 0x7fbd9c0ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.410+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 >> 172.21.15.6:0/4263655515 conn(0x7fbd9c0cb930 msgr2=0x7fbd9c157c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:23.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.411+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 shutdown_connections 2024-03-22T12:56:23.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.411+0000 7fbda4400640 1 -- 172.21.15.6:0/4263655515 wait complete. 2024-03-22T12:56:23.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:23.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.686+0000 7fadff200640 1 Processor -- start 2024-03-22T12:56:23.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.686+0000 7fadff200640 1 -- start start 2024-03-22T12:56:23.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.688+0000 7fadff200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fae000cc380 0x7fae0006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.688+0000 7fadff200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae0015ae30 0x7fae000cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.688+0000 7fadff200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0015b260 con 0x7fae000ce8a0 2024-03-22T12:56:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.688+0000 7fadff200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae000cee00 con 0x7fae000cc380 2024-03-22T12:56:23.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.688+0000 7fadff200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae000cf0c0 con 0x7fae0015ae30 2024-03-22T12:56:23.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.689+0000 7fadfde00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fae000cc380 0x7fae0006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:23.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.690+0000 7fadfe800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae0015ae30 0x7fae000cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:23.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.690+0000 7fadfde00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fae000cc380 0x7fae0006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58656/0 (socket says 172.21.15.6:58656) 2024-03-22T12:56:23.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.690+0000 7fadfde00640 1 -- 172.21.15.6:0/915704201 learned_addr learned my addr 172.21.15.6:0/915704201 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.692+0000 7fadfde00640 1 -- 172.21.15.6:0/915704201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae0015ae30 msgr2=0x7fae000cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.692+0000 7fadfde00640 1 --2- 172.21.15.6:0/915704201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae0015ae30 0x7fae000cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.692+0000 7fadfde00640 1 -- 172.21.15.6:0/915704201 >> v1:172.21.15.6:6789/0 conn(0x7fae000ce8a0 legacy=0x7fae000cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:23.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.692+0000 7fadfde00640 1 -- 172.21.15.6:0/915704201 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000cf3a0 con 0x7fae000cc380 2024-03-22T12:56:23.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.742+0000 7fadfde00640 1 --2- 172.21.15.6:0/915704201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000cc380 0x7fae0006c230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009870 tx=0x7fadf002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78beed349a94b62 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.767+0000 7fadfca00640 1 -- 172.21.15.6:0/915704201 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0035020 con 0x7fae000cc380 2024-03-22T12:56:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.767+0000 7fadfca00640 1 -- 172.21.15.6:0/915704201 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0031640 con 0x7fae000cc380 2024-03-22T12:56:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.767+0000 7fadfca00640 1 -- 172.21.15.6:0/915704201 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf002d040 con 0x7fae000cc380 2024-03-22T12:56:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 -- 172.21.15.6:0/915704201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000cc380 msgr2=0x7fae0006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 --2- 172.21.15.6:0/915704201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000cc380 0x7fae0006c230 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009870 tx=0x7fadf002b8f0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 -- 172.21.15.6:0/915704201 shutdown_connections 2024-03-22T12:56:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 --2- 172.21.15.6:0/915704201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae0015ae30 0x7fae000cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 --2- 172.21.15.6:0/915704201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000cc380 0x7fae0006c230 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 -- 172.21.15.6:0/915704201 >> 172.21.15.6:0/915704201 conn(0x7fae000b1480 msgr2=0x7fae000b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 -- 172.21.15.6:0/915704201 shutdown_connections 2024-03-22T12:56:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.769+0000 7fadff200640 1 -- 172.21.15.6:0/915704201 wait complete. 2024-03-22T12:56:23.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.772+0000 7fadff200640 1 Processor -- start 2024-03-22T12:56:23.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.773+0000 7fadff200640 1 -- start start 2024-03-22T12:56:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.774+0000 7fadff200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae000cc380 0x7fae001e4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.774+0000 7fadff200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000ce8a0 0x7fae0014a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.774+0000 7fadff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae0015ae30 0x7fae0014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.774+0000 7fadff200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0015a910 con 0x7fae0015ae30 2024-03-22T12:56:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.774+0000 7fadff200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0015a630 con 0x7fae000ce8a0 2024-03-22T12:56:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.774+0000 7fadff200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae001549d0 con 0x7fae000cc380 2024-03-22T12:56:23.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.775+0000 7fadfd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000ce8a0 0x7fae0014a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:23.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.775+0000 7fadfde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae000cc380 0x7fae001e4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:23.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.775+0000 7fadfd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000ce8a0 0x7fae0014a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58662/0 (socket says 172.21.15.6:58662) 2024-03-22T12:56:23.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.775+0000 7fadfd400640 1 -- 172.21.15.6:0/2739968477 learned_addr learned my addr 172.21.15.6:0/2739968477 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.778+0000 7fadfde00640 1 -- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000ce8a0 msgr2=0x7fae0014a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.778+0000 7fadfde00640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000ce8a0 0x7fae0014a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.778+0000 7fadfde00640 1 -- 172.21.15.6:0/2739968477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae0015ae30 msgr2=0x7fae0014aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.778+0000 7fadfde00640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae0015ae30 0x7fae0014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.778+0000 7fadfde00640 1 -- 172.21.15.6:0/2739968477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0015cd70 con 0x7fae000cc380 2024-03-22T12:56:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.778+0000 7fadfd400640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000ce8a0 0x7fae0014a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:23.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.781+0000 7fadfde00640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae000cc380 0x7fae001e4520 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fadf0033000 tx=0x7fadf00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.786+0000 7fadf6000640 1 -- 172.21.15.6:0/2739968477 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0040070 con 0x7fae000cc380 2024-03-22T12:56:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.786+0000 7fadf6000640 1 -- 172.21.15.6:0/2739968477 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0039970 con 0x7fae000cc380 2024-03-22T12:56:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.786+0000 7fadf6000640 1 -- 172.21.15.6:0/2739968477 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0035040 con 0x7fae000cc380 2024-03-22T12:56:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.786+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae00151a70 con 0x7fae000cc380 2024-03-22T12:56:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.787+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0015b8c0 con 0x7fae000cc380 2024-03-22T12:56:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.788+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae001e4200 con 0x7fae000cc380 2024-03-22T12:56:23.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.808+0000 7fadf6000640 1 -- 172.21.15.6:0/2739968477 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fadf004b060 con 0x7fae000cc380 2024-03-22T12:56:23.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.810+0000 7fadf6000640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fade006bc20 0x7fade006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:23.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.811+0000 7fadf6000640 1 -- 172.21.15.6:0/2739968477 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fadf00ac180 con 0x7fae000cc380 2024-03-22T12:56:23.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.811+0000 7fadfd400640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fade006bc20 0x7fade006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:23.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:23.811+0000 7fadfd400640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fade006bc20 0x7fade006e0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fae0015e870 tx=0x7fadec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.054+0000 7fadf6000640 1 -- 172.21.15.6:0/2739968477 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fae001e4200 con 0x7fae000cc380 2024-03-22T12:56:24.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.265+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae00075650 con 0x7fae000cc380 2024-03-22T12:56:24.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.294+0000 7fadf6000640 1 -- 172.21.15.6:0/2739968477 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12788 (secure 0 0 0) 0x7fae0015b260 con 0x7fae000cc380 2024-03-22T12:56:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:24.295 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:18.546574+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fade006bc20 msgr2=0x7fade006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fade006bc20 0x7fade006e0e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fae0015e870 tx=0x7fadec009040 comp rx=0 tx=0).stop 2024-03-22T12:56:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae000cc380 msgr2=0x7fae001e4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae000cc380 0x7fae001e4520 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fadf0033000 tx=0x7fadf00046c0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 shutdown_connections 2024-03-22T12:56:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae0015ae30 0x7fae0014aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fade006bc20 0x7fade006e0e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae000ce8a0 0x7fae0014a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 --2- 172.21.15.6:0/2739968477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae000cc380 0x7fae001e4520 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 >> 172.21.15.6:0/2739968477 conn(0x7fae000b1480 msgr2=0x7fae00153c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 shutdown_connections 2024-03-22T12:56:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.297+0000 7fadff200640 1 -- 172.21.15.6:0/2739968477 wait complete. 2024-03-22T12:56:24.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T12:56:14.196164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T12:56:24.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T12:56:24.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.545+0000 7f556b200640 1 Processor -- start 2024-03-22T12:56:24.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.545+0000 7f556b200640 1 -- start start 2024-03-22T12:56:24.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.547+0000 7f556b200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c151280 0x7f556c15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:24.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.547+0000 7f556b200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f556c0c5ba0 0x7f556c0c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:24.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.547+0000 7f556b200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0c2ae0 con 0x7f556c0c8200 2024-03-22T12:56:24.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.547+0000 7f556b200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0c5710 con 0x7f556c0c5ba0 2024-03-22T12:56:24.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.547+0000 7f556b200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0c3400 con 0x7f556c151280 2024-03-22T12:56:24.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.547+0000 7f5569400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f556c0c5ba0 0x7f556c0c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:24.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.548+0000 7f5569400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f556c0c5ba0 0x7f556c0c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58672/0 (socket says 172.21.15.6:58672) 2024-03-22T12:56:24.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.548+0000 7f5569400640 1 -- 172.21.15.6:0/3581676178 learned_addr learned my addr 172.21.15.6:0/3581676178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:24.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.549+0000 7f5569e00640 1 --2- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c151280 0x7f556c15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:24.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.551+0000 7f5569400640 1 -- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c151280 msgr2=0x7f556c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:24.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.551+0000 7f5569400640 1 --2- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c151280 0x7f556c15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.551+0000 7f5569400640 1 -- 172.21.15.6:0/3581676178 >> v1:172.21.15.92:6789/0 conn(0x7f556c0c8200 legacy=0x7f556c0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:24.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.551+0000 7f5569400640 1 -- 172.21.15.6:0/3581676178 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c0c36e0 con 0x7f556c0c5ba0 2024-03-22T12:56:24.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.555+0000 7f5569400640 1 --2- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c0c7440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5554009870 tx=0x7f555402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e553f2c6bf837e3f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.559+0000 7f5568a00640 1 -- 172.21.15.6:0/3581676178 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554035020 con 0x7f556c0c5ba0 2024-03-22T12:56:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.559+0000 7f5568a00640 1 -- 172.21.15.6:0/3581676178 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5554031640 con 0x7f556c0c5ba0 2024-03-22T12:56:24.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.559+0000 7f5568a00640 1 -- 172.21.15.6:0/3581676178 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555402d040 con 0x7f556c0c5ba0 2024-03-22T12:56:24.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.559+0000 7f556b200640 1 -- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 msgr2=0x7f556c0c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:24.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.559+0000 7f556b200640 1 --2- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c0c7440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5554009870 tx=0x7f555402b8f0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.560+0000 7f556b200640 1 -- 172.21.15.6:0/3581676178 shutdown_connections 2024-03-22T12:56:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.560+0000 7f556b200640 1 --2- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c0c7440 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.560+0000 7f556b200640 1 --2- 172.21.15.6:0/3581676178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c151280 0x7f556c15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.560+0000 7f556b200640 1 -- 172.21.15.6:0/3581676178 >> 172.21.15.6:0/3581676178 conn(0x7f556c06f0e0 msgr2=0x7f556c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.560+0000 7f556b200640 1 -- 172.21.15.6:0/3581676178 shutdown_connections 2024-03-22T12:56:24.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.560+0000 7f556b200640 1 -- 172.21.15.6:0/3581676178 wait complete. 2024-03-22T12:56:24.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.562+0000 7f556b200640 1 Processor -- start 2024-03-22T12:56:24.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.562+0000 7f556b200640 1 -- start start 2024-03-22T12:56:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.563+0000 7f556b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.563+0000 7f556b200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c0c8200 0x7f556c142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.563+0000 7f556b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f556c151280 0x7f556c153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.563+0000 7f556b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c151660 con 0x7f556c151280 2024-03-22T12:56:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.563+0000 7f556b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0c53a0 con 0x7f556c0c5ba0 2024-03-22T12:56:24.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.563+0000 7f556b200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0c39c0 con 0x7f556c0c8200 2024-03-22T12:56:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.564+0000 7f5569e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.564+0000 7f5569400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c0c8200 0x7f556c142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.564+0000 7f556a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f556c151280 0x7f556c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.564+0000 7f5569e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58674/0 (socket says 172.21.15.6:58674) 2024-03-22T12:56:24.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.564+0000 7f5569e00640 1 -- 172.21.15.6:0/850186563 learned_addr learned my addr 172.21.15.6:0/850186563 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:24.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.567+0000 7f5569e00640 1 -- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c0c8200 msgr2=0x7f556c142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:24.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.567+0000 7f5569e00640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c0c8200 0x7f556c142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.567+0000 7f5569e00640 1 -- 172.21.15.6:0/850186563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f556c151280 msgr2=0x7f556c153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:24.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.567+0000 7f5569e00640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f556c151280 0x7f556c153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:24.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.567+0000 7f5569e00640 1 -- 172.21.15.6:0/850186563 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c1416f0 con 0x7f556c0c5ba0 2024-03-22T12:56:24.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.567+0000 7f5569400640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c0c8200 0x7f556c142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:24.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.570+0000 7f5569e00640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c141ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f555c00efc0 tx=0x7f555c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.573+0000 7f5562000640 1 -- 172.21.15.6:0/850186563 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c018020 con 0x7f556c0c5ba0 2024-03-22T12:56:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.573+0000 7f5562000640 1 -- 172.21.15.6:0/850186563 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c014480 con 0x7f556c0c5ba0 2024-03-22T12:56:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.573+0000 7f5562000640 1 -- 172.21.15.6:0/850186563 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c014850 con 0x7f556c0c5ba0 2024-03-22T12:56:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.573+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c1efe30 con 0x7f556c0c5ba0 2024-03-22T12:56:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.574+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c1f03d0 con 0x7f556c0c5ba0 2024-03-22T12:56:24.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.575+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c14c0a0 con 0x7f556c0c5ba0 2024-03-22T12:56:24.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.594+0000 7f5562000640 1 -- 172.21.15.6:0/850186563 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f555c005d10 con 0x7f556c0c5ba0 2024-03-22T12:56:24.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.595+0000 7f5562000640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554806bc50 0x7f554806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:24.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.595+0000 7f5562000640 1 -- 172.21.15.6:0/850186563 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f555c08b880 con 0x7f556c0c5ba0 2024-03-22T12:56:24.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.595+0000 7f5569400640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554806bc50 0x7f554806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:24.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.596+0000 7f5569400640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554806bc50 0x7f554806e110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5554036000 tx=0x7f555403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:24.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:24.837+0000 7f5562000640 1 -- 172.21.15.6:0/850186563 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f556c14c0a0 con 0x7f556c0c5ba0 2024-03-22T12:56:25.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.053+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f556c075650 con 0x7f556c0c5ba0 2024-03-22T12:56:25.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.223+0000 7f5562000640 1 -- 172.21.15.6:0/850186563 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f555c022030 con 0x7f556c0c5ba0 2024-03-22T12:56:25.226 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554806bc50 msgr2=0x7f554806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554806bc50 0x7f554806e110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5554036000 tx=0x7f555403c040 comp rx=0 tx=0).stop 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 msgr2=0x7f556c141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c141ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f555c00efc0 tx=0x7f555c00c490 comp rx=0 tx=0).stop 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 shutdown_connections 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f556c151280 0x7f556c153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556c0c8200 0x7f556c142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554806bc50 0x7f554806e110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 --2- 172.21.15.6:0/850186563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556c0c5ba0 0x7f556c141ae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.226+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 >> 172.21.15.6:0/850186563 conn(0x7f556c06f0e0 msgr2=0x7f556c0c7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:25.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.227+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 shutdown_connections 2024-03-22T12:56:25.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.227+0000 7f556b200640 1 -- 172.21.15.6:0/850186563 wait complete. 2024-03-22T12:56:25.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-22T12:56:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.477+0000 7fabda800640 1 Processor -- start 2024-03-22T12:56:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.477+0000 7fabda800640 1 -- start start 2024-03-22T12:56:25.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.478+0000 7fabda800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd4151010 0x7fabd40c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:25.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.478+0000 7fabda800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd40c56b0 con 0x7fabd415a4a0 2024-03-22T12:56:25.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.478+0000 7fabda800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd40c6d70 con 0x7fabd415a8e0 2024-03-22T12:56:25.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.478+0000 7fabda800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd40c2d90 con 0x7fabd4151010 2024-03-22T12:56:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.479+0000 7fabd9e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fabd415a8e0 0x7fabd40725b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:46518/0 (socket says 172.21.15.6:46518) 2024-03-22T12:56:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.479+0000 7fabd8a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fabd415a4a0 0x7fabd40c6660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37562/0 (socket says 172.21.15.6:37562) 2024-03-22T12:56:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.479+0000 7fabd9e00640 1 -- 172.21.15.6:0/2605226543 learned_addr learned my addr 172.21.15.6:0/2605226543 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.479+0000 7fabd9400640 1 --2- 172.21.15.6:0/2605226543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd4151010 0x7fabd40c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.482+0000 7fabd9400640 1 -- 172.21.15.6:0/2605226543 >> v1:172.21.15.92:6789/0 conn(0x7fabd415a8e0 legacy=0x7fabd40725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.482+0000 7fabd3e00640 1 -- 172.21.15.6:0/2605226543 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262590111 0 0) 0x7fabd40c6d70 con 0x7fabd415a8e0 2024-03-22T12:56:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.482+0000 7fabd9400640 1 -- 172.21.15.6:0/2605226543 >> v1:172.21.15.6:6789/0 conn(0x7fabd415a4a0 legacy=0x7fabd40c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.482+0000 7fabd9400640 1 -- 172.21.15.6:0/2605226543 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd40c5940 con 0x7fabd4151010 2024-03-22T12:56:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.484+0000 7fabd9400640 1 --2- 172.21.15.6:0/2605226543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd4151010 0x7fabd40c5170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fabc80097d0 tx=0x7fabc802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=181be7fe3fc8a573 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:25.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.488+0000 7fabd3e00640 1 -- 172.21.15.6:0/2605226543 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8035020 con 0x7fabd4151010 2024-03-22T12:56:25.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.488+0000 7fabd3e00640 1 -- 172.21.15.6:0/2605226543 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc8031800 con 0x7fabd4151010 2024-03-22T12:56:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.488+0000 7fabd3e00640 1 -- 172.21.15.6:0/2605226543 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8031bd0 con 0x7fabd4151010 2024-03-22T12:56:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.488+0000 7fabda800640 1 -- 172.21.15.6:0/2605226543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd4151010 msgr2=0x7fabd40c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.488+0000 7fabda800640 1 --2- 172.21.15.6:0/2605226543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd4151010 0x7fabd40c5170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fabc80097d0 tx=0x7fabc802b8a0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.489+0000 7fabda800640 1 -- 172.21.15.6:0/2605226543 shutdown_connections 2024-03-22T12:56:25.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.489+0000 7fabda800640 1 --2- 172.21.15.6:0/2605226543 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd4151010 0x7fabd40c5170 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.489+0000 7fabda800640 1 -- 172.21.15.6:0/2605226543 >> 172.21.15.6:0/2605226543 conn(0x7fabd40cb890 msgr2=0x7fabd40cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:25.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.489+0000 7fabda800640 1 -- 172.21.15.6:0/2605226543 shutdown_connections 2024-03-22T12:56:25.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.489+0000 7fabda800640 1 -- 172.21.15.6:0/2605226543 wait complete. 2024-03-22T12:56:25.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.491+0000 7fabda800640 1 Processor -- start 2024-03-22T12:56:25.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.491+0000 7fabda800640 1 -- start start 2024-03-22T12:56:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.492+0000 7fabda800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabd4151010 0x7fabd41e3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.492+0000 7fabda800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd415a4a0 0x7fabd41e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.492+0000 7fabda800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabd415a8e0 0x7fabd41efc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.492+0000 7fabda800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd40c32c0 con 0x7fabd4151010 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.492+0000 7fabda800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd40c2fe0 con 0x7fabd415a8e0 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.492+0000 7fabda800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd40c35a0 con 0x7fabd415a4a0 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.493+0000 7fabd8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd415a4a0 0x7fabd41e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.493+0000 7fabd9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabd415a8e0 0x7fabd41efc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.493+0000 7fabd8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd415a4a0 0x7fabd41e21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50800/0 (socket says 172.21.15.6:50800) 2024-03-22T12:56:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.493+0000 7fabd8a00640 1 -- 172.21.15.6:0/174872623 learned_addr learned my addr 172.21.15.6:0/174872623 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.493+0000 7fabd9400640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabd4151010 0x7fabd41e3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.497+0000 7fabd9e00640 1 -- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd415a4a0 msgr2=0x7fabd41e21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.497+0000 7fabd9e00640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd415a4a0 0x7fabd41e21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.497+0000 7fabd9e00640 1 -- 172.21.15.6:0/174872623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabd4151010 msgr2=0x7fabd41e3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.497+0000 7fabd9e00640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabd4151010 0x7fabd41e3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.497+0000 7fabd9e00640 1 -- 172.21.15.6:0/174872623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd41e3df0 con 0x7fabd415a8e0 2024-03-22T12:56:25.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.500+0000 7fabd9e00640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabd415a8e0 0x7fabd41efc90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00c730 tx=0x7fabcc00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.503+0000 7fabd1600640 1 -- 172.21.15.6:0/174872623 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc018070 con 0x7fabd415a8e0 2024-03-22T12:56:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.503+0000 7fabd1600640 1 -- 172.21.15.6:0/174872623 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc00ce40 con 0x7fabd415a8e0 2024-03-22T12:56:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.503+0000 7fabd1600640 1 -- 172.21.15.6:0/174872623 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00a880 con 0x7fabd415a8e0 2024-03-22T12:56:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.503+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd41f40d0 con 0x7fabd415a8e0 2024-03-22T12:56:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.504+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd41f3310 con 0x7fabd415a8e0 2024-03-22T12:56:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.505+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd4075650 con 0x7fabd415a8e0 2024-03-22T12:56:25.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.524+0000 7fabd1600640 1 -- 172.21.15.6:0/174872623 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fabcc01b070 con 0x7fabd415a8e0 2024-03-22T12:56:25.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.526+0000 7fabd1600640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabb406bc20 0x7fabb406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:25.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.526+0000 7fabd9400640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabb406bc20 0x7fabb406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:25.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.527+0000 7fabd9400640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabb406bc20 0x7fabb406e0e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fabc8036000 tx=0x7fabc803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.757+0000 7fabd1600640 1 -- 172.21.15.6:0/174872623 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fabcc05ecb0 con 0x7fabd415a8e0 2024-03-22T12:56:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.768+0000 7fabd1600640 1 -- 172.21.15.6:0/174872623 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fabcc022020 con 0x7fabd415a8e0 2024-03-22T12:56:25.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.991+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fabd40c6360 con 0x7fabd415a8e0 2024-03-22T12:56:25.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.997+0000 7fabd1600640 1 -- 172.21.15.6:0/174872623 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+65 (secure 0 0 0) 0x7fabcc058990 con 0x7fabd415a8e0 2024-03-22T12:56:25.997 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:25.997 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-22T12:56:25.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-22T12:56:25.997 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-22T12:56:25.997 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-22T12:56:25.997 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-22T12:56:25.997 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-22T12:56:25.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.999+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabb406bc20 msgr2=0x7fabb406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.999+0000 7fabda800640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabb406bc20 0x7fabb406e0e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fabc8036000 tx=0x7fabc803c040 comp rx=0 tx=0).stop 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.999+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabd415a8e0 msgr2=0x7fabd41efc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:25.999+0000 7fabda800640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabd415a8e0 0x7fabd41efc90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00c730 tx=0x7fabcc00cbf0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 shutdown_connections 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabb406bc20 0x7fabb406e0e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabd415a8e0 0x7fabd41efc90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabd415a4a0 0x7fabd41e21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 --2- 172.21.15.6:0/174872623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabd4151010 0x7fabd41e3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 >> 172.21.15.6:0/174872623 conn(0x7fabd40cb890 msgr2=0x7fabd4072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 shutdown_connections 2024-03-22T12:56:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.000+0000 7fabda800640 1 -- 172.21.15.6:0/174872623 wait complete. 2024-03-22T12:56:26.017 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-22T12:56:26.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-22T12:56:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.251+0000 7ff868c00640 1 Processor -- start 2024-03-22T12:56:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.251+0000 7ff868c00640 1 -- start start 2024-03-22T12:56:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.253+0000 7ff868c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.253+0000 7ff868c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8640ce520 0x7ff86406c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.253+0000 7ff868c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640cc9b0 con 0x7ff8640cea70 2024-03-22T12:56:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.253+0000 7ff868c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86415aca0 con 0x7ff8640ce520 2024-03-22T12:56:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.253+0000 7ff868c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640ccc90 con 0x7ff8641549a0 2024-03-22T12:56:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.254+0000 7ff862800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8640ce520 0x7ff86406c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.254+0000 7ff861e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.254+0000 7ff862800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8640ce520 0x7ff86406c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58702/0 (socket says 172.21.15.6:58702) 2024-03-22T12:56:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.254+0000 7ff862800640 1 -- 172.21.15.6:0/141025571 learned_addr learned my addr 172.21.15.6:0/141025571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:26.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.256+0000 7ff861e00640 1 -- 172.21.15.6:0/141025571 >> v2:172.21.15.92:3300/0 conn(0x7ff8640ce520 msgr2=0x7ff86406c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:26.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.256+0000 7ff861e00640 1 --2- 172.21.15.6:0/141025571 >> v2:172.21.15.92:3300/0 conn(0x7ff8640ce520 0x7ff86406c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.256+0000 7ff861e00640 1 -- 172.21.15.6:0/141025571 >> v1:172.21.15.92:6789/0 conn(0x7ff8640cea70 legacy=0x7ff8640cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:26.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.256+0000 7ff861e00640 1 -- 172.21.15.6:0/141025571 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86415a650 con 0x7ff8641549a0 2024-03-22T12:56:26.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.261+0000 7ff861e00640 1 --2- 172.21.15.6:0/141025571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415b130 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8540097a0 tx=0x7ff85402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c663415c23265494 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.308+0000 7ff860a00640 1 -- 172.21.15.6:0/141025571 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854035020 con 0x7ff8641549a0 2024-03-22T12:56:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.308+0000 7ff860a00640 1 -- 172.21.15.6:0/141025571 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff854031830 con 0x7ff8641549a0 2024-03-22T12:56:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.308+0000 7ff860a00640 1 -- 172.21.15.6:0/141025571 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854031c00 con 0x7ff8641549a0 2024-03-22T12:56:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.309+0000 7ff868c00640 1 -- 172.21.15.6:0/141025571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 msgr2=0x7ff86415b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.309+0000 7ff868c00640 1 --2- 172.21.15.6:0/141025571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415b130 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8540097a0 tx=0x7ff85402b8d0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.310+0000 7ff868c00640 1 -- 172.21.15.6:0/141025571 shutdown_connections 2024-03-22T12:56:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.310+0000 7ff868c00640 1 --2- 172.21.15.6:0/141025571 >> v2:172.21.15.92:3300/0 conn(0x7ff8640ce520 0x7ff86406c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.310+0000 7ff868c00640 1 --2- 172.21.15.6:0/141025571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415b130 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.310+0000 7ff868c00640 1 -- 172.21.15.6:0/141025571 >> 172.21.15.6:0/141025571 conn(0x7ff8640b1700 msgr2=0x7ff8640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.310+0000 7ff868c00640 1 -- 172.21.15.6:0/141025571 shutdown_connections 2024-03-22T12:56:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.310+0000 7ff868c00640 1 -- 172.21.15.6:0/141025571 wait complete. 2024-03-22T12:56:26.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.313+0000 7ff868c00640 1 Processor -- start 2024-03-22T12:56:26.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.313+0000 7ff868c00640 1 -- start start 2024-03-22T12:56:26.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.315+0000 7ff868c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8640ce520 0x7ff86415c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:26.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.316+0000 7ff868c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8640cea70 0x7ff8640b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:26.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.316+0000 7ff868c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.316+0000 7ff868c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640cd430 con 0x7ff8640cea70 2024-03-22T12:56:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.316+0000 7ff868c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640ccf70 con 0x7ff8640ce520 2024-03-22T12:56:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.316+0000 7ff868c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640cd5b0 con 0x7ff8641549a0 2024-03-22T12:56:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.316+0000 7ff861e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8640ce520 0x7ff86415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:26.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.316+0000 7ff862800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.317+0000 7ff862800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50816/0 (socket says 172.21.15.6:50816) 2024-03-22T12:56:26.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.317+0000 7ff862800640 1 -- 172.21.15.6:0/2604351433 learned_addr learned my addr 172.21.15.6:0/2604351433 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.320+0000 7ff861e00640 1 -- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 msgr2=0x7ff86415e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.320+0000 7ff861e00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.321+0000 7ff861e00640 1 -- 172.21.15.6:0/2604351433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8640cea70 msgr2=0x7ff8640b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:26.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.321+0000 7ff861e00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8640cea70 0x7ff8640b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:26.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.321+0000 7ff861e00640 1 -- 172.21.15.6:0/2604351433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8641f1520 con 0x7ff8640ce520 2024-03-22T12:56:26.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.321+0000 7ff862800640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.323+0000 7ff861e00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8640ce520 0x7ff86415c550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff854033000 tx=0x7ff8540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.326+0000 7ff85a000640 1 -- 172.21.15.6:0/2604351433 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854041070 con 0x7ff8640ce520 2024-03-22T12:56:26.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.326+0000 7ff85a000640 1 -- 172.21.15.6:0/2604351433 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85403a9d0 con 0x7ff8640ce520 2024-03-22T12:56:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.327+0000 7ff85a000640 1 -- 172.21.15.6:0/2604351433 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854035040 con 0x7ff8640ce520 2024-03-22T12:56:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.327+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86415b7d0 con 0x7ff8640ce520 2024-03-22T12:56:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.327+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86415ba70 con 0x7ff8640ce520 2024-03-22T12:56:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.329+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8640757d0 con 0x7ff8640ce520 2024-03-22T12:56:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.578+0000 7ff85a000640 1 -- 172.21.15.6:0/2604351433 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff85404c060 con 0x7ff8640ce520 2024-03-22T12:56:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.580+0000 7ff85a000640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff84406bc50 0x7ff84406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.581+0000 7ff85a000640 1 -- 172.21.15.6:0/2604351433 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff8540ac130 con 0x7ff8640ce520 2024-03-22T12:56:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.581+0000 7ff861400640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff84406bc50 0x7ff84406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:26.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.582+0000 7ff861400640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff84406bc50 0x7ff84406e110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff864151ee0 tx=0x7ff850006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:26.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.593+0000 7ff85a000640 1 -- 172.21.15.6:0/2604351433 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff85407b0e0 con 0x7ff8640ce520 2024-03-22T12:56:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:26.825+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff86415a430 con 0x7ff8640ce520 2024-03-22T12:56:27.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.641+0000 7ff85a000640 1 -- 172.21.15.6:0/2604351433 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff85407a9e0 con 0x7ff8640ce520 2024-03-22T12:56:27.642 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-22T12:56:27.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff84406bc50 msgr2=0x7ff84406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:27.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff84406bc50 0x7ff84406e110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff864151ee0 tx=0x7ff850006ba0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8640ce520 msgr2=0x7ff86415c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:27.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8640ce520 0x7ff86415c550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff854033000 tx=0x7ff8540046c0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 shutdown_connections 2024-03-22T12:56:27.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8641549a0 0x7ff86415e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8640cea70 0x7ff8640b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff84406bc50 0x7ff84406e110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 --2- 172.21.15.6:0/2604351433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8640ce520 0x7ff86415c550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 >> 172.21.15.6:0/2604351433 conn(0x7ff8640b1700 msgr2=0x7ff8640cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:27.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 shutdown_connections 2024-03-22T12:56:27.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.645+0000 7ff868c00640 1 -- 172.21.15.6:0/2604351433 wait complete. 2024-03-22T12:56:27.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-22T12:56:27.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.878+0000 7f3c91400640 1 Processor -- start 2024-03-22T12:56:27.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.879+0000 7f3c91400640 1 -- start start 2024-03-22T12:56:27.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.880+0000 7f3c91400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c158040 0x7f3c8c0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:27.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.880+0000 7f3c91400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c0c2d70 con 0x7f3c8c152410 2024-03-22T12:56:27.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.880+0000 7f3c91400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c1585a0 con 0x7f3c8c0cab10 2024-03-22T12:56:27.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.880+0000 7f3c91400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c15b630 con 0x7f3c8c158040 2024-03-22T12:56:27.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.881+0000 7f3c89e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c158040 0x7f3c8c0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:27.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.881+0000 7f3c8b200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3c8c0cab10 0x7f3c8c0c8860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:46538/0 (socket says 172.21.15.6:46538) 2024-03-22T12:56:27.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.881+0000 7f3c8b200640 1 -- 172.21.15.6:0/3136749748 learned_addr learned my addr 172.21.15.6:0/3136749748 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:27.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.884+0000 7f3c89e00640 1 -- 172.21.15.6:0/3136749748 >> v1:172.21.15.92:6789/0 conn(0x7f3c8c0cab10 legacy=0x7f3c8c0c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:27.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.884+0000 7f3c89e00640 1 -- 172.21.15.6:0/3136749748 >> v1:172.21.15.6:6789/0 conn(0x7f3c8c152410 legacy=0x7f3c8c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:27.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.884+0000 7f3c89e00640 1 -- 172.21.15.6:0/3136749748 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c158880 con 0x7f3c8c158040 2024-03-22T12:56:27.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.886+0000 7f3c89e00640 1 --2- 172.21.15.6:0/3136749748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c158040 0x7f3c8c0ca5d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c780097d0 tx=0x7f3c7802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3111103fea6bc06 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:27.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c89400640 1 -- 172.21.15.6:0/3136749748 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78035020 con 0x7f3c8c158040 2024-03-22T12:56:27.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c89400640 1 -- 172.21.15.6:0/3136749748 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c78031890 con 0x7f3c8c158040 2024-03-22T12:56:27.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c89400640 1 -- 172.21.15.6:0/3136749748 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78031c60 con 0x7f3c8c158040 2024-03-22T12:56:27.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c91400640 1 -- 172.21.15.6:0/3136749748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c158040 msgr2=0x7f3c8c0ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:27.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c91400640 1 --2- 172.21.15.6:0/3136749748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c158040 0x7f3c8c0ca5d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c780097d0 tx=0x7f3c7802b930 comp rx=0 tx=0).stop 2024-03-22T12:56:27.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c91400640 1 -- 172.21.15.6:0/3136749748 shutdown_connections 2024-03-22T12:56:27.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c91400640 1 --2- 172.21.15.6:0/3136749748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c158040 0x7f3c8c0ca5d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.890+0000 7f3c91400640 1 -- 172.21.15.6:0/3136749748 >> 172.21.15.6:0/3136749748 conn(0x7f3c8c071410 msgr2=0x7f3c8c0717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:27.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.891+0000 7f3c91400640 1 -- 172.21.15.6:0/3136749748 shutdown_connections 2024-03-22T12:56:27.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.891+0000 7f3c91400640 1 -- 172.21.15.6:0/3136749748 wait complete. 2024-03-22T12:56:27.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.892+0000 7f3c91400640 1 Processor -- start 2024-03-22T12:56:27.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.892+0000 7f3c91400640 1 -- start start 2024-03-22T12:56:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.893+0000 7f3c91400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c8c0cab10 0x7f3c8c0b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.893+0000 7f3c91400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c152410 0x7f3c8c1ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.893+0000 7f3c91400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c8c158040 0x7f3c8c1ed800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.893+0000 7f3c91400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c158e40 con 0x7f3c8c0cab10 2024-03-22T12:56:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.893+0000 7f3c91400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c0ca280 con 0x7f3c8c158040 2024-03-22T12:56:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.893+0000 7f3c91400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c15b880 con 0x7f3c8c152410 2024-03-22T12:56:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.894+0000 7f3c89e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c152410 0x7f3c8c1ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.894+0000 7f3c8b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c8c158040 0x7f3c8c1ed800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.894+0000 7f3c8a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c8c0cab10 0x7f3c8c0b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.894+0000 7f3c89e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c152410 0x7f3c8c1ed2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50834/0 (socket says 172.21.15.6:50834) 2024-03-22T12:56:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.894+0000 7f3c8b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c8c158040 0x7f3c8c1ed800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58716/0 (socket says 172.21.15.6:58716) 2024-03-22T12:56:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.894+0000 7f3c89e00640 1 -- 172.21.15.6:0/2514708032 learned_addr learned my addr 172.21.15.6:0/2514708032 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.898+0000 7f3c8b200640 1 -- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c152410 msgr2=0x7f3c8c1ed2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.898+0000 7f3c8b200640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c152410 0x7f3c8c1ed2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.898+0000 7f3c8b200640 1 -- 172.21.15.6:0/2514708032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c8c0cab10 msgr2=0x7f3c8c0b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.898+0000 7f3c8b200640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c8c0cab10 0x7f3c8c0b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.898+0000 7f3c8b200640 1 -- 172.21.15.6:0/2514708032 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c0cce30 con 0x7f3c8c158040 2024-03-22T12:56:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.900+0000 7f3c8b200640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c8c158040 0x7f3c8c1ed800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400c3c0 tx=0x7f3c8400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:27.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.904+0000 7f3c82a00640 1 -- 172.21.15.6:0/2514708032 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c84017070 con 0x7f3c8c158040 2024-03-22T12:56:27.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.904+0000 7f3c82a00640 1 -- 172.21.15.6:0/2514708032 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c84005ce0 con 0x7f3c8c158040 2024-03-22T12:56:27.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.904+0000 7f3c82a00640 1 -- 172.21.15.6:0/2514708032 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c840072d0 con 0x7f3c8c158040 2024-03-22T12:56:27.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.904+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c1ef610 con 0x7f3c8c158040 2024-03-22T12:56:27.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.904+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c1efbd0 con 0x7f3c8c158040 2024-03-22T12:56:27.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.905+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c0757d0 con 0x7f3c8c158040 2024-03-22T12:56:27.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.924+0000 7f3c82a00640 1 -- 172.21.15.6:0/2514708032 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3c8401a070 con 0x7f3c8c158040 2024-03-22T12:56:27.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.925+0000 7f3c82a00640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:27.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.925+0000 7f3c8a800640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:27.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.926+0000 7f3c8a800640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c001f10 tx=0x7f3c7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:27.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:27.926+0000 7f3c82a00640 1 -- 172.21.15.6:0/2514708032 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3c84056720 con 0x7f3c8c158040 2024-03-22T12:56:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.163+0000 7f3c82a00640 1 -- 172.21.15.6:0/2514708032 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3c84058680 con 0x7f3c8c158040 2024-03-22T12:56:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.367+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f3c8c156b60 con 0x7f3c8c158040 2024-03-22T12:56:28.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.747+0000 7f3c82a00640 1 -- 172.21.15.6:0/2514708032 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v21) v1 ==== 171+0+0 (secure 0 0 0) 0x7f3c8405c720 con 0x7f3c8c158040 2024-03-22T12:56:28.747 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-22T12:56:28.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.753+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3c6c06bc20 msgr2=0x7f3c6c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:28.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.754+0000 7f3c91400640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c001f10 tx=0x7f3c7c009040 comp rx=0 tx=0).stop 2024-03-22T12:56:28.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.754+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c8c158040 msgr2=0x7f3c8c1ed800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:28.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.754+0000 7f3c91400640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c8c158040 0x7f3c8c1ed800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400c3c0 tx=0x7f3c8400c880 comp rx=0 tx=0).stop 2024-03-22T12:56:28.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.756+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 shutdown_connections 2024-03-22T12:56:28.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.756+0000 7f3c91400640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:28.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.756+0000 7f3c91400640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c8c158040 0x7f3c8c1ed800 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:28.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.756+0000 7f3c91400640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3c8c152410 0x7f3c8c1ed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:28.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.756+0000 7f3c91400640 1 --2- 172.21.15.6:0/2514708032 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c8c0cab10 0x7f3c8c0b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:28.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.756+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 >> 172.21.15.6:0/2514708032 conn(0x7f3c8c071410 msgr2=0x7f3c8c157360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:28.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.759+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 shutdown_connections 2024-03-22T12:56:28.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.760+0000 7f3c91400640 1 -- 172.21.15.6:0/2514708032 wait complete. 2024-03-22T12:56:28.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-22T12:56:28.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.981+0000 7fac5e000640 1 Processor -- start 2024-03-22T12:56:28.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.981+0000 7fac5e000640 1 -- start start 2024-03-22T12:56:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.983+0000 7fac5e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580ceae0 0x7fac580ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.983+0000 7fac5e000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac580ccf20 con 0x7fac580cf660 2024-03-22T12:56:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.984+0000 7fac5e000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5815a6b0 con 0x7fac580caa80 2024-03-22T12:56:28.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.984+0000 7fac5e000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580cfa40 con 0x7fac580ceae0 2024-03-22T12:56:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.984+0000 7fac4ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580ceae0 0x7fac580ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.984+0000 7fac57e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fac580caa80 0x7fac580cc3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:46542/0 (socket says 172.21.15.6:46542) 2024-03-22T12:56:28.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.984+0000 7fac57e00640 1 -- 172.21.15.6:0/3884010554 learned_addr learned my addr 172.21.15.6:0/3884010554 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:28.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.987+0000 7fac4ea00640 1 -- 172.21.15.6:0/3884010554 >> v1:172.21.15.92:6789/0 conn(0x7fac580caa80 legacy=0x7fac580cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:28.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.987+0000 7fac4ea00640 1 -- 172.21.15.6:0/3884010554 >> v1:172.21.15.6:6789/0 conn(0x7fac580cf660 legacy=0x7fac580cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:28.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.987+0000 7fac4ea00640 1 -- 172.21.15.6:0/3884010554 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5815a990 con 0x7fac580ceae0 2024-03-22T12:56:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.989+0000 7fac4ea00640 1 --2- 172.21.15.6:0/3884010554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580ceae0 0x7fac580ca540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fac40009dc0 tx=0x7fac4002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0d01af2d9f589ec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:28.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.993+0000 7fac56a00640 1 -- 172.21.15.6:0/3884010554 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40039070 con 0x7fac580ceae0 2024-03-22T12:56:28.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.993+0000 7fac56a00640 1 -- 172.21.15.6:0/3884010554 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4002ccd0 con 0x7fac580ceae0 2024-03-22T12:56:28.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.993+0000 7fac56a00640 1 -- 172.21.15.6:0/3884010554 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40034660 con 0x7fac580ceae0 2024-03-22T12:56:28.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.994+0000 7fac5e000640 1 -- 172.21.15.6:0/3884010554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580ceae0 msgr2=0x7fac580ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:28.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.994+0000 7fac5e000640 1 --2- 172.21.15.6:0/3884010554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580ceae0 0x7fac580ca540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fac40009dc0 tx=0x7fac4002bf20 comp rx=0 tx=0).stop 2024-03-22T12:56:28.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.994+0000 7fac5e000640 1 -- 172.21.15.6:0/3884010554 shutdown_connections 2024-03-22T12:56:28.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.994+0000 7fac5e000640 1 --2- 172.21.15.6:0/3884010554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580ceae0 0x7fac580ca540 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:28.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.994+0000 7fac5e000640 1 -- 172.21.15.6:0/3884010554 >> 172.21.15.6:0/3884010554 conn(0x7fac580b16d0 msgr2=0x7fac580b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:28.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.994+0000 7fac5e000640 1 -- 172.21.15.6:0/3884010554 shutdown_connections 2024-03-22T12:56:28.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.994+0000 7fac5e000640 1 -- 172.21.15.6:0/3884010554 wait complete. 2024-03-22T12:56:28.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.996+0000 7fac5e000640 1 Processor -- start 2024-03-22T12:56:28.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.996+0000 7fac5e000640 1 -- start start 2024-03-22T12:56:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.997+0000 7fac5e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580caa80 0x7fac5806c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.997+0000 7fac5e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac580ceae0 0x7fac580c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.997+0000 7fac5e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac580cf660 0x7fac58151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.997+0000 7fac5e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5815afe0 con 0x7fac580cf660 2024-03-22T12:56:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.997+0000 7fac5e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5815b470 con 0x7fac580ceae0 2024-03-22T12:56:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.997+0000 7fac5e000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580ce0d0 con 0x7fac580caa80 2024-03-22T12:56:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.998+0000 7fac57400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580caa80 0x7fac5806c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:28.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.998+0000 7fac4ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac580ceae0 0x7fac580c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:28.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.998+0000 7fac57400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580caa80 0x7fac5806c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50840/0 (socket says 172.21.15.6:50840) 2024-03-22T12:56:28.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:28.998+0000 7fac57400640 1 -- 172.21.15.6:0/2092495933 learned_addr learned my addr 172.21.15.6:0/2092495933 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.003+0000 7fac57400640 1 -- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac580ceae0 msgr2=0x7fac580c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.003+0000 7fac57400640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac580ceae0 0x7fac580c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.003+0000 7fac57400640 1 -- 172.21.15.6:0/2092495933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac580cf660 msgr2=0x7fac58151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.003+0000 7fac57400640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac580cf660 0x7fac58151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.003+0000 7fac57400640 1 -- 172.21.15.6:0/2092495933 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58154680 con 0x7fac580caa80 2024-03-22T12:56:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.006+0000 7fac57400640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580caa80 0x7fac5806c8c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac4800b3e0 tx=0x7fac4800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.009+0000 7fac4fe00640 1 -- 172.21.15.6:0/2092495933 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48015020 con 0x7fac580caa80 2024-03-22T12:56:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.009+0000 7fac4fe00640 1 -- 172.21.15.6:0/2092495933 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48011850 con 0x7fac580caa80 2024-03-22T12:56:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.009+0000 7fac4fe00640 1 -- 172.21.15.6:0/2092495933 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48011c40 con 0x7fac580caa80 2024-03-22T12:56:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.009+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac580c5810 con 0x7fac580caa80 2024-03-22T12:56:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.009+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac58157d60 con 0x7fac580caa80 2024-03-22T12:56:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.011+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac580757d0 con 0x7fac580caa80 2024-03-22T12:56:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.031+0000 7fac4fe00640 1 -- 172.21.15.6:0/2092495933 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac48018020 con 0x7fac580caa80 2024-03-22T12:56:29.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.032+0000 7fac4fe00640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac3406bc20 0x7fac3406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:29.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.032+0000 7fac4fe00640 1 -- 172.21.15.6:0/2092495933 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fac4808aef0 con 0x7fac580caa80 2024-03-22T12:56:29.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.033+0000 7fac4ea00640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac3406bc20 0x7fac3406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:29.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.033+0000 7fac4ea00640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac3406bc20 0x7fac3406e0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac40036000 tx=0x7fac40030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:29.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:56:29.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.301+0000 7fac4fe00640 1 -- 172.21.15.6:0/2092495933 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac4801f020 con 0x7fac580caa80 2024-03-22T12:56:29.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:56:29.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:29.571+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fac5815a340 con 0x7fac580caa80 2024-03-22T12:56:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.169+0000 7fac4fe00640 1 -- 172.21.15.6:0/2092495933 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fac48059a90 con 0x7fac580caa80 2024-03-22T12:56:31.170 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 2 and data pool 3 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.175+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac3406bc20 msgr2=0x7fac3406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.175+0000 7fac5e000640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac3406bc20 0x7fac3406e0e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac40036000 tx=0x7fac40030000 comp rx=0 tx=0).stop 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.175+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580caa80 msgr2=0x7fac5806c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.175+0000 7fac5e000640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580caa80 0x7fac5806c8c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac4800b3e0 tx=0x7fac4800b8a0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 shutdown_connections 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac580cf660 0x7fac58151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac3406bc20 0x7fac3406e0e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac580ceae0 0x7fac580c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 --2- 172.21.15.6:0/2092495933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac580caa80 0x7fac5806c8c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 >> 172.21.15.6:0/2092495933 conn(0x7fac580b16d0 msgr2=0x7fac58158e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 shutdown_connections 2024-03-22T12:56:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.176+0000 7fac5e000640 1 -- 172.21.15.6:0/2092495933 wait complete. 2024-03-22T12:56:31.199 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-22T12:56:31.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-22T12:56:31.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.399+0000 7fc217200640 1 Processor -- start 2024-03-22T12:56:31.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.399+0000 7fc217200640 1 -- start start 2024-03-22T12:56:31.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.401+0000 7fc217200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180ca350 0x7fc2180ca730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:31.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.401+0000 7fc217200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2180cd7d0 0x7fc2180cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:31.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.401+0000 7fc217200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21815a340 con 0x7fc21815a840 2024-03-22T12:56:31.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.401+0000 7fc217200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2180cc3d0 con 0x7fc2180cd7d0 2024-03-22T12:56:31.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.401+0000 7fc217200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180cc6b0 con 0x7fc2180ca350 2024-03-22T12:56:31.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.408+0000 7fc215e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180ca350 0x7fc2180ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:31.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.408+0000 7fc215e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180ca350 0x7fc2180ca730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50848/0 (socket says 172.21.15.6:50848) 2024-03-22T12:56:31.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.408+0000 7fc215e00640 1 -- 172.21.15.6:0/2471278708 learned_addr learned my addr 172.21.15.6:0/2471278708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:31.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.410+0000 7fc215e00640 1 -- 172.21.15.6:0/2471278708 >> v2:172.21.15.6:3300/0 conn(0x7fc2180cd7d0 msgr2=0x7fc2180cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:31.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.410+0000 7fc215e00640 1 --2- 172.21.15.6:0/2471278708 >> v2:172.21.15.6:3300/0 conn(0x7fc2180cd7d0 0x7fc2180cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.410+0000 7fc215e00640 1 -- 172.21.15.6:0/2471278708 >> v1:172.21.15.6:6789/0 conn(0x7fc21815a840 legacy=0x7fc21815acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:31.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.410+0000 7fc215e00640 1 -- 172.21.15.6:0/2471278708 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2180cc990 con 0x7fc2180ca350 2024-03-22T12:56:31.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.412+0000 7fc215e00640 1 --2- 172.21.15.6:0/2471278708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180ca350 0x7fc2180ca730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc208009870 tx=0x7fc20802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7857e8c3c5af17e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:31.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.416+0000 7fc215400640 1 -- 172.21.15.6:0/2471278708 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208034070 con 0x7fc2180ca350 2024-03-22T12:56:31.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.416+0000 7fc215400640 1 -- 172.21.15.6:0/2471278708 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc208030840 con 0x7fc2180ca350 2024-03-22T12:56:31.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.416+0000 7fc215400640 1 -- 172.21.15.6:0/2471278708 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208030c10 con 0x7fc2180ca350 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 -- 172.21.15.6:0/2471278708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180ca350 msgr2=0x7fc2180ca730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 --2- 172.21.15.6:0/2471278708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180ca350 0x7fc2180ca730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc208009870 tx=0x7fc20802b980 comp rx=0 tx=0).stop 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 -- 172.21.15.6:0/2471278708 shutdown_connections 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 --2- 172.21.15.6:0/2471278708 >> v2:172.21.15.6:3300/0 conn(0x7fc2180cd7d0 0x7fc2180cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 --2- 172.21.15.6:0/2471278708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180ca350 0x7fc2180ca730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 -- 172.21.15.6:0/2471278708 >> 172.21.15.6:0/2471278708 conn(0x7fc2180cf9d0 msgr2=0x7fc2180cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 -- 172.21.15.6:0/2471278708 shutdown_connections 2024-03-22T12:56:31.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.417+0000 7fc217200640 1 -- 172.21.15.6:0/2471278708 wait complete. 2024-03-22T12:56:31.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.419+0000 7fc217200640 1 Processor -- start 2024-03-22T12:56:31.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.419+0000 7fc217200640 1 -- start start 2024-03-22T12:56:31.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.420+0000 7fc217200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2180ca350 0x7fc218157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:31.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.420+0000 7fc217200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180cd7d0 0x7fc218155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:31.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.420+0000 7fc217200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc21815a840 0x7fc2181f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:31.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.420+0000 7fc217200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180cdc10 con 0x7fc2180ca350 2024-03-22T12:56:31.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.420+0000 7fc217200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180cf600 con 0x7fc21815a840 2024-03-22T12:56:31.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.420+0000 7fc217200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180cdd90 con 0x7fc2180cd7d0 2024-03-22T12:56:31.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.421+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180cd7d0 0x7fc218155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:31.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.421+0000 7fc216800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc21815a840 0x7fc2181f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:31.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.421+0000 7fc20fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180cd7d0 0x7fc218155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50862/0 (socket says 172.21.15.6:50862) 2024-03-22T12:56:31.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.421+0000 7fc20fe00640 1 -- 172.21.15.6:0/3689949216 learned_addr learned my addr 172.21.15.6:0/3689949216 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.424+0000 7fc20fe00640 1 -- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc21815a840 msgr2=0x7fc2181f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.424+0000 7fc20fe00640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc21815a840 0x7fc2181f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.424+0000 7fc20fe00640 1 -- 172.21.15.6:0/3689949216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2180ca350 msgr2=0x7fc218157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.424+0000 7fc20fe00640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2180ca350 0x7fc218157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.424+0000 7fc20fe00640 1 -- 172.21.15.6:0/3689949216 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2181f40f0 con 0x7fc2180cd7d0 2024-03-22T12:56:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.426+0000 7fc20fe00640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180cd7d0 0x7fc218155e70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc20000b4a0 tx=0x7fc20000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.429+0000 7fc20e000640 1 -- 172.21.15.6:0/3689949216 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc200014070 con 0x7fc2180cd7d0 2024-03-22T12:56:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.429+0000 7fc20e000640 1 -- 172.21.15.6:0/3689949216 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc200010610 con 0x7fc2180cd7d0 2024-03-22T12:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.429+0000 7fc20e000640 1 -- 172.21.15.6:0/3689949216 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc200010a00 con 0x7fc2180cd7d0 2024-03-22T12:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.430+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc218156540 con 0x7fc2180cd7d0 2024-03-22T12:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.430+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc218156b00 con 0x7fc2180cd7d0 2024-03-22T12:56:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.431+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc218075650 con 0x7fc2180cd7d0 2024-03-22T12:56:31.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.681+0000 7fc20e000640 1 -- 172.21.15.6:0/3689949216 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc200017070 con 0x7fc2180cd7d0 2024-03-22T12:56:31.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.683+0000 7fc20e000640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1f406bc20 0x7fc1f406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:31.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.684+0000 7fc20e000640 1 -- 172.21.15.6:0/3689949216 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc20008a770 con 0x7fc2180cd7d0 2024-03-22T12:56:31.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.684+0000 7fc215e00640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1f406bc20 0x7fc1f406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:31.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.685+0000 7fc215e00640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1f406bc20 0x7fc1f406e0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc208036000 tx=0x7fc208030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:31.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.697+0000 7fc20e000640 1 -- 172.21.15.6:0/3689949216 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc200058cf0 con 0x7fc2180cd7d0 2024-03-22T12:56:31.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:31.915+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fc2180c9de0 con 0x7fc2180cd7d0 2024-03-22T12:56:32.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.680+0000 7fc20e000640 1 -- 172.21.15.6:0/3689949216 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v24) v1 ==== 157+0+0 (secure 0 0 0) 0x7fc20005cd90 con 0x7fc2180cd7d0 2024-03-22T12:56:32.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.685+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1f406bc20 msgr2=0x7fc1f406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:32.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1f406bc20 0x7fc1f406e0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc208036000 tx=0x7fc208030000 comp rx=0 tx=0).stop 2024-03-22T12:56:32.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180cd7d0 msgr2=0x7fc218155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:32.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180cd7d0 0x7fc218155e70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc20000b4a0 tx=0x7fc20000b960 comp rx=0 tx=0).stop 2024-03-22T12:56:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 shutdown_connections 2024-03-22T12:56:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1f406bc20 0x7fc1f406e0e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc21815a840 0x7fc2181f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2180cd7d0 0x7fc218155e70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 --2- 172.21.15.6:0/3689949216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2180ca350 0x7fc218157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 >> 172.21.15.6:0/3689949216 conn(0x7fc2180cf9d0 msgr2=0x7fc2180cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.686+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 shutdown_connections 2024-03-22T12:56:32.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.687+0000 7fc217200640 1 -- 172.21.15.6:0/3689949216 wait complete. 2024-03-22T12:56:32.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-22T12:56:32.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.927+0000 7f1b08400640 1 Processor -- start 2024-03-22T12:56:32.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.928+0000 7f1b08400640 1 -- start start 2024-03-22T12:56:32.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.929+0000 7f1b08400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b00154350 0x7f1b0015a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:32.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.929+0000 7f1b08400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b000c7680 0x7f1b0006cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:32.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.929+0000 7f1b08400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b0015d9c0 0x7f1b0015de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:32.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.929+0000 7f1b08400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0006d2d0 con 0x7f1b000c7680 2024-03-22T12:56:32.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.929+0000 7f1b08400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b000c3ea0 con 0x7f1b0015d9c0 2024-03-22T12:56:32.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.929+0000 7f1b08400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0015e400 con 0x7f1b00154350 2024-03-22T12:56:32.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.930+0000 7f1b06400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b0015d9c0 0x7f1b0015de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:32.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.930+0000 7f1b05000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b000c7680 0x7f1b0006cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:32.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.931+0000 7f1b06400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b0015d9c0 0x7f1b0015de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:35382/0 (socket says 172.21.15.6:35382) 2024-03-22T12:56:32.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.931+0000 7f1b05000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b000c7680 0x7f1b0006cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44146/0 (socket says 172.21.15.6:44146) 2024-03-22T12:56:32.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.931+0000 7f1b06400640 1 -- 172.21.15.6:0/552514526 learned_addr learned my addr 172.21.15.6:0/552514526 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:32.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.931+0000 7f1b05a00640 1 --2- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b00154350 0x7f1b0015a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:32.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.933+0000 7f1b06400640 1 -- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b00154350 msgr2=0x7f1b0015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:32.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.933+0000 7f1b06400640 1 --2- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b00154350 0x7f1b0015a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.933+0000 7f1b06400640 1 -- 172.21.15.6:0/552514526 >> v2:172.21.15.6:3300/0 conn(0x7f1b000c7680 msgr2=0x7f1b0006cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:32.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.933+0000 7f1b06400640 1 --2- 172.21.15.6:0/552514526 >> v2:172.21.15.6:3300/0 conn(0x7f1b000c7680 0x7f1b0006cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.933+0000 7f1b06400640 1 -- 172.21.15.6:0/552514526 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0015e6e0 con 0x7f1b0015d9c0 2024-03-22T12:56:32.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.935+0000 7f1b06400640 1 --2- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b0015d9c0 0x7f1b0015de10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b00072730 tx=0x7f1af802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=784153bbb0bd2bf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:32.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.939+0000 7f1affe00640 1 -- 172.21.15.6:0/552514526 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8034070 con 0x7f1b0015d9c0 2024-03-22T12:56:32.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.939+0000 7f1affe00640 1 -- 172.21.15.6:0/552514526 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af8030860 con 0x7f1b0015d9c0 2024-03-22T12:56:32.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.939+0000 7f1affe00640 1 -- 172.21.15.6:0/552514526 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8030c30 con 0x7f1b0015d9c0 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 -- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b0015d9c0 msgr2=0x7f1b0015de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 --2- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b0015d9c0 0x7f1b0015de10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b00072730 tx=0x7f1af802b9a0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 -- 172.21.15.6:0/552514526 shutdown_connections 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 --2- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b0015d9c0 0x7f1b0015de10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 --2- 172.21.15.6:0/552514526 >> v2:172.21.15.6:3300/0 conn(0x7f1b000c7680 0x7f1b0006cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 --2- 172.21.15.6:0/552514526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b00154350 0x7f1b0015a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 -- 172.21.15.6:0/552514526 >> 172.21.15.6:0/552514526 conn(0x7f1b000ccc80 msgr2=0x7f1b000cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 -- 172.21.15.6:0/552514526 shutdown_connections 2024-03-22T12:56:32.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.940+0000 7f1b08400640 1 -- 172.21.15.6:0/552514526 wait complete. 2024-03-22T12:56:32.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.942+0000 7f1b08400640 1 Processor -- start 2024-03-22T12:56:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.942+0000 7f1b08400640 1 -- start start 2024-03-22T12:56:32.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.943+0000 7f1b08400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b000c7680 0x7f1b001f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.943+0000 7f1b08400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b00154350 0x7f1b001e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.943+0000 7f1b08400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b0015d9c0 0x7f1b0015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.943+0000 7f1b08400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b00154730 con 0x7f1b000c7680 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.943+0000 7f1b08400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0006c590 con 0x7f1b00154350 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.943+0000 7f1b08400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001548b0 con 0x7f1b0015d9c0 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.944+0000 7f1b06400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b0015d9c0 0x7f1b0015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.944+0000 7f1b05000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b00154350 0x7f1b001e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:32.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.944+0000 7f1b06400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b0015d9c0 0x7f1b0015bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57590/0 (socket says 172.21.15.6:57590) 2024-03-22T12:56:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.944+0000 7f1b06400640 1 -- 172.21.15.6:0/3114948052 learned_addr learned my addr 172.21.15.6:0/3114948052 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:32.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.944+0000 7f1b05000640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b00154350 0x7f1b001e6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:35392/0 (socket says 172.21.15.6:35392) 2024-03-22T12:56:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.947+0000 7f1b06400640 1 -- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b00154350 msgr2=0x7f1b001e6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.947+0000 7f1b06400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b00154350 0x7f1b001e6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.947+0000 7f1b06400640 1 -- 172.21.15.6:0/3114948052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b000c7680 msgr2=0x7f1b001f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.947+0000 7f1b06400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b000c7680 0x7f1b001f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.947+0000 7f1b06400640 1 -- 172.21.15.6:0/3114948052 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b001efe90 con 0x7f1b0015d9c0 2024-03-22T12:56:32.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.949+0000 7f1b06400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b0015d9c0 0x7f1b0015bbe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009960 tx=0x7f1af80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:32.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.953+0000 7f1afd600640 1 -- 172.21.15.6:0/3114948052 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8034040 con 0x7f1b0015d9c0 2024-03-22T12:56:32.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.953+0000 7f1afd600640 1 -- 172.21.15.6:0/3114948052 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af8039c90 con 0x7f1b0015d9c0 2024-03-22T12:56:32.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.953+0000 7f1afd600640 1 -- 172.21.15.6:0/3114948052 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8043610 con 0x7f1b0015d9c0 2024-03-22T12:56:32.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.953+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b001f0170 con 0x7f1b0015d9c0 2024-03-22T12:56:32.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.953+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b001f0730 con 0x7f1b0015d9c0 2024-03-22T12:56:32.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.954+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b00074d50 con 0x7f1b0015d9c0 2024-03-22T12:56:32.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.973+0000 7f1afd600640 1 -- 172.21.15.6:0/3114948052 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1af8037070 con 0x7f1b0015d9c0 2024-03-22T12:56:32.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.974+0000 7f1afd600640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1ae006bc20 0x7f1ae006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:32.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.975+0000 7f1afd600640 1 -- 172.21.15.6:0/3114948052 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f1af80ad260 con 0x7f1b0015d9c0 2024-03-22T12:56:32.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.975+0000 7f1b05a00640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1ae006bc20 0x7f1ae006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:32.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:32.976+0000 7f1b05a00640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1ae006bc20 0x7f1ae006e0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1af000e190 tx=0x7f1af000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:33.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:33.218+0000 7f1afd600640 1 -- 172.21.15.6:0/3114948052 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1af803e070 con 0x7f1b0015d9c0 2024-03-22T12:56:33.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:33.431+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1b000c5460 con 0x7f1b0015d9c0 2024-03-22T12:56:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.695+0000 7f1afd600640 1 -- 172.21.15.6:0/3114948052 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v26) v1 ==== 228+0+0 (secure 0 0 0) 0x7f1af807b6b0 con 0x7f1b0015d9c0 2024-03-22T12:56:35.696 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data_ec' created 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1ae006bc20 msgr2=0x7f1ae006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1ae006bc20 0x7f1ae006e0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1af000e190 tx=0x7f1af000b040 comp rx=0 tx=0).stop 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b0015d9c0 msgr2=0x7f1b0015bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b0015d9c0 0x7f1b0015bbe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009960 tx=0x7f1af80046c0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 shutdown_connections 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b0015d9c0 0x7f1b0015bbe0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1ae006bc20 0x7f1ae006e0e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b00154350 0x7f1b001e6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 --2- 172.21.15.6:0/3114948052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b000c7680 0x7f1b001f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 >> 172.21.15.6:0/3114948052 conn(0x7f1b000ccc80 msgr2=0x7f1b000c8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:35.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 shutdown_connections 2024-03-22T12:56:35.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.699+0000 7f1b08400640 1 -- 172.21.15.6:0/3114948052 wait complete. 2024-03-22T12:56:35.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-22T12:56:35.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.904+0000 7f2108000640 1 Processor -- start 2024-03-22T12:56:35.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.904+0000 7f2108000640 1 -- start start 2024-03-22T12:56:35.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.905+0000 7f2108000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f21000c6cc0 0x7f210013dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:35.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.905+0000 7f2108000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2100072750 0x7f21000c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:35.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.906+0000 7f2108000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2100072b50 0x7f21000c9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:35.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.906+0000 7f2108000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210013e110 con 0x7f21000c6cc0 2024-03-22T12:56:35.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.906+0000 7f2108000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2100158030 con 0x7f2100072b50 2024-03-22T12:56:35.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.906+0000 7f2108000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21001582f0 con 0x7f2100072750 2024-03-22T12:56:35.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.906+0000 7f2105600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f21000c6cc0 0x7f210013dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:35.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.906+0000 7f2106000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2100072b50 0x7f21000c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:35.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.906+0000 7f2104c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2100072750 0x7f21000c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:35.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.907+0000 7f2106000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2100072b50 0x7f21000c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:35396/0 (socket says 172.21.15.6:35396) 2024-03-22T12:56:35.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.907+0000 7f2106000640 1 -- 172.21.15.6:0/2441351978 learned_addr learned my addr 172.21.15.6:0/2441351978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:35.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.907+0000 7f2104c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2100072750 0x7f21000c93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57598/0 (socket says 172.21.15.6:57598) 2024-03-22T12:56:35.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.909+0000 7f2106000640 1 -- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2100072750 msgr2=0x7f21000c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:35.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.909+0000 7f2106000640 1 --2- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2100072750 0x7f21000c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.909+0000 7f2106000640 1 -- 172.21.15.6:0/2441351978 >> v2:172.21.15.6:3300/0 conn(0x7f21000c6cc0 msgr2=0x7f210013dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:35.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.909+0000 7f2106000640 1 --2- 172.21.15.6:0/2441351978 >> v2:172.21.15.6:3300/0 conn(0x7f21000c6cc0 0x7f210013dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.909+0000 7f2106000640 1 -- 172.21.15.6:0/2441351978 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21001585d0 con 0x7f2100072b50 2024-03-22T12:56:35.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.909+0000 7f2104c00640 1 --2- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2100072750 0x7f21000c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:35.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.911+0000 7f2106000640 1 --2- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2100072b50 0x7f21000c9930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21000710e0 tx=0x7f20f802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ec8cc28c68807a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:35.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.917+0000 7f20ffe00640 1 -- 172.21.15.6:0/2441351978 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f802c9e0 con 0x7f2100072b50 2024-03-22T12:56:35.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.917+0000 7f20ffe00640 1 -- 172.21.15.6:0/2441351978 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f802cb80 con 0x7f2100072b50 2024-03-22T12:56:35.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.917+0000 7f20ffe00640 1 -- 172.21.15.6:0/2441351978 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f80343f0 con 0x7f2100072b50 2024-03-22T12:56:35.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.917+0000 7f2108000640 1 -- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2100072b50 msgr2=0x7f21000c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:35.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.917+0000 7f2108000640 1 --2- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2100072b50 0x7f21000c9930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21000710e0 tx=0x7f20f802be40 comp rx=0 tx=0).stop 2024-03-22T12:56:35.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.918+0000 7f2108000640 1 -- 172.21.15.6:0/2441351978 shutdown_connections 2024-03-22T12:56:35.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.918+0000 7f2108000640 1 --2- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2100072b50 0x7f21000c9930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.918+0000 7f2108000640 1 --2- 172.21.15.6:0/2441351978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2100072750 0x7f21000c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.918+0000 7f2108000640 1 --2- 172.21.15.6:0/2441351978 >> v2:172.21.15.6:3300/0 conn(0x7f21000c6cc0 0x7f210013dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.918+0000 7f2108000640 1 -- 172.21.15.6:0/2441351978 >> 172.21.15.6:0/2441351978 conn(0x7f210006c7c0 msgr2=0x7f21000702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:35.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.918+0000 7f2108000640 1 -- 172.21.15.6:0/2441351978 shutdown_connections 2024-03-22T12:56:35.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.918+0000 7f2108000640 1 -- 172.21.15.6:0/2441351978 wait complete. 2024-03-22T12:56:35.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.920+0000 7f2108000640 1 Processor -- start 2024-03-22T12:56:35.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.920+0000 7f2108000640 1 -- start start 2024-03-22T12:56:35.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.921+0000 7f2108000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2100072750 0x7f210013b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:35.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.921+0000 7f2108000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21000c6cc0 0x7f21000cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:35.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.921+0000 7f2108000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21000cd900 0x7f21000cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:35.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.921+0000 7f2108000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000c70a0 con 0x7f2100072750 2024-03-22T12:56:35.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.921+0000 7f2108000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2100158b90 con 0x7f21000c6cc0 2024-03-22T12:56:35.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.921+0000 7f2108000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210015b7d0 con 0x7f21000cd900 2024-03-22T12:56:35.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.922+0000 7f2105600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2100072750 0x7f210013b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:35.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.922+0000 7f2104c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21000c6cc0 0x7f21000cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:35.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.922+0000 7f2105600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2100072750 0x7f210013b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44166/0 (socket says 172.21.15.6:44166) 2024-03-22T12:56:35.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.922+0000 7f2105600640 1 -- 172.21.15.6:0/20090403 learned_addr learned my addr 172.21.15.6:0/20090403 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:35.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.922+0000 7f2106000640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21000cd900 0x7f21000cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:35.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.926+0000 7f2104c00640 1 -- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21000cd900 msgr2=0x7f21000cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:35.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.926+0000 7f2104c00640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21000cd900 0x7f21000cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.926+0000 7f2104c00640 1 -- 172.21.15.6:0/20090403 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2100072750 msgr2=0x7f210013b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:35.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.926+0000 7f2104c00640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2100072750 0x7f210013b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:35.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.926+0000 7f2104c00640 1 -- 172.21.15.6:0/20090403 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21001f3f20 con 0x7f21000c6cc0 2024-03-22T12:56:35.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.928+0000 7f2104c00640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21000c6cc0 0x7f21000cc370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e900 tx=0x7f20ec00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:35.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.931+0000 7f20fd600640 1 -- 172.21.15.6:0/20090403 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec017070 con 0x7f21000c6cc0 2024-03-22T12:56:35.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.931+0000 7f20fd600640 1 -- 172.21.15.6:0/20090403 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ec00ca00 con 0x7f21000c6cc0 2024-03-22T12:56:35.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.931+0000 7f20fd600640 1 -- 172.21.15.6:0/20090403 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec00cdf0 con 0x7f21000c6cc0 2024-03-22T12:56:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.932+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210013c7f0 con 0x7f21000c6cc0 2024-03-22T12:56:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.932+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21000cbb70 con 0x7f21000c6cc0 2024-03-22T12:56:35.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.933+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21000c4e70 con 0x7f21000c6cc0 2024-03-22T12:56:35.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.957+0000 7f20fd600640 1 -- 172.21.15.6:0/20090403 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f20ec01a070 con 0x7f21000c6cc0 2024-03-22T12:56:35.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.958+0000 7f20fd600640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20e006bc20 0x7f20e006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:35.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.958+0000 7f2105600640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20e006bc20 0x7f20e006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:35.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.959+0000 7f2105600640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20e006bc20 0x7f20e006e0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009ae0 tx=0x7f20f4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:35.960+0000 7f20fd600640 1 -- 172.21.15.6:0/20090403 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f20ec057540 con 0x7f21000c6cc0 2024-03-22T12:56:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.201+0000 7f20fd600640 1 -- 172.21.15.6:0/20090403 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f21000c4e70 con 0x7f21000c6cc0 2024-03-22T12:56:36.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.415+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f21000757d0 con 0x7f21000c6cc0 2024-03-22T12:56:36.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.995+0000 7f20fd600640 1 -- 172.21.15.6:0/20090403 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f20ec0594a0 con 0x7f21000c6cc0 2024-03-22T12:56:36.995 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 4 allow_ec_overwrites to true 2024-03-22T12:56:36.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20e006bc20 msgr2=0x7f20e006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:36.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20e006bc20 0x7f20e006e0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009ae0 tx=0x7f20f4009190 comp rx=0 tx=0).stop 2024-03-22T12:56:36.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21000c6cc0 msgr2=0x7f21000cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21000c6cc0 0x7f21000cc370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e900 tx=0x7f20ec00edc0 comp rx=0 tx=0).stop 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 shutdown_connections 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21000cd900 0x7f21000cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20e006bc20 0x7f20e006e0e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21000c6cc0 0x7f21000cc370 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 --2- 172.21.15.6:0/20090403 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2100072750 0x7f210013b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.998+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 >> 172.21.15.6:0/20090403 conn(0x7f210006c7c0 msgr2=0x7f210006ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.999+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 shutdown_connections 2024-03-22T12:56:36.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:36.999+0000 7f2108000640 1 -- 172.21.15.6:0/20090403 wait complete. 2024-03-22T12:56:37.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-22T12:56:37.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.227+0000 7f29eee00640 1 Processor -- start 2024-03-22T12:56:37.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.227+0000 7f29eee00640 1 -- start start 2024-03-22T12:56:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.229+0000 7f29eee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c8bf0 0x7f29e80732c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:37.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.229+0000 7f29eee00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f29e80c6c20 0x7f29e815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:37.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.229+0000 7f29eee00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e80c7180 con 0x7f29e8158c20 2024-03-22T12:56:37.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.229+0000 7f29eee00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e8161d40 con 0x7f29e80c6c20 2024-03-22T12:56:37.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.229+0000 7f29eee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e81628c0 con 0x7f29e80c8bf0 2024-03-22T12:56:37.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.230+0000 7f29e7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c8bf0 0x7f29e80732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.230+0000 7f29e7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c8bf0 0x7f29e80732c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57616/0 (socket says 172.21.15.6:57616) 2024-03-22T12:56:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.230+0000 7f29e7400640 1 -- 172.21.15.6:0/1841289663 learned_addr learned my addr 172.21.15.6:0/1841289663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.230+0000 7f29ece00640 1 --2- 172.21.15.6:0/1841289663 >> v2:172.21.15.92:3300/0 conn(0x7f29e80c6c20 0x7f29e815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:37.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.234+0000 7f29e7400640 1 -- 172.21.15.6:0/1841289663 >> v2:172.21.15.92:3300/0 conn(0x7f29e80c6c20 msgr2=0x7f29e815e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:37.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.234+0000 7f29e7400640 1 --2- 172.21.15.6:0/1841289663 >> v2:172.21.15.92:3300/0 conn(0x7f29e80c6c20 0x7f29e815e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:37.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.234+0000 7f29e7400640 1 -- 172.21.15.6:0/1841289663 >> v1:172.21.15.92:6789/0 conn(0x7f29e8158c20 legacy=0x7f29e80c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:37.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.234+0000 7f29e7400640 1 -- 172.21.15.6:0/1841289663 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e8161f90 con 0x7f29e80c8bf0 2024-03-22T12:56:37.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.264+0000 7f29e7400640 1 --2- 172.21.15.6:0/1841289663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c8bf0 0x7f29e80732c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009870 tx=0x7f29d402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5df0ee45f1941931 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:37.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.272+0000 7f29e6a00640 1 -- 172.21.15.6:0/1841289663 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4035020 con 0x7f29e80c8bf0 2024-03-22T12:56:37.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.272+0000 7f29e6a00640 1 -- 172.21.15.6:0/1841289663 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4031640 con 0x7f29e80c8bf0 2024-03-22T12:56:37.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.272+0000 7f29e6a00640 1 -- 172.21.15.6:0/1841289663 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d402d040 con 0x7f29e80c8bf0 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 -- 172.21.15.6:0/1841289663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c8bf0 msgr2=0x7f29e80732c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 --2- 172.21.15.6:0/1841289663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c8bf0 0x7f29e80732c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009870 tx=0x7f29d402b8f0 comp rx=0 tx=0).stop 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 -- 172.21.15.6:0/1841289663 shutdown_connections 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 --2- 172.21.15.6:0/1841289663 >> v2:172.21.15.92:3300/0 conn(0x7f29e80c6c20 0x7f29e815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 --2- 172.21.15.6:0/1841289663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c8bf0 0x7f29e80732c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 -- 172.21.15.6:0/1841289663 >> 172.21.15.6:0/1841289663 conn(0x7f29e806e310 msgr2=0x7f29e806f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 -- 172.21.15.6:0/1841289663 shutdown_connections 2024-03-22T12:56:37.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.274+0000 7f29eee00640 1 -- 172.21.15.6:0/1841289663 wait complete. 2024-03-22T12:56:37.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.277+0000 7f29eee00640 1 Processor -- start 2024-03-22T12:56:37.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.277+0000 7f29eee00640 1 -- start start 2024-03-22T12:56:37.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.278+0000 7f29eee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c6c20 0x7f29e814fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:37.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.278+0000 7f29eee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e80c8bf0 0x7f29e815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:37.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.278+0000 7f29eee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e8158c20 0x7f29e815b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:37.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.278+0000 7f29eee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8162550 con 0x7f29e80c8bf0 2024-03-22T12:56:37.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.278+0000 7f29eee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80b2040 con 0x7f29e8158c20 2024-03-22T12:56:37.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.278+0000 7f29eee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80c7710 con 0x7f29e80c6c20 2024-03-22T12:56:37.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.279+0000 7f29ece00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e8158c20 0x7f29e815b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:37.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.279+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c6c20 0x7f29e814fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:37.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.279+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c6c20 0x7f29e814fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57628/0 (socket says 172.21.15.6:57628) 2024-03-22T12:56:37.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.279+0000 7f29e7e00640 1 -- 172.21.15.6:0/521043887 learned_addr learned my addr 172.21.15.6:0/521043887 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:37.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.283+0000 7f29e7e00640 1 -- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e8158c20 msgr2=0x7f29e815b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:37.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.283+0000 7f29e7e00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e8158c20 0x7f29e815b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:37.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.283+0000 7f29e7e00640 1 -- 172.21.15.6:0/521043887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e80c8bf0 msgr2=0x7f29e815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:37.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.283+0000 7f29e7e00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e80c8bf0 0x7f29e815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:37.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.283+0000 7f29e7e00640 1 -- 172.21.15.6:0/521043887 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e81f5170 con 0x7f29e80c6c20 2024-03-22T12:56:37.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.286+0000 7f29e7e00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c6c20 0x7f29e814fe20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f29d800b3f0 tx=0x7f29d800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:37.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.289+0000 7f29dfe00640 1 -- 172.21.15.6:0/521043887 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8015020 con 0x7f29e80c6c20 2024-03-22T12:56:37.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.290+0000 7f29dfe00640 1 -- 172.21.15.6:0/521043887 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d8011830 con 0x7f29e80c6c20 2024-03-22T12:56:37.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.290+0000 7f29dfe00640 1 -- 172.21.15.6:0/521043887 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8011c20 con 0x7f29e80c6c20 2024-03-22T12:56:37.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.290+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e81f5450 con 0x7f29e80c6c20 2024-03-22T12:56:37.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.290+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e81f5a10 con 0x7f29e80c6c20 2024-03-22T12:56:37.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.291+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e80757d0 con 0x7f29e80c6c20 2024-03-22T12:56:37.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.310+0000 7f29dfe00640 1 -- 172.21.15.6:0/521043887 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f29d8018020 con 0x7f29e80c6c20 2024-03-22T12:56:37.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.312+0000 7f29dfe00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f29c806bc20 0x7f29c806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:37.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.312+0000 7f29e7400640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f29c806bc20 0x7f29c806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:37.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.313+0000 7f29dfe00640 1 -- 172.21.15.6:0/521043887 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f29d8056fb0 con 0x7f29e80c6c20 2024-03-22T12:56:37.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.313+0000 7f29e7400640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f29c806bc20 0x7f29c806e0e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036000 tx=0x7f29d4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.555+0000 7f29dfe00640 1 -- 172.21.15.6:0/521043887 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f29d801f030 con 0x7f29e80c6c20 2024-03-22T12:56:37.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:37.811+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f29e8154840 con 0x7f29e80c6c20 2024-03-22T12:56:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.444+0000 7f29dfe00640 1 -- 172.21.15.6:0/521043887 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f29d8059610 con 0x7f29e80c6c20 2024-03-22T12:56:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:added data pool 4 to fsmap 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f29c806bc20 msgr2=0x7f29c806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f29c806bc20 0x7f29c806e0e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036000 tx=0x7f29d4030000 comp rx=0 tx=0).stop 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c6c20 msgr2=0x7f29e814fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c6c20 0x7f29e814fe20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f29d800b3f0 tx=0x7f29d800b8b0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 shutdown_connections 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f29c806bc20 0x7f29c806e0e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e8158c20 0x7f29e815b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e80c8bf0 0x7f29e815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 --2- 172.21.15.6:0/521043887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e80c6c20 0x7f29e814fe20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 >> 172.21.15.6:0/521043887 conn(0x7f29e806e310 msgr2=0x7f29e815fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 shutdown_connections 2024-03-22T12:56:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.448+0000 7f29eee00640 1 -- 172.21.15.6:0/521043887 wait complete. 2024-03-22T12:56:38.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:56:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.685+0000 7f11a7200640 1 Processor -- start 2024-03-22T12:56:38.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.685+0000 7f11a7200640 1 -- start start 2024-03-22T12:56:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.686+0000 7f11a7200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.686+0000 7f11a7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11a00cd300 0x7f11a00ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.686+0000 7f11a7200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a00cc380 con 0x7f11a015a9b0 2024-03-22T12:56:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.686+0000 7f11a7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a015a520 con 0x7f11a00cd300 2024-03-22T12:56:38.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.686+0000 7f11a7200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00cc660 con 0x7f11a0154980 2024-03-22T12:56:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.687+0000 7f119f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11a00cd300 0x7f11a00ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.687+0000 7f119fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.687+0000 7f11a5200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f11a015a9b0 0x7f11a00ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:44544/0 (socket says 172.21.15.6:44544) 2024-03-22T12:56:38.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.687+0000 7f11a5200640 1 -- 172.21.15.6:0/3951617559 learned_addr learned my addr 172.21.15.6:0/3951617559 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.690+0000 7f119ea00640 1 -- 172.21.15.6:0/3951617559 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259332132 0 0) 0x7f11a00cc380 con 0x7f11a015a9b0 2024-03-22T12:56:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.690+0000 7f119ea00640 1 -- 172.21.15.6:0/3951617559 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1184003520 con 0x7f11a015a9b0 2024-03-22T12:56:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.690+0000 7f119f400640 1 -- 172.21.15.6:0/3951617559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 msgr2=0x7f11a00ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.690+0000 7f119f400640 1 --2- 172.21.15.6:0/3951617559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.690+0000 7f119f400640 1 -- 172.21.15.6:0/3951617559 >> v1:172.21.15.92:6789/0 conn(0x7f11a015a9b0 legacy=0x7f11a00ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.690+0000 7f119f400640 1 -- 172.21.15.6:0/3951617559 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a00cc940 con 0x7f11a00cd300 2024-03-22T12:56:38.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.693+0000 7f119f400640 1 --2- 172.21.15.6:0/3951617559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a00cd300 0x7f11a00ce840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4abb24075dc44d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.697+0000 7f119ea00640 1 -- 172.21.15.6:0/3951617559 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c035020 con 0x7f11a00cd300 2024-03-22T12:56:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.697+0000 7f119ea00640 1 -- 172.21.15.6:0/3951617559 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c031830 con 0x7f11a00cd300 2024-03-22T12:56:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.697+0000 7f119ea00640 1 -- 172.21.15.6:0/3951617559 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c031c00 con 0x7f11a00cd300 2024-03-22T12:56:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 -- 172.21.15.6:0/3951617559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a00cd300 msgr2=0x7f11a00ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 --2- 172.21.15.6:0/3951617559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a00cd300 0x7f11a00ce840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c02b8d0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 -- 172.21.15.6:0/3951617559 shutdown_connections 2024-03-22T12:56:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 --2- 172.21.15.6:0/3951617559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a00cd300 0x7f11a00ce840 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 --2- 172.21.15.6:0/3951617559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 -- 172.21.15.6:0/3951617559 >> 172.21.15.6:0/3951617559 conn(0x7f11a00b16c0 msgr2=0x7f11a00b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 -- 172.21.15.6:0/3951617559 shutdown_connections 2024-03-22T12:56:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.698+0000 7f11a7200640 1 -- 172.21.15.6:0/3951617559 wait complete. 2024-03-22T12:56:38.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.700+0000 7f11a7200640 1 Processor -- start 2024-03-22T12:56:38.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.700+0000 7f11a7200640 1 -- start start 2024-03-22T12:56:38.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.701+0000 7f11a7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11a00cd300 0x7f11a00b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:38.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.701+0000 7f11a7200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:38.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.701+0000 7f11a7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a015a9b0 0x7f11a00b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:38.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.701+0000 7f11a7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00cf5b0 con 0x7f11a015a9b0 2024-03-22T12:56:38.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.701+0000 7f11a7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00ccf00 con 0x7f11a00cd300 2024-03-22T12:56:38.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.701+0000 7f11a7200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00cf730 con 0x7f11a0154980 2024-03-22T12:56:38.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.702+0000 7f119fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11a00cd300 0x7f11a00b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:38.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.702+0000 7f11a5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a015a9b0 0x7f11a00b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:38.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.702+0000 7f11a5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a015a9b0 0x7f11a00b30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44190/0 (socket says 172.21.15.6:44190) 2024-03-22T12:56:38.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.702+0000 7f11a5200640 1 -- 172.21.15.6:0/3801176132 learned_addr learned my addr 172.21.15.6:0/3801176132 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:38.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.703+0000 7f119f400640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:38.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.706+0000 7f11a5200640 1 -- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 msgr2=0x7f11a00c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:38.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.706+0000 7f11a5200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.706+0000 7f11a5200640 1 -- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11a00cd300 msgr2=0x7f11a00b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:38.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.706+0000 7f11a5200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11a00cd300 0x7f11a00b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:38.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.706+0000 7f11a5200640 1 -- 172.21.15.6:0/3801176132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a01f1660 con 0x7f11a015a9b0 2024-03-22T12:56:38.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.706+0000 7f119fe00640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11a00cd300 0x7f11a00b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:38.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.709+0000 7f11a5200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a015a9b0 0x7f11a00b30b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f119800caa0 tx=0x7f119800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:38.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.713+0000 7f1197e00640 1 -- 172.21.15.6:0/3801176132 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198017070 con 0x7f11a015a9b0 2024-03-22T12:56:38.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.713+0000 7f1197e00640 1 -- 172.21.15.6:0/3801176132 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119800ac00 con 0x7f11a015a9b0 2024-03-22T12:56:38.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.713+0000 7f1197e00640 1 -- 172.21.15.6:0/3801176132 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198012400 con 0x7f11a015a9b0 2024-03-22T12:56:38.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.713+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a00b4640 con 0x7f11a015a9b0 2024-03-22T12:56:38.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.713+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a01f1bb0 con 0x7f11a015a9b0 2024-03-22T12:56:38.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:38.715+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a015ae60 con 0x7f11a015a9b0 2024-03-22T12:56:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.012+0000 7f1197e00640 1 -- 172.21.15.6:0/3801176132 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f119801a070 con 0x7f11a015a9b0 2024-03-22T12:56:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.013+0000 7f1197e00640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f118006bc20 0x7f118006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.013+0000 7f119fe00640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f118006bc20 0x7f118006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.014+0000 7f119fe00640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f118006bc20 0x7f118006e0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1190009f30 tx=0x7f1190009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:39.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.040+0000 7f1197e00640 1 -- 172.21.15.6:0/3801176132 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1198057100 con 0x7f11a015a9b0 2024-03-22T12:56:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.042+0000 7f1197e00640 1 -- 172.21.15.6:0/3801176132 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11a015ae60 con 0x7f11a015a9b0 2024-03-22T12:56:39.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.297+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11a00756c0 con 0x7f11a015a9b0 2024-03-22T12:56:39.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.376+0000 7f1197e00640 1 -- 172.21.15.6:0/3801176132 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6541 (secure 0 0 0) 0x7f1198021020 con 0x7f11a015a9b0 2024-03-22T12:56:39.377 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:39.377 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2825293724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2825293724},{"type":"v1","addr":"172.21.15.92:6837","nonce":2825293724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:56:37.865363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4268,"qdb_cluster":[4268]},"id":1}]} 2024-03-22T12:56:39.377 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f118006bc20 msgr2=0x7f118006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f118006bc20 0x7f118006e0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1190009f30 tx=0x7f1190009190 comp rx=0 tx=0).stop 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a015a9b0 msgr2=0x7f11a00b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a015a9b0 0x7f11a00b30b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f119800caa0 tx=0x7f119800cf60 comp rx=0 tx=0).stop 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 shutdown_connections 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11a015a9b0 0x7f11a00b30b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11a0154980 0x7f11a00c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f118006bc20 0x7f118006e0e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 --2- 172.21.15.6:0/3801176132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11a00cd300 0x7f11a00b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 >> 172.21.15.6:0/3801176132 conn(0x7f11a00b16c0 msgr2=0x7f11a00cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:39.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 shutdown_connections 2024-03-22T12:56:39.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.380+0000 7f11a7200640 1 -- 172.21.15.6:0/3801176132 wait complete. 2024-03-22T12:56:39.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:39.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.600+0000 7fea22c00640 1 Processor -- start 2024-03-22T12:56:39.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.601+0000 7fea22c00640 1 -- start start 2024-03-22T12:56:39.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.602+0000 7fea22c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fea1c15b0c0 0x7fea1c0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.602+0000 7fea22c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c0cc380 con 0x7fea1c154980 2024-03-22T12:56:39.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.602+0000 7fea22c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c0cc660 con 0x7fea1c0cd300 2024-03-22T12:56:39.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.602+0000 7fea22c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1c15a520 con 0x7fea1c15b0c0 2024-03-22T12:56:39.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.603+0000 7fea1be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea1c154980 0x7fea1c0ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49322/0 (socket says 172.21.15.6:49322) 2024-03-22T12:56:39.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.603+0000 7fea1be00640 1 -- 172.21.15.6:0/2172518416 learned_addr learned my addr 172.21.15.6:0/2172518416 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:39.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.605+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640660501 0 0) 0x7fea1c0cc660 con 0x7fea1c0cd300 2024-03-22T12:56:39.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.605+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea00003520 con 0x7fea1c0cd300 2024-03-22T12:56:39.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.606+0000 7fea20c00640 1 --2- 172.21.15.6:0/2172518416 >> v2:172.21.15.6:3300/0 conn(0x7fea1c15b0c0 0x7fea1c0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.606+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657253750 0 0) 0x7fea1c0cc380 con 0x7fea1c154980 2024-03-22T12:56:39.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.606+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c0cc660 con 0x7fea1c154980 2024-03-22T12:56:39.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.610+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2079298824 0 0) 0x7fea00003520 con 0x7fea1c0cd300 2024-03-22T12:56:39.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.610+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 >> v2:172.21.15.6:3300/0 conn(0x7fea1c15b0c0 msgr2=0x7fea1c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:39.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.610+0000 7fea1aa00640 1 --2- 172.21.15.6:0/2172518416 >> v2:172.21.15.6:3300/0 conn(0x7fea1c15b0c0 0x7fea1c0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.610+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 >> v1:172.21.15.6:6789/0 conn(0x7fea1c154980 legacy=0x7fea1c0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:39.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c0cc940 con 0x7fea1c0cd300 2024-03-22T12:56:39.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7fea040015a0 con 0x7fea1c0cd300 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 >> v1:172.21.15.92:6789/0 conn(0x7fea1c0cd300 legacy=0x7fea1c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea00003b90 0x7fea00003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea00004800 0x7fea00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea000051c0 0x7fea0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15a520 con 0x7fea00003b90 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0000c090 con 0x7fea00004800 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.611+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0000c290 con 0x7fea000051c0 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.612+0000 7fea20c00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea000051c0 0x7fea0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.612+0000 7fea1b400640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea00004800 0x7fea00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.614+0000 7fea20c00640 1 -- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea00004800 msgr2=0x7fea00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:39.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.614+0000 7fea20c00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea00004800 0x7fea00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.614+0000 7fea20c00640 1 -- 172.21.15.6:0/2172518416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea00003b90 msgr2=0x7fea00003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:39.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.614+0000 7fea20c00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea00003b90 0x7fea00003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.614+0000 7fea20c00640 1 -- 172.21.15.6:0/2172518416 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c0cc940 con 0x7fea000051c0 2024-03-22T12:56:39.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.615+0000 7fea20c00640 1 -- 172.21.15.6:0/2172518416 reap_dead start 2024-03-22T12:56:39.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.617+0000 7fea20c00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea000051c0 0x7fea0000b9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea14009ac0 tx=0x7fea00005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20f9b53e4fedc30b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.620+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14009690 con 0x7fea000051c0 2024-03-22T12:56:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.620+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea14004020 con 0x7fea000051c0 2024-03-22T12:56:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.620+0000 7fea1aa00640 1 -- 172.21.15.6:0/2172518416 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea140043d0 con 0x7fea000051c0 2024-03-22T12:56:39.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.621+0000 7fea22c00640 1 -- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea000051c0 msgr2=0x7fea0000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:39.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.621+0000 7fea22c00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea000051c0 0x7fea0000b9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea14009ac0 tx=0x7fea00005830 comp rx=0 tx=0).stop 2024-03-22T12:56:39.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.621+0000 7fea22c00640 1 -- 172.21.15.6:0/2172518416 shutdown_connections 2024-03-22T12:56:39.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.621+0000 7fea22c00640 1 --2- 172.21.15.6:0/2172518416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea000051c0 0x7fea0000b9c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.621+0000 7fea22c00640 1 -- 172.21.15.6:0/2172518416 >> 172.21.15.6:0/2172518416 conn(0x7fea1c0b16c0 msgr2=0x7fea1c0b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:39.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.621+0000 7fea22c00640 1 -- 172.21.15.6:0/2172518416 shutdown_connections 2024-03-22T12:56:39.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.621+0000 7fea22c00640 1 -- 172.21.15.6:0/2172518416 wait complete. 2024-03-22T12:56:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.623+0000 7fea22c00640 1 Processor -- start 2024-03-22T12:56:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.623+0000 7fea22c00640 1 -- start start 2024-03-22T12:56:39.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.624+0000 7fea22c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea000051c0 0x7fea1c1efb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.624+0000 7fea22c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea1c0c9d20 0x7fea1c158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.624+0000 7fea22c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea1c1f0040 0x7fea1c159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.624+0000 7fea22c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15a9b0 con 0x7fea000051c0 2024-03-22T12:56:39.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.624+0000 7fea22c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c0ccf00 con 0x7fea1c0c9d20 2024-03-22T12:56:39.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.624+0000 7fea22c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15ab30 con 0x7fea1c1f0040 2024-03-22T12:56:39.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.625+0000 7fea20c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea1c1f0040 0x7fea1c159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.625+0000 7fea1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea1c0c9d20 0x7fea1c158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.625+0000 7fea1be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea000051c0 0x7fea1c1efb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.625+0000 7fea1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea1c0c9d20 0x7fea1c158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:35442/0 (socket says 172.21.15.6:35442) 2024-03-22T12:56:39.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.625+0000 7fea1b400640 1 -- 172.21.15.6:0/2973544783 learned_addr learned my addr 172.21.15.6:0/2973544783 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:39.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.629+0000 7fea1b400640 1 -- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea1c1f0040 msgr2=0x7fea1c159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:39.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.629+0000 7fea1b400640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea1c1f0040 0x7fea1c159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.629+0000 7fea1b400640 1 -- 172.21.15.6:0/2973544783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea000051c0 msgr2=0x7fea1c1efb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:39.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.629+0000 7fea1b400640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea000051c0 0x7fea1c1efb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:39.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.629+0000 7fea1b400640 1 -- 172.21.15.6:0/2973544783 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c157ac0 con 0x7fea1c0c9d20 2024-03-22T12:56:39.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.632+0000 7fea1b400640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea1c0c9d20 0x7fea1c158fe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea0400f990 tx=0x7fea0400fe50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:39.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.635+0000 7fea13e00640 1 -- 172.21.15.6:0/2973544783 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea040053a0 con 0x7fea1c0c9d20 2024-03-22T12:56:39.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.635+0000 7fea13e00640 1 -- 172.21.15.6:0/2973544783 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea04018070 con 0x7fea1c0c9d20 2024-03-22T12:56:39.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.636+0000 7fea13e00640 1 -- 172.21.15.6:0/2973544783 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0400d990 con 0x7fea1c0c9d20 2024-03-22T12:56:39.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.636+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c1f3110 con 0x7fea1c0c9d20 2024-03-22T12:56:39.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.636+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c1f36d0 con 0x7fea1c0c9d20 2024-03-22T12:56:39.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.637+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c0756c0 con 0x7fea1c0c9d20 2024-03-22T12:56:39.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.656+0000 7fea13e00640 1 -- 172.21.15.6:0/2973544783 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fea0400db30 con 0x7fea1c0c9d20 2024-03-22T12:56:39.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.657+0000 7fea13e00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9f806bc20 0x7fe9f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:39.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.657+0000 7fea1be00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9f806bc20 0x7fe9f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:39.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.658+0000 7fea1be00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9f806bc20 0x7fe9f806e0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0012e0 tx=0x7fea0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:39.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.889+0000 7fea13e00640 1 -- 172.21.15.6:0/2973544783 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fea0400de20 con 0x7fea1c0c9d20 2024-03-22T12:56:39.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:39.902+0000 7fea13e00640 1 -- 172.21.15.6:0/2973544783 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fea040599b0 con 0x7fea1c0c9d20 2024-03-22T12:56:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.108+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea1c0b5b30 con 0x7fea1c0c9d20 2024-03-22T12:56:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.221+0000 7fea13e00640 1 -- 172.21.15.6:0/2973544783 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18418 (secure 0 0 0) 0x7fea0405da50 con 0x7fea1c0c9d20 2024-03-22T12:56:40.221 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:40.221 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:37.855636+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9f806bc20 msgr2=0x7fe9f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9f806bc20 0x7fe9f806e0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0012e0 tx=0x7fea0c009040 comp rx=0 tx=0).stop 2024-03-22T12:56:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea1c0c9d20 msgr2=0x7fea1c158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea1c0c9d20 0x7fea1c158fe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea0400f990 tx=0x7fea0400fe50 comp rx=0 tx=0).stop 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 shutdown_connections 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea1c1f0040 0x7fea1c159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe9f806bc20 0x7fe9f806e0e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea1c0c9d20 0x7fea1c158fe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 --2- 172.21.15.6:0/2973544783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea000051c0 0x7fea1c1efb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 >> 172.21.15.6:0/2973544783 conn(0x7fea1c0b16c0 msgr2=0x7fea1c0ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 shutdown_connections 2024-03-22T12:56:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.224+0000 7fea22c00640 1 -- 172.21.15.6:0/2973544783 wait complete. 2024-03-22T12:56:40.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:40.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.475+0000 7f04e4e00640 1 Processor -- start 2024-03-22T12:56:40.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.475+0000 7f04e4e00640 1 -- start start 2024-03-22T12:56:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.476+0000 7f04e4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04e00ca520 0x7f04e006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.476+0000 7f04e4e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e006c7a0 con 0x7f04e00ccad0 2024-03-22T12:56:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.477+0000 7f04e4e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e006ca80 con 0x7f04e015b0e0 2024-03-22T12:56:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.477+0000 7f04e4e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e00cd000 con 0x7f04e00ca520 2024-03-22T12:56:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.478+0000 7f04dde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f04e00ca520 0x7f04e006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.478+0000 7f04de800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f04e00ccad0 0x7f04e0164db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49326/0 (socket says 172.21.15.6:49326) 2024-03-22T12:56:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.478+0000 7f04de800640 1 -- 172.21.15.6:0/3366653324 learned_addr learned my addr 172.21.15.6:0/3366653324 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:40.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.481+0000 7f04dde00640 1 -- 172.21.15.6:0/3366653324 >> v1:172.21.15.92:6789/0 conn(0x7f04e015b0e0 legacy=0x7f04e00cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:40.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.481+0000 7f04dde00640 1 -- 172.21.15.6:0/3366653324 >> v1:172.21.15.6:6789/0 conn(0x7f04e00ccad0 legacy=0x7f04e0164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:40.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.481+0000 7f04dde00640 1 -- 172.21.15.6:0/3366653324 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e006cd60 con 0x7f04e00ca520 2024-03-22T12:56:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.485+0000 7f04dde00640 1 --2- 172.21.15.6:0/3366653324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e00ca520 0x7f04e006d840 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009d90 tx=0x7f04d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9707e6a2b596db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:40.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.489+0000 7f04dd400640 1 -- 172.21.15.6:0/3366653324 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0039070 con 0x7f04e00ca520 2024-03-22T12:56:40.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.490+0000 7f04dd400640 1 -- 172.21.15.6:0/3366653324 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d002cc60 con 0x7f04e00ca520 2024-03-22T12:56:40.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.490+0000 7f04dd400640 1 -- 172.21.15.6:0/3366653324 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0034660 con 0x7f04e00ca520 2024-03-22T12:56:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.490+0000 7f04e4e00640 1 -- 172.21.15.6:0/3366653324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e00ca520 msgr2=0x7f04e006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.490+0000 7f04e4e00640 1 --2- 172.21.15.6:0/3366653324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e00ca520 0x7f04e006d840 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009d90 tx=0x7f04d002beb0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.491+0000 7f04e4e00640 1 -- 172.21.15.6:0/3366653324 shutdown_connections 2024-03-22T12:56:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.491+0000 7f04e4e00640 1 --2- 172.21.15.6:0/3366653324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e00ca520 0x7f04e006d840 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.491+0000 7f04e4e00640 1 -- 172.21.15.6:0/3366653324 >> 172.21.15.6:0/3366653324 conn(0x7f04e00b16b0 msgr2=0x7f04e00b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.491+0000 7f04e4e00640 1 -- 172.21.15.6:0/3366653324 shutdown_connections 2024-03-22T12:56:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.491+0000 7f04e4e00640 1 -- 172.21.15.6:0/3366653324 wait complete. 2024-03-22T12:56:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.492+0000 7f04e4e00640 1 Processor -- start 2024-03-22T12:56:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.493+0000 7f04e4e00640 1 -- start start 2024-03-22T12:56:40.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.493+0000 7f04e4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04e00ca520 0x7f04e01f10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:40.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.493+0000 7f04e4e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04e00ccad0 0x7f04e00b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:40.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.493+0000 7f04e4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e015b0e0 0x7f04e0152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:40.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.493+0000 7f04e4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0154c20 con 0x7f04e015b0e0 2024-03-22T12:56:40.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.493+0000 7f04e4e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e006d320 con 0x7f04e00ca520 2024-03-22T12:56:40.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.493+0000 7f04e4e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e015b980 con 0x7f04e00ccad0 2024-03-22T12:56:40.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.494+0000 7f04de800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e015b0e0 0x7f04e0152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:40.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.494+0000 7f04de800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e015b0e0 0x7f04e0152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44232/0 (socket says 172.21.15.6:44232) 2024-03-22T12:56:40.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.494+0000 7f04de800640 1 -- 172.21.15.6:0/715506386 learned_addr learned my addr 172.21.15.6:0/715506386 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:40.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.497+0000 7f04d5400640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04e00ccad0 0x7f04e00b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:40.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.498+0000 7f04de800640 1 -- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04e00ccad0 msgr2=0x7f04e00b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:40.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.498+0000 7f04de800640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04e00ccad0 0x7f04e00b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.498+0000 7f04de800640 1 -- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04e00ca520 msgr2=0x7f04e01f10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:40.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.498+0000 7f04de800640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04e00ca520 0x7f04e01f10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:40.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.498+0000 7f04de800640 1 -- 172.21.15.6:0/715506386 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00b44b0 con 0x7f04e015b0e0 2024-03-22T12:56:40.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.503+0000 7f04de800640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e015b0e0 0x7f04e0152f10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04d800c450 tx=0x7f04d800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.510+0000 7f04d6a00640 1 -- 172.21.15.6:0/715506386 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8017070 con 0x7f04e015b0e0 2024-03-22T12:56:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.510+0000 7f04d6a00640 1 -- 172.21.15.6:0/715506386 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d8005ce0 con 0x7f04e015b0e0 2024-03-22T12:56:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.510+0000 7f04d6a00640 1 -- 172.21.15.6:0/715506386 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8007270 con 0x7f04e015b0e0 2024-03-22T12:56:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.510+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e00b4750 con 0x7f04e015b0e0 2024-03-22T12:56:40.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.510+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e00b2c80 con 0x7f04e015b0e0 2024-03-22T12:56:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.511+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e00756c0 con 0x7f04e015b0e0 2024-03-22T12:56:40.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.540+0000 7f04d6a00640 1 -- 172.21.15.6:0/715506386 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f04d801a070 con 0x7f04e015b0e0 2024-03-22T12:56:40.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.541+0000 7f04d6a00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04bc06bc20 0x7f04bc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:40.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.541+0000 7f04dde00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04bc06bc20 0x7f04bc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:40.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.542+0000 7f04dde00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04bc06bc20 0x7f04bc06e0e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f04d0036000 tx=0x7f04d002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:40.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.873+0000 7f04d6a00640 1 -- 172.21.15.6:0/715506386 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f04d801de30 con 0x7f04e015b0e0 2024-03-22T12:56:40.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:40.874+0000 7f04d6a00640 1 -- 172.21.15.6:0/715506386 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f04d8058600 con 0x7f04e015b0e0 2024-03-22T12:56:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.085+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04e00cb080 con 0x7f04e015b0e0 2024-03-22T12:56:41.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.256+0000 7f04d6a00640 1 -- 172.21.15.6:0/715506386 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18418 (secure 0 0 0) 0x7f04d805c6a0 con 0x7f04e015b0e0 2024-03-22T12:56:41.257 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:41.257 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:37.855636+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04bc06bc20 msgr2=0x7f04bc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04bc06bc20 0x7f04bc06e0e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f04d0036000 tx=0x7f04d002e000 comp rx=0 tx=0).stop 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e015b0e0 msgr2=0x7f04e0152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e015b0e0 0x7f04e0152f10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04d800c450 tx=0x7f04d800c910 comp rx=0 tx=0).stop 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 shutdown_connections 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04e015b0e0 0x7f04e0152f10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04e00ccad0 0x7f04e00b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04bc06bc20 0x7f04bc06e0e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 --2- 172.21.15.6:0/715506386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04e00ca520 0x7f04e01f10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 >> 172.21.15.6:0/715506386 conn(0x7f04e00b16b0 msgr2=0x7f04e00b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 shutdown_connections 2024-03-22T12:56:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.260+0000 7f04e4e00640 1 -- 172.21.15.6:0/715506386 wait complete. 2024-03-22T12:56:41.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:56:41.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.484+0000 7f81d8a00640 1 Processor -- start 2024-03-22T12:56:41.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.485+0000 7f81d8a00640 1 -- start start 2024-03-22T12:56:41.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.486+0000 7f81d8a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f81d40ca520 0x7f81d406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:41.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.486+0000 7f81d8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d415b0e0 0x7f81d415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:41.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.486+0000 7f81d8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d406c7a0 con 0x7f81d40ca900 2024-03-22T12:56:41.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.486+0000 7f81d8a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d40cd000 con 0x7f81d40ca520 2024-03-22T12:56:41.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.486+0000 7f81d8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d406ca80 con 0x7f81d415b0e0 2024-03-22T12:56:41.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.487+0000 7f81cbe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d415b0e0 0x7f81d415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:41.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.487+0000 7f81d2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81d40ca900 0x7f81d40cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49332/0 (socket says 172.21.15.6:49332) 2024-03-22T12:56:41.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.487+0000 7f81d2800640 1 -- 172.21.15.6:0/3539836759 learned_addr learned my addr 172.21.15.6:0/3539836759 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:41.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.487+0000 7f81d1e00640 1 --2- 172.21.15.6:0/3539836759 >> v2:172.21.15.92:3300/0 conn(0x7f81d40ca520 0x7f81d406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:41.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.490+0000 7f81d1400640 1 -- 172.21.15.6:0/3539836759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583248419 0 0) 0x7f81d406c7a0 con 0x7f81d40ca900 2024-03-22T12:56:41.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.490+0000 7f81d1400640 1 -- 172.21.15.6:0/3539836759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b8003520 con 0x7f81d40ca900 2024-03-22T12:56:41.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.490+0000 7f81cbe00640 1 -- 172.21.15.6:0/3539836759 >> v2:172.21.15.92:3300/0 conn(0x7f81d40ca520 msgr2=0x7f81d406d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:41.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.490+0000 7f81cbe00640 1 --2- 172.21.15.6:0/3539836759 >> v2:172.21.15.92:3300/0 conn(0x7f81d40ca520 0x7f81d406d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.491+0000 7f81cbe00640 1 -- 172.21.15.6:0/3539836759 >> v1:172.21.15.6:6789/0 conn(0x7f81d40ca900 legacy=0x7f81d40cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:41.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.491+0000 7f81cbe00640 1 -- 172.21.15.6:0/3539836759 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d406cd60 con 0x7f81d415b0e0 2024-03-22T12:56:41.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.493+0000 7f81cbe00640 1 --2- 172.21.15.6:0/3539836759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d415b0e0 0x7f81d415b980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0097a0 tx=0x7f81bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa19af765038d5c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:41.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d1400640 1 -- 172.21.15.6:0/3539836759 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc035020 con 0x7f81d415b0e0 2024-03-22T12:56:41.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d1400640 1 -- 172.21.15.6:0/3539836759 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc031830 con 0x7f81d415b0e0 2024-03-22T12:56:41.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d1400640 1 -- 172.21.15.6:0/3539836759 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc031c00 con 0x7f81d415b0e0 2024-03-22T12:56:41.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d8a00640 1 -- 172.21.15.6:0/3539836759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d415b0e0 msgr2=0x7f81d415b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:41.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d8a00640 1 --2- 172.21.15.6:0/3539836759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d415b0e0 0x7f81d415b980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0097a0 tx=0x7f81bc02b8d0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d8a00640 1 -- 172.21.15.6:0/3539836759 shutdown_connections 2024-03-22T12:56:41.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d8a00640 1 --2- 172.21.15.6:0/3539836759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d415b0e0 0x7f81d415b980 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.497+0000 7f81d8a00640 1 --2- 172.21.15.6:0/3539836759 >> v2:172.21.15.92:3300/0 conn(0x7f81d40ca520 0x7f81d406d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.498+0000 7f81d8a00640 1 -- 172.21.15.6:0/3539836759 >> 172.21.15.6:0/3539836759 conn(0x7f81d40b16b0 msgr2=0x7f81d40b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:41.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.498+0000 7f81d8a00640 1 -- 172.21.15.6:0/3539836759 shutdown_connections 2024-03-22T12:56:41.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.498+0000 7f81d8a00640 1 -- 172.21.15.6:0/3539836759 wait complete. 2024-03-22T12:56:41.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.500+0000 7f81d8a00640 1 Processor -- start 2024-03-22T12:56:41.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.500+0000 7f81d8a00640 1 -- start start 2024-03-22T12:56:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.501+0000 7f81d8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d40ca520 0x7f81d40b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.501+0000 7f81d8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d40ca900 0x7f81d40c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.501+0000 7f81d8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81d415b0e0 0x7f81d41f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.501+0000 7f81d8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d40cad60 con 0x7f81d40ca900 2024-03-22T12:56:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.501+0000 7f81d8a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d406d320 con 0x7f81d415b0e0 2024-03-22T12:56:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.501+0000 7f81d8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d4154c20 con 0x7f81d40ca520 2024-03-22T12:56:41.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.502+0000 7f81cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d40ca900 0x7f81d40c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:41.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.502+0000 7f81d2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81d415b0e0 0x7f81d41f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:41.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.503+0000 7f81d1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d40ca520 0x7f81d40b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:41.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.503+0000 7f81cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d40ca900 0x7f81d40c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44246/0 (socket says 172.21.15.6:44246) 2024-03-22T12:56:41.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.503+0000 7f81cbe00640 1 -- 172.21.15.6:0/2723836601 learned_addr learned my addr 172.21.15.6:0/2723836601 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:41.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.508+0000 7f81d2800640 1 -- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d40ca520 msgr2=0x7f81d40b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:41.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.508+0000 7f81d2800640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d40ca520 0x7f81d40b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.508+0000 7f81d2800640 1 -- 172.21.15.6:0/2723836601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d40ca900 msgr2=0x7f81d40c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:41.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.508+0000 7f81d2800640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d40ca900 0x7f81d40c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:41.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.508+0000 7f81d2800640 1 -- 172.21.15.6:0/2723836601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d40c8310 con 0x7f81d415b0e0 2024-03-22T12:56:41.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.510+0000 7f81d2800640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81d415b0e0 0x7f81d41f01c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00cad0 tx=0x7f81cc00ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:41.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.514+0000 7f81ca000640 1 -- 172.21.15.6:0/2723836601 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc017070 con 0x7f81d415b0e0 2024-03-22T12:56:41.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.514+0000 7f81ca000640 1 -- 172.21.15.6:0/2723836601 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00ab70 con 0x7f81d415b0e0 2024-03-22T12:56:41.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.514+0000 7f81ca000640 1 -- 172.21.15.6:0/2723836601 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc012400 con 0x7f81d415b0e0 2024-03-22T12:56:41.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.514+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d40b2880 con 0x7f81d415b0e0 2024-03-22T12:56:41.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.514+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d40c8870 con 0x7f81d415b0e0 2024-03-22T12:56:41.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.515+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d40cf8e0 con 0x7f81d415b0e0 2024-03-22T12:56:41.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.533+0000 7f81ca000640 1 -- 172.21.15.6:0/2723836601 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f81cc01a070 con 0x7f81d415b0e0 2024-03-22T12:56:41.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.535+0000 7f81ca000640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f81b006bc20 0x7f81b006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:41.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.536+0000 7f81ca000640 1 -- 172.21.15.6:0/2723836601 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f81cc08ad80 con 0x7f81d415b0e0 2024-03-22T12:56:41.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.536+0000 7f81d1e00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f81b006bc20 0x7f81b006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:41.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.536+0000 7f81d1e00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f81b006bc20 0x7f81b006e0e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009aa0 tx=0x7f81c4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:41.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:41.782+0000 7f81ca000640 1 -- 172.21.15.6:0/2723836601 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f81d40cf8e0 con 0x7f81d415b0e0 2024-03-22T12:56:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.040+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81d40cb080 con 0x7f81d415b0e0 2024-03-22T12:56:42.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.073+0000 7f81ca000640 1 -- 172.21.15.6:0/2723836601 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6541 (secure 0 0 0) 0x7f81cc059050 con 0x7f81d415b0e0 2024-03-22T12:56:42.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:42.074 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2825293724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2825293724},{"type":"v1","addr":"172.21.15.92:6837","nonce":2825293724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:56:37.865363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4268,"qdb_cluster":[4268]},"id":1}]} 2024-03-22T12:56:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 7 2024-03-22T12:56:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f81b006bc20 msgr2=0x7f81b006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f81b006bc20 0x7f81b006e0e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009aa0 tx=0x7f81c4009190 comp rx=0 tx=0).stop 2024-03-22T12:56:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81d415b0e0 msgr2=0x7f81d41f01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81d415b0e0 0x7f81d41f01c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00cad0 tx=0x7f81cc00ced0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 shutdown_connections 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f81b006bc20 0x7f81b006e0e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81d415b0e0 0x7f81d41f01c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d40ca900 0x7f81d40c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 --2- 172.21.15.6:0/2723836601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f81d40ca520 0x7f81d40b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 >> 172.21.15.6:0/2723836601 conn(0x7f81d40b16b0 msgr2=0x7f81d40b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 shutdown_connections 2024-03-22T12:56:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.076+0000 7f81d8a00640 1 -- 172.21.15.6:0/2723836601 wait complete. 2024-03-22T12:56:42.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:42.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.282+0000 7fb486e00640 1 Processor -- start 2024-03-22T12:56:42.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.282+0000 7fb486e00640 1 -- start start 2024-03-22T12:56:42.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.284+0000 7fb486e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb4800cc790 0x7fb4800ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:42.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.284+0000 7fb486e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb48015ab40 0x7fb48015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:42.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.284+0000 7fb486e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4800ce230 con 0x7fb4800cc3b0 2024-03-22T12:56:42.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.284+0000 7fb486e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4800cd0f0 con 0x7fb48015ab40 2024-03-22T12:56:42.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.284+0000 7fb486e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4800ce510 con 0x7fb4800cc790 2024-03-22T12:56:42.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.284+0000 7fb47fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb4800cc3b0 0x7fb4800cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49346/0 (socket says 172.21.15.6:49346) 2024-03-22T12:56:42.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.284+0000 7fb47fe00640 1 -- 172.21.15.6:0/3826308786 learned_addr learned my addr 172.21.15.6:0/3826308786 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:42.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.286+0000 7fb477e00640 1 --2- 172.21.15.6:0/3826308786 >> v2:172.21.15.92:3300/0 conn(0x7fb4800cc790 0x7fb4800ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:42.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.287+0000 7fb484e00640 1 --2- 172.21.15.6:0/3826308786 >> v2:172.21.15.6:3300/0 conn(0x7fb48015ab40 0x7fb48015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.288+0000 7fb477e00640 1 -- 172.21.15.6:0/3826308786 >> v2:172.21.15.6:3300/0 conn(0x7fb48015ab40 msgr2=0x7fb48015afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.288+0000 7fb477e00640 1 --2- 172.21.15.6:0/3826308786 >> v2:172.21.15.6:3300/0 conn(0x7fb48015ab40 0x7fb48015afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.288+0000 7fb477e00640 1 -- 172.21.15.6:0/3826308786 >> v1:172.21.15.6:6789/0 conn(0x7fb4800cc3b0 legacy=0x7fb4800cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.288+0000 7fb477e00640 1 -- 172.21.15.6:0/3826308786 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4800ce7f0 con 0x7fb4800cc790 2024-03-22T12:56:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.291+0000 7fb477e00640 1 --2- 172.21.15.6:0/3826308786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4800cc790 0x7fb4800ccbb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb48008dd40 tx=0x7fb46802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0e54ae7779fe6a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.294+0000 7fb47f400640 1 -- 172.21.15.6:0/3826308786 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4680315a0 con 0x7fb4800cc790 2024-03-22T12:56:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.294+0000 7fb47f400640 1 -- 172.21.15.6:0/3826308786 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb468037070 con 0x7fb4800cc790 2024-03-22T12:56:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.294+0000 7fb47f400640 1 -- 172.21.15.6:0/3826308786 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468031960 con 0x7fb4800cc790 2024-03-22T12:56:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 -- 172.21.15.6:0/3826308786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4800cc790 msgr2=0x7fb4800ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 --2- 172.21.15.6:0/3826308786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4800cc790 0x7fb4800ccbb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb48008dd40 tx=0x7fb46802b840 comp rx=0 tx=0).stop 2024-03-22T12:56:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 -- 172.21.15.6:0/3826308786 shutdown_connections 2024-03-22T12:56:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 --2- 172.21.15.6:0/3826308786 >> v2:172.21.15.6:3300/0 conn(0x7fb48015ab40 0x7fb48015afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 --2- 172.21.15.6:0/3826308786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4800cc790 0x7fb4800ccbb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 -- 172.21.15.6:0/3826308786 >> 172.21.15.6:0/3826308786 conn(0x7fb4800b16d0 msgr2=0x7fb4800b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 -- 172.21.15.6:0/3826308786 shutdown_connections 2024-03-22T12:56:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.295+0000 7fb486e00640 1 -- 172.21.15.6:0/3826308786 wait complete. 2024-03-22T12:56:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.298+0000 7fb486e00640 1 Processor -- start 2024-03-22T12:56:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.298+0000 7fb486e00640 1 -- start start 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.300+0000 7fb486e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4800cc3b0 0x7fb480156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.300+0000 7fb486e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4800cc790 0x7fb4800c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.300+0000 7fb486e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb48015ab40 0x7fb4801f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.300+0000 7fb486e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48015b4f0 con 0x7fb4800cc790 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.300+0000 7fb486e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800cedb0 con 0x7fb48015ab40 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.300+0000 7fb486e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48006dc10 con 0x7fb4800cc3b0 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.300+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4800cc3b0 0x7fb480156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:42.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.301+0000 7fb484e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb48015ab40 0x7fb4801f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.301+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4800cc3b0 0x7fb480156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57736/0 (socket says 172.21.15.6:57736) 2024-03-22T12:56:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.301+0000 7fb47fe00640 1 -- 172.21.15.6:0/3857740649 learned_addr learned my addr 172.21.15.6:0/3857740649 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:42.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.306+0000 7fb484e00640 1 -- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4800cc3b0 msgr2=0x7fb480156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.306+0000 7fb484e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4800cc3b0 0x7fb480156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.306+0000 7fb484e00640 1 -- 172.21.15.6:0/3857740649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4800cc790 msgr2=0x7fb4800c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.306+0000 7fb484e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4800cc790 0x7fb4800c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.306+0000 7fb484e00640 1 -- 172.21.15.6:0/3857740649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480157ef0 con 0x7fb48015ab40 2024-03-22T12:56:42.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.309+0000 7fb484e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb48015ab40 0x7fb4801f3810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb478007cf0 tx=0x7fb47800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:42.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.312+0000 7fb47cc00640 1 -- 172.21.15.6:0/3857740649 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47800edd0 con 0x7fb48015ab40 2024-03-22T12:56:42.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.312+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4801f42d0 con 0x7fb48015ab40 2024-03-22T12:56:42.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.312+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb480157410 con 0x7fb48015ab40 2024-03-22T12:56:42.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.313+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4800756c0 con 0x7fb48015ab40 2024-03-22T12:56:42.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.615+0000 7fb47cc00640 1 -- 172.21.15.6:0/3857740649 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb478003990 con 0x7fb48015ab40 2024-03-22T12:56:42.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.615+0000 7fb47cc00640 1 -- 172.21.15.6:0/3857740649 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4780098b0 con 0x7fb48015ab40 2024-03-22T12:56:42.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.618+0000 7fb47cc00640 1 -- 172.21.15.6:0/3857740649 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb47801b020 con 0x7fb48015ab40 2024-03-22T12:56:42.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.620+0000 7fb47cc00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb45c06bbd0 0x7fb45c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:42.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.621+0000 7fb47cc00640 1 -- 172.21.15.6:0/3857740649 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb47808bca0 con 0x7fb48015ab40 2024-03-22T12:56:42.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.621+0000 7fb47fe00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb45c06bbd0 0x7fb45c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:42.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.622+0000 7fb47fe00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb45c06bbd0 0x7fb45c06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb470004e40 tx=0x7fb470007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:42.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.630+0000 7fb47cc00640 1 -- 172.21.15.6:0/3857740649 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb47805a530 con 0x7fb48015ab40 2024-03-22T12:56:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.851+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb48006de80 con 0x7fb48015ab40 2024-03-22T12:56:42.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.890+0000 7fb47cc00640 1 -- 172.21.15.6:0/3857740649 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18418 (secure 0 0 0) 0x7fb478059e30 con 0x7fb48015ab40 2024-03-22T12:56:42.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:42.890 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:37.855636+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:42.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb45c06bbd0 msgr2=0x7fb45c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb45c06bbd0 0x7fb45c06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb470004e40 tx=0x7fb470007f10 comp rx=0 tx=0).stop 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb48015ab40 msgr2=0x7fb4801f3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb48015ab40 0x7fb4801f3810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb478007cf0 tx=0x7fb47800e4b0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 shutdown_connections 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb45c06bbd0 0x7fb45c06e090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb48015ab40 0x7fb4801f3810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4800cc790 0x7fb4800c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.892+0000 7fb486e00640 1 --2- 172.21.15.6:0/3857740649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4800cc3b0 0x7fb480156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.893+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 >> 172.21.15.6:0/3857740649 conn(0x7fb4800b16d0 msgr2=0x7fb480149c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.893+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 shutdown_connections 2024-03-22T12:56:42.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:42.893+0000 7fb486e00640 1 -- 172.21.15.6:0/3857740649 wait complete. 2024-03-22T12:56:43.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:56:37.865363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4268, 'qdb_cluster': [4268]} 2024-03-22T12:56:43.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-22T12:56:43.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-22T12:56:43.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.723+0000 7fd69e200640 1 Processor -- start 2024-03-22T12:56:43.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.723+0000 7fd69e200640 1 -- start start 2024-03-22T12:56:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.725+0000 7fd69e200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd6980cc400 0x7fd6980cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.725+0000 7fd69e200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd698164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.725+0000 7fd69e200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69815a670 con 0x7fd6980cd7f0 2024-03-22T12:56:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.725+0000 7fd69e200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69806dd30 con 0x7fd6980cc400 2024-03-22T12:56:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.725+0000 7fd69e200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69815a930 con 0x7fd6980cf0e0 2024-03-22T12:56:43.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.728+0000 7fd69d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd698164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.728+0000 7fd69d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd698164c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47440/0 (socket says 172.21.15.6:47440) 2024-03-22T12:56:43.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.728+0000 7fd69d800640 1 -- 172.21.15.6:0/585353671 learned_addr learned my addr 172.21.15.6:0/585353671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.730+0000 7fd69d800640 1 -- 172.21.15.6:0/585353671 >> v2:172.21.15.92:3300/0 conn(0x7fd6980cc400 msgr2=0x7fd6980cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.730+0000 7fd69d800640 1 --2- 172.21.15.6:0/585353671 >> v2:172.21.15.92:3300/0 conn(0x7fd6980cc400 0x7fd6980cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.730+0000 7fd69d800640 1 -- 172.21.15.6:0/585353671 >> v1:172.21.15.92:6789/0 conn(0x7fd6980cd7f0 legacy=0x7fd6980ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.730+0000 7fd69d800640 1 -- 172.21.15.6:0/585353671 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69815ac10 con 0x7fd6980cf0e0 2024-03-22T12:56:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.732+0000 7fd69d800640 1 --2- 172.21.15.6:0/585353671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd698164c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd690009e50 tx=0x7fd69002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a5a1c82daa4fc34 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.738+0000 7fd697e00640 1 -- 172.21.15.6:0/585353671 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69002c9e0 con 0x7fd6980cf0e0 2024-03-22T12:56:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.738+0000 7fd697e00640 1 -- 172.21.15.6:0/585353671 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69002cb80 con 0x7fd6980cf0e0 2024-03-22T12:56:43.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.738+0000 7fd697e00640 1 -- 172.21.15.6:0/585353671 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6900343f0 con 0x7fd6980cf0e0 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 -- 172.21.15.6:0/585353671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 msgr2=0x7fd698164c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 --2- 172.21.15.6:0/585353671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd698164c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd690009e50 tx=0x7fd69002be40 comp rx=0 tx=0).stop 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 -- 172.21.15.6:0/585353671 shutdown_connections 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 --2- 172.21.15.6:0/585353671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd698164c00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 --2- 172.21.15.6:0/585353671 >> v2:172.21.15.92:3300/0 conn(0x7fd6980cc400 0x7fd6980cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 -- 172.21.15.6:0/585353671 >> 172.21.15.6:0/585353671 conn(0x7fd6980cf9f0 msgr2=0x7fd6980cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 -- 172.21.15.6:0/585353671 shutdown_connections 2024-03-22T12:56:43.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.739+0000 7fd69e200640 1 -- 172.21.15.6:0/585353671 wait complete. 2024-03-22T12:56:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.741+0000 7fd69e200640 1 Processor -- start 2024-03-22T12:56:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.741+0000 7fd69e200640 1 -- start start 2024-03-22T12:56:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.742+0000 7fd69e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6980cc400 0x7fd6981f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.742+0000 7fd69e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6980cd7f0 0x7fd698153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.742+0000 7fd69e200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd6980b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.742+0000 7fd69e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980ce1d0 con 0x7fd6980cc400 2024-03-22T12:56:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.742+0000 7fd69e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69815b1d0 con 0x7fd6980cd7f0 2024-03-22T12:56:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.742+0000 7fd69e200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980ce4b0 con 0x7fd6980cf0e0 2024-03-22T12:56:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.743+0000 7fd69ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6980cc400 0x7fd6981f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.743+0000 7fd69ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6980cc400 0x7fd6981f0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37026/0 (socket says 172.21.15.6:37026) 2024-03-22T12:56:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.743+0000 7fd69ce00640 1 -- 172.21.15.6:0/3474515280 learned_addr learned my addr 172.21.15.6:0/3474515280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.744+0000 7fd69d800640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd6980b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.747+0000 7fd69ce00640 1 -- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 msgr2=0x7fd6980b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.747+0000 7fd69ce00640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd6980b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.747+0000 7fd69ce00640 1 -- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6980cd7f0 msgr2=0x7fd698153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.747+0000 7fd69ce00640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6980cd7f0 0x7fd698153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.747+0000 7fd69ce00640 1 -- 172.21.15.6:0/3474515280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6980b2660 con 0x7fd6980cc400 2024-03-22T12:56:43.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.752+0000 7fd69ce00640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6980cc400 0x7fd6981f0940 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b4d0 tx=0x7fd68000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.792+0000 7fd695600640 1 -- 172.21.15.6:0/3474515280 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680014070 con 0x7fd6980cc400 2024-03-22T12:56:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.792+0000 7fd695600640 1 -- 172.21.15.6:0/3474515280 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd680010870 con 0x7fd6980cc400 2024-03-22T12:56:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.792+0000 7fd695600640 1 -- 172.21.15.6:0/3474515280 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680010c60 con 0x7fd6980cc400 2024-03-22T12:56:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.792+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd698159d30 con 0x7fd6980cc400 2024-03-22T12:56:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.792+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6980b2be0 con 0x7fd6980cc400 2024-03-22T12:56:43.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.796+0000 7fd6a2635740 1 Processor -- start 2024-03-22T12:56:43.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.796+0000 7fd6a2635740 1 -- start start 2024-03-22T12:56:43.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.797+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc317ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.797+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31c86b0 0x563cc317f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.797+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317d290 0x563cc31be1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.797+0000 7fd6a2635740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc30e2c90 con 0x563cc317e830 2024-03-22T12:56:43.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.797+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc308b980 con 0x563cc317d290 2024-03-22T12:56:43.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.797+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x563cc30ded30 con 0x563cc31c86b0 2024-03-22T12:56:43.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.798+0000 7fd69ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31c86b0 0x563cc317f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.798+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317d290 0x563cc31be1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.799+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317d290 0x563cc31be1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51420/0 (socket says 172.21.15.6:51420) 2024-03-22T12:56:43.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.799+0000 7fd68fe00640 1 -- 172.21.15.6:0/1568840363 learned_addr learned my addr 172.21.15.6:0/1568840363 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:43.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.801+0000 7fd69ce00640 1 -- 172.21.15.6:0/1568840363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317d290 msgr2=0x563cc31be1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:43.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.801+0000 7fd69ce00640 1 --2- 172.21.15.6:0/1568840363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317d290 0x563cc31be1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.801+0000 7fd69ce00640 1 -- 172.21.15.6:0/1568840363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 msgr2=0x563cc317ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:43.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.801+0000 7fd69ce00640 1 --2- 172.21.15.6:0/1568840363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc317ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.801+0000 7fd69ce00640 1 -- 172.21.15.6:0/1568840363 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cc30a3510 con 0x563cc31c86b0 2024-03-22T12:56:43.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.803+0000 7fd69ce00640 1 --2- 172.21.15.6:0/1568840363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31c86b0 0x563cc317f150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b4a0 tx=0x7fd680010000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39be6c96229d2589 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.806+0000 7fd68d600640 1 -- 172.21.15.6:0/1568840363 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680033070 con 0x563cc31c86b0 2024-03-22T12:56:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.806+0000 7fd68d600640 1 -- 172.21.15.6:0/1568840363 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6800234b0 con 0x563cc31c86b0 2024-03-22T12:56:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.806+0000 7fd68d600640 1 -- 172.21.15.6:0/1568840363 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68001e060 con 0x563cc31c86b0 2024-03-22T12:56:43.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 -- 172.21.15.6:0/1568840363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31c86b0 msgr2=0x563cc317f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:43.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1568840363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31c86b0 0x563cc317f150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b4a0 tx=0x7fd680010000 comp rx=0 tx=0).stop 2024-03-22T12:56:43.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 -- 172.21.15.6:0/1568840363 shutdown_connections 2024-03-22T12:56:43.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1568840363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317d290 0x563cc31be1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1568840363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31c86b0 0x563cc317f150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1568840363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc317ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 -- 172.21.15.6:0/1568840363 >> 172.21.15.6:0/1568840363 conn(0x563cc317afa0 msgr2=0x563cc317d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 -- 172.21.15.6:0/1568840363 shutdown_connections 2024-03-22T12:56:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.807+0000 7fd6a2635740 1 -- 172.21.15.6:0/1568840363 wait complete. 2024-03-22T12:56:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.808+0000 7fd6a2635740 1 -- start start 2024-03-22T12:56:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.808+0000 7fd6a2635740 1 Processor -- start 2024-03-22T12:56:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.809+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317e830 0x563cc31ba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.809+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 0x563cc31c19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.809+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc31bc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.809+0000 7fd6a2635740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc309edd0 con 0x563cc31c86b0 2024-03-22T12:56:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.809+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc31c8bf0 con 0x563cc317e830 2024-03-22T12:56:43.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.809+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x563cc31a4940 con 0x563cc31a3f30 2024-03-22T12:56:43.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.809+0000 7fd6a2635740 10 client.0 _ll_register_callbacks cb 0x563cc2df6ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T12:56:43.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.810+0000 7fd69d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317e830 0x563cc31ba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.810+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc31bc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.810+0000 7fd69d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317e830 0x563cc31ba960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51426/0 (socket says 172.21.15.6:51426) 2024-03-22T12:56:43.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.810+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc31bc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47472/0 (socket says 172.21.15.6:47472) 2024-03-22T12:56:43.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.810+0000 7fd69d800640 1 -- 172.21.15.6:0/3630046055 learned_addr learned my addr 172.21.15.6:0/3630046055 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:43.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.813+0000 7fd69d800640 1 -- 172.21.15.6:0/3630046055 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 msgr2=0x563cc31bc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:43.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.813+0000 7fd69d800640 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc31bc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.813+0000 7fd69d800640 1 -- 172.21.15.6:0/3630046055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 msgr2=0x563cc31c19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:43.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.813+0000 7fd69d800640 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 0x563cc31c19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:43.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.813+0000 7fd69d800640 1 -- 172.21.15.6:0/3630046055 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cc2df72d0 con 0x563cc317e830 2024-03-22T12:56:43.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.815+0000 7fd69d800640 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317e830 0x563cc31ba960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd690009f70 tx=0x7fd690004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c912eee6d123a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:43.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.815+0000 7fd686000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-03-22T12:56:43.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.819+0000 7fd686000640 1 -- 172.21.15.6:0/3630046055 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690045070 con 0x563cc317e830 2024-03-22T12:56:43.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.819+0000 7fd686000640 1 -- 172.21.15.6:0/3630046055 <== mon.1 v2:172.21.15.92:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd690034930 con 0x563cc317e830 2024-03-22T12:56:43.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.819+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6800234b0 con 0x563cc317e830 2024-03-22T12:56:43.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.820+0000 7fd695600640 1 -- 172.21.15.6:0/3474515280 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd680017070 con 0x7fd6980cc400 2024-03-22T12:56:43.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.821+0000 7fd686000640 1 -- 172.21.15.6:0/3630046055 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd690050050 con 0x563cc317e830 2024-03-22T12:56:43.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.821+0000 7fd6a2635740 10 client.4354 fetch_fsmap learned FSMap version 7 2024-03-22T12:56:43.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.821+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x563cc2ec16b0 con 0x563cc317e830 2024-03-22T12:56:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.822+0000 7fd695600640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd67406bb90 0x7fd67406e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.822+0000 7fd69ce00640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd67406bb90 0x7fd67406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.823+0000 7fd69ce00640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd67406bb90 0x7fd67406e050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd680062ec0 tx=0x7fd68005d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd695600640 1 -- 172.21.15.6:0/3474515280 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd68005eb40 con 0x7fd6980cc400 2024-03-22T12:56:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd686000640 1 -- 172.21.15.6:0/3630046055 <== mon.1 v2:172.21.15.92:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6800234b0 con 0x563cc317e830 2024-03-22T12:56:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd686000640 1 -- 172.21.15.6:0/3630046055 <== mon.1 v2:172.21.15.92:3300/0 5 ==== fsmap(e 7) v1 ==== 4472+0+0 (secure 0 0 0) 0x7fd690053020 con 0x563cc317e830 2024-03-22T12:56:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd6a2635740 10 client.4354 fetch_fsmap finished waiting for FSMap version 7 2024-03-22T12:56:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd6a2635740 10 client.4354 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-22T12:56:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd6a2635740 20 client.4354 populate_metadata read hostname 'smithi006' 2024-03-22T12:56:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd6a2635740 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31d2840 0x563cc31ddd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd6a2635740 4 client.4354 mds_command: new command op to 4268 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T12:56:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.825+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 --> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cc3084520 con 0x563cc31d2840 2024-03-22T12:56:43.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.835+0000 7fd68fe00640 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31d2840 0x563cc31ddd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:43.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.932+0000 7fd68fe00640 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31d2840 0x563cc31ddd00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3d875fb20f54767 server_cookie=c45c65558a93651f in_seq=0 out_seq=0 2024-03-22T12:56:43.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:43.932+0000 7fd686000640 10 client.4354 ms_handle_connect on v2:172.21.15.6:6834/1733016967 2024-03-22T12:56:44.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.154+0000 7fd686000640 1 -- 172.21.15.6:0/3630046055 <== mds.0 v2:172.21.15.6:6834/1733016967 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563cc3084520 con 0x563cc31d2840 2024-03-22T12:56:44.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.154+0000 7fd686000640 10 client.4354 handle_command_reply: tid=0 2024-03-22T12:56:44.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.155+0000 7fd6a2635740 1 client.4354 shutdown 2024-03-22T12:56:44.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.155+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317e830 msgr2=0x563cc31ba960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.155+0000 7fd6a2635740 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317e830 0x563cc31ba960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd690009f70 tx=0x7fd690004a40 comp rx=0 tx=0).stop 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.155+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 shutdown_connections 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31d2840 0x563cc31ddd00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc31bc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 0x563cc31c19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 1 --2- 172.21.15.6:0/3630046055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc317e830 0x563cc31ba960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 >> 172.21.15.6:0/3630046055 conn(0x563cc317d290 msgr2=0x563cc31a7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd686000640 0 client.4354 ms_handle_reset on v2:172.21.15.6:6834/1733016967 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 shutdown_connections 2024-03-22T12:56:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 1 -- 172.21.15.6:0/3630046055 wait complete. 2024-03-22T12:56:44.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.156+0000 7fd6a2635740 20 client.4354 trim_cache size 0 max 16384 2024-03-22T12:56:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.217+0000 7fd6a2635740 1 Processor -- start 2024-03-22T12:56:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.217+0000 7fd6a2635740 1 -- start start 2024-03-22T12:56:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.219+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc317e830 0x563cc31a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.219+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31a3f30 0x563cc31b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.219+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 0x563cc3177c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.219+0000 7fd6a2635740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6981fc370 con 0x563cc31c86b0 2024-03-22T12:56:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.219+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6981fc970 con 0x563cc31a3f30 2024-03-22T12:56:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.219+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x563cc31a4770 con 0x563cc317e830 2024-03-22T12:56:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.220+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31a3f30 0x563cc31b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.220+0000 7fd69d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc317e830 0x563cc31a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.220+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31a3f30 0x563cc31b8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51428/0 (socket says 172.21.15.6:51428) 2024-03-22T12:56:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.220+0000 7fd68fe00640 1 -- 172.21.15.6:0/1512404749 learned_addr learned my addr 172.21.15.6:0/1512404749 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.221+0000 7fd69d800640 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 0x563cc3177c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.223+0000 7fd68fe00640 1 -- 172.21.15.6:0/1512404749 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc317e830 msgr2=0x563cc31a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:44.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.223+0000 7fd68fe00640 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc317e830 0x563cc31a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.223+0000 7fd68fe00640 1 -- 172.21.15.6:0/1512404749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 msgr2=0x563cc3177c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:44.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.224+0000 7fd68fe00640 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 0x563cc3177c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.224+0000 7fd68fe00640 1 -- 172.21.15.6:0/1512404749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cc30ef310 con 0x563cc31a3f30 2024-03-22T12:56:44.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.226+0000 7fd68fe00640 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31a3f30 0x563cc31b8e20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x563cc31803f0 tx=0x7fd688007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997f2ac5f1a8183a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:44.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.230+0000 7fd68cc00640 1 -- 172.21.15.6:0/1512404749 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688001d20 con 0x563cc31a3f30 2024-03-22T12:56:44.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.230+0000 7fd68cc00640 1 -- 172.21.15.6:0/1512404749 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6880050d0 con 0x563cc31a3f30 2024-03-22T12:56:44.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.230+0000 7fd68cc00640 1 -- 172.21.15.6:0/1512404749 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688015860 con 0x563cc31a3f30 2024-03-22T12:56:44.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 -- 172.21.15.6:0/1512404749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31a3f30 msgr2=0x563cc31b8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31a3f30 0x563cc31b8e20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x563cc31803f0 tx=0x7fd688007df0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 -- 172.21.15.6:0/1512404749 shutdown_connections 2024-03-22T12:56:44.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc31c86b0 0x563cc3177c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31a3f30 0x563cc31b8e20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1512404749 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc317e830 0x563cc31a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 -- 172.21.15.6:0/1512404749 >> 172.21.15.6:0/1512404749 conn(0x563cc317d290 msgr2=0x563cc31bcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.231+0000 7fd6a2635740 1 -- 172.21.15.6:0/1512404749 shutdown_connections 2024-03-22T12:56:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.232+0000 7fd6a2635740 1 -- 172.21.15.6:0/1512404749 wait complete. 2024-03-22T12:56:44.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.233+0000 7fd6a2635740 1 -- start start 2024-03-22T12:56:44.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.233+0000 7fd6a2635740 1 Processor -- start 2024-03-22T12:56:44.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.235+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc31b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.235+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc3232c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.235+0000 7fd6a2635740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31c86b0 0x563cc3228f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.235+0000 7fd6a2635740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc31d26c0 con 0x563cc317e830 2024-03-22T12:56:44.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.235+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cc31a25f0 con 0x563cc31c86b0 2024-03-22T12:56:44.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.235+0000 7fd6a2635740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x563cc31d2cc0 con 0x563cc31a3f30 2024-03-22T12:56:44.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.235+0000 7fd6a2635740 10 client.0 _ll_register_callbacks cb 0x7fd69002cc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T12:56:44.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.236+0000 7fd69d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31c86b0 0x563cc3228f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.236+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc3232c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.236+0000 7fd69d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc31b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.236+0000 7fd69d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31c86b0 0x563cc3228f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51430/0 (socket says 172.21.15.6:51430) 2024-03-22T12:56:44.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.236+0000 7fd69d800640 1 -- 172.21.15.6:0/1888533082 learned_addr learned my addr 172.21.15.6:0/1888533082 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:44.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.236+0000 7fd68fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc3232c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47492/0 (socket says 172.21.15.6:47492) 2024-03-22T12:56:44.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.241+0000 7fd69d800640 1 -- 172.21.15.6:0/1888533082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 msgr2=0x563cc3232c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:44.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.241+0000 7fd69d800640 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc3232c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.241+0000 7fd69d800640 1 -- 172.21.15.6:0/1888533082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31c86b0 msgr2=0x563cc3228f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:44.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.241+0000 7fd69d800640 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31c86b0 0x563cc3228f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.241+0000 7fd69d800640 1 -- 172.21.15.6:0/1888533082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cc3084520 con 0x563cc317e830 2024-03-22T12:56:44.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.243+0000 7fd69d800640 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc31b9030 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd69002cd00 tx=0x7fd690040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f9be865eb1f599 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:44.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.243+0000 7fd686000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-22T12:56:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.249+0000 7fd686000640 1 -- 172.21.15.6:0/1888533082 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690045040 con 0x563cc317e830 2024-03-22T12:56:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.249+0000 7fd686000640 1 -- 172.21.15.6:0/1888533082 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd690036a30 con 0x563cc317e830 2024-03-22T12:56:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.250+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6880050d0 con 0x563cc317e830 2024-03-22T12:56:44.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.261+0000 7fd686000640 1 -- 172.21.15.6:0/1888533082 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd690036c30 con 0x563cc317e830 2024-03-22T12:56:44.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.261+0000 7fd6a2635740 10 client.4260 fetch_fsmap learned FSMap version 7 2024-03-22T12:56:44.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.261+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x563cc31c8480 con 0x563cc317e830 2024-03-22T12:56:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd686000640 1 -- 172.21.15.6:0/1888533082 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6880050d0 con 0x563cc317e830 2024-03-22T12:56:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd686000640 1 -- 172.21.15.6:0/1888533082 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 7) v1 ==== 4472+0+0 (secure 0 0 0) 0x7fd690008070 con 0x563cc317e830 2024-03-22T12:56:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd6a2635740 10 client.4260 fetch_fsmap finished waiting for FSMap version 7 2024-03-22T12:56:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd6a2635740 10 client.4260 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-22T12:56:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd6a2635740 20 client.4260 populate_metadata read hostname 'smithi006' 2024-03-22T12:56:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31a0bf0 0x563cc31a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd6a2635740 4 client.4260 mds_command: new command op to 4268 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T12:56:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.270+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 --> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563cc317b700 con 0x563cc31a0bf0 2024-03-22T12:56:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.272+0000 7fd68fe00640 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31a0bf0 0x563cc31a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.277+0000 7fd68fe00640 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31a0bf0 0x563cc31a0fd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a21e0e15bde7c632 server_cookie=3e7b48fba5fcd03c in_seq=0 out_seq=0 2024-03-22T12:56:44.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.277+0000 7fd686000640 10 client.4260 ms_handle_connect on v2:172.21.15.6:6834/1733016967 2024-03-22T12:56:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.315+0000 7fd686000640 1 -- 172.21.15.6:0/1888533082 <== mds.0 v2:172.21.15.6:6834/1733016967 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x563cc317b700 con 0x563cc31a0bf0 2024-03-22T12:56:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.315+0000 7fd686000640 10 client.4260 handle_command_reply: tid=0 2024-03-22T12:56:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.315+0000 7fd6a2635740 1 client.4260 shutdown 2024-03-22T12:56:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.315+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 msgr2=0x563cc31b9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc31b9030 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd69002cd00 tx=0x7fd690040000 comp rx=0 tx=0).stop 2024-03-22T12:56:44.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 shutdown_connections 2024-03-22T12:56:44.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:6834/1733016967,v1:172.21.15.6:6835/1733016967] conn(0x563cc31a0bf0 0x563cc31a0fd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563cc31c86b0 0x563cc3228f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x563cc31a3f30 0x563cc3232c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 --2- 172.21.15.6:0/1888533082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563cc317e830 0x563cc31b9030 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd686000640 0 client.4260 ms_handle_reset on v2:172.21.15.6:6834/1733016967 2024-03-22T12:56:44.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 >> 172.21.15.6:0/1888533082 conn(0x563cc317d290 msgr2=0x563cc31a1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:44.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 shutdown_connections 2024-03-22T12:56:44.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.316+0000 7fd6a2635740 1 -- 172.21.15.6:0/1888533082 wait complete. 2024-03-22T12:56:44.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.317+0000 7fd6a2635740 20 client.4260 trim_cache size 0 max 16384 2024-03-22T12:56:44.317 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "1722e499-f2bc-49ac-a1c9-7ab0d7863db9", 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4268, 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 12.989606075999999, 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 7, 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 28, 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 28, 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 22.523284344 2024-03-22T12:56:44.318 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T12:56:44.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.318+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd67406bb90 msgr2=0x7fd67406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.318+0000 7fd69e200640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd67406bb90 0x7fd67406e050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd680062ec0 tx=0x7fd68005d040 comp rx=0 tx=0).stop 2024-03-22T12:56:44.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.318+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6980cc400 msgr2=0x7fd6981f0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.318+0000 7fd69e200640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6980cc400 0x7fd6981f0940 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b4d0 tx=0x7fd68000b990 comp rx=0 tx=0).stop 2024-03-22T12:56:44.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 shutdown_connections 2024-03-22T12:56:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6980cf0e0 0x7fd6980b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd67406bb90 0x7fd67406e050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6980cd7f0 0x7fd698153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 --2- 172.21.15.6:0/3474515280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6980cc400 0x7fd6981f0940 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 >> 172.21.15.6:0/3474515280 conn(0x7fd6980cf9f0 msgr2=0x7fd6980ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 shutdown_connections 2024-03-22T12:56:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.319+0000 7fd69e200640 1 -- 172.21.15.6:0/3474515280 wait complete. 2024-03-22T12:56:44.351 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-22T12:56:44.351 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:44.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-22T12:56:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.573+0000 7f1a30800640 1 Processor -- start 2024-03-22T12:56:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.573+0000 7f1a30800640 1 -- start start 2024-03-22T12:56:44.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.575+0000 7f1a30800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a28152400 0x7f1a2815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.575+0000 7f1a30800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.575+0000 7f1a30800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a281590b0 con 0x7f1a2806f8c0 2024-03-22T12:56:44.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.575+0000 7f1a30800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a280724d0 con 0x7f1a28152400 2024-03-22T12:56:44.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.575+0000 7f1a30800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280721f0 con 0x7f1a2815bc80 2024-03-22T12:56:44.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.576+0000 7f1a2de00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a28152400 0x7f1a2815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.576+0000 7f1a2d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.576+0000 7f1a2e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1a2806f8c0 0x7f1a280c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41566/0 (socket says 172.21.15.6:41566) 2024-03-22T12:56:44.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.576+0000 7f1a2e800640 1 -- 172.21.15.6:0/4026247571 learned_addr learned my addr 172.21.15.6:0/4026247571 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.579+0000 7f1a2de00640 1 -- 172.21.15.6:0/4026247571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 msgr2=0x7f1a280716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.579+0000 7f1a2de00640 1 --2- 172.21.15.6:0/4026247571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.579+0000 7f1a2de00640 1 -- 172.21.15.6:0/4026247571 >> v1:172.21.15.6:6789/0 conn(0x7f1a2806f8c0 legacy=0x7f1a280c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.579+0000 7f1a2de00640 1 -- 172.21.15.6:0/4026247571 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a281582f0 con 0x7f1a28152400 2024-03-22T12:56:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.582+0000 7f1a2de00640 1 --2- 172.21.15.6:0/4026247571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a28152400 0x7f1a2815b740 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0097a0 tx=0x7f1a1c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a955479d164ec544 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.585+0000 7f1a2ca00640 1 -- 172.21.15.6:0/4026247571 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c034070 con 0x7f1a28152400 2024-03-22T12:56:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a2ca00640 1 -- 172.21.15.6:0/4026247571 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c030610 con 0x7f1a28152400 2024-03-22T12:56:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a2ca00640 1 -- 172.21.15.6:0/4026247571 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0309e0 con 0x7f1a28152400 2024-03-22T12:56:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a30800640 1 -- 172.21.15.6:0/4026247571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a28152400 msgr2=0x7f1a2815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a30800640 1 --2- 172.21.15.6:0/4026247571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a28152400 0x7f1a2815b740 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0097a0 tx=0x7f1a1c02b960 comp rx=0 tx=0).stop 2024-03-22T12:56:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a30800640 1 -- 172.21.15.6:0/4026247571 shutdown_connections 2024-03-22T12:56:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a30800640 1 --2- 172.21.15.6:0/4026247571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a30800640 1 --2- 172.21.15.6:0/4026247571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a28152400 0x7f1a2815b740 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.586+0000 7f1a30800640 1 -- 172.21.15.6:0/4026247571 >> 172.21.15.6:0/4026247571 conn(0x7f1a280cb930 msgr2=0x7f1a280cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.587+0000 7f1a30800640 1 -- 172.21.15.6:0/4026247571 shutdown_connections 2024-03-22T12:56:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.587+0000 7f1a30800640 1 -- 172.21.15.6:0/4026247571 wait complete. 2024-03-22T12:56:44.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.588+0000 7f1a30800640 1 Processor -- start 2024-03-22T12:56:44.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.589+0000 7f1a30800640 1 -- start start 2024-03-22T12:56:44.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.589+0000 7f1a30800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a2806f8c0 0x7f1a280c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.589+0000 7f1a30800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a28152400 0x7f1a280c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.589+0000 7f1a30800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.589+0000 7f1a30800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28158b90 con 0x7f1a28152400 2024-03-22T12:56:44.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.589+0000 7f1a30800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a281588b0 con 0x7f1a2806f8c0 2024-03-22T12:56:44.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.589+0000 7f1a30800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a281527e0 con 0x7f1a2815bc80 2024-03-22T12:56:44.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.590+0000 7f1a2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a28152400 0x7f1a280c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.590+0000 7f1a2de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a2806f8c0 0x7f1a280c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.590+0000 7f1a2e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.590+0000 7f1a2d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a28152400 0x7f1a280c4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37056/0 (socket says 172.21.15.6:37056) 2024-03-22T12:56:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.590+0000 7f1a2d400640 1 -- 172.21.15.6:0/2265941113 learned_addr learned my addr 172.21.15.6:0/2265941113 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:44.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.594+0000 7f1a2d400640 1 -- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 msgr2=0x7f1a280c19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.594+0000 7f1a2d400640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280c19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.594+0000 7f1a2d400640 1 -- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a2806f8c0 msgr2=0x7f1a280c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:44.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.594+0000 7f1a2d400640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a2806f8c0 0x7f1a280c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:44.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.594+0000 7f1a2d400640 1 -- 172.21.15.6:0/2265941113 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a280c4090 con 0x7f1a28152400 2024-03-22T12:56:44.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.597+0000 7f1a2d400640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a28152400 0x7f1a280c4870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800e840 tx=0x7f1a1800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:44.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.602+0000 7f1a22000640 1 -- 172.21.15.6:0/2265941113 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18017070 con 0x7f1a28152400 2024-03-22T12:56:44.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.602+0000 7f1a22000640 1 -- 172.21.15.6:0/2265941113 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1800c940 con 0x7f1a28152400 2024-03-22T12:56:44.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.602+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a280c4db0 con 0x7f1a28152400 2024-03-22T12:56:44.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.602+0000 7f1a22000640 1 -- 172.21.15.6:0/2265941113 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1800cd10 con 0x7f1a28152400 2024-03-22T12:56:44.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.602+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2815ac50 con 0x7f1a28152400 2024-03-22T12:56:44.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.603+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2814f9c0 con 0x7f1a28152400 2024-03-22T12:56:44.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.637+0000 7f1a22000640 1 -- 172.21.15.6:0/2265941113 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a1801a070 con 0x7f1a28152400 2024-03-22T12:56:44.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.640+0000 7f1a22000640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a0c06bc20 0x7f1a0c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.640+0000 7f1a2de00640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a0c06bc20 0x7f1a0c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.641+0000 7f1a22000640 1 -- 172.21.15.6:0/2265941113 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1a18057590 con 0x7f1a28152400 2024-03-22T12:56:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.641+0000 7f1a2de00640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a0c06bc20 0x7f1a0c06e0e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c036000 tx=0x7f1a1c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:44.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:44.952+0000 7f1a22000640 1 -- 172.21.15.6:0/2265941113 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a2814f9c0 con 0x7f1a28152400 2024-03-22T12:56:45.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.209+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1a280756c0 con 0x7f1a28152400 2024-03-22T12:56:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.279+0000 7f1a22000640 1 -- 172.21.15.6:0/2265941113 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1a180594f0 con 0x7f1a28152400 2024-03-22T12:56:45.279 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-22T12:56:45.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a0c06bc20 msgr2=0x7f1a0c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:45.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a0c06bc20 0x7f1a0c06e0e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c036000 tx=0x7f1a1c03b040 comp rx=0 tx=0).stop 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a28152400 msgr2=0x7f1a280c4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a28152400 0x7f1a280c4870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800e840 tx=0x7f1a1800ed00 comp rx=0 tx=0).stop 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 shutdown_connections 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2815bc80 0x7f1a280c19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a28152400 0x7f1a280c4870 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a0c06bc20 0x7f1a0c06e0e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 --2- 172.21.15.6:0/2265941113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a2806f8c0 0x7f1a280c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 >> 172.21.15.6:0/2265941113 conn(0x7f1a280cb930 msgr2=0x7f1a2814e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 shutdown_connections 2024-03-22T12:56:45.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:45.283+0000 7f1a30800640 1 -- 172.21.15.6:0/2265941113 wait complete. 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-22T12:56:45.304 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-22T12:56:45.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:45.305 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-22T12:56:45.353 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-22T12:56:45.353 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-22T12:56:45.353 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-22T12:56:45.354 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-22T12:56:45.355 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42176sec preferred_lft 42176sec 2024-03-22T12:56:45.355 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-22T12:56:45.355 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-22T12:56:45.355 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T12:56:45.355 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-22T12:56:45.355 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-22T12:56:45.355 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-22T12:56:45.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:45.356 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-22T12:56:45.356 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-22T12:56:45.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-03-22T12:56:45.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-03-22T12:56:45.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-03-22T12:56:45.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-22T12:56:45.356 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-22T12:56:45.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:45.781 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-22T12:56:45.854 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-22T12:56:45.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:45.857 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-22T12:56:45.904 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-22T12:56:45.905 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-22T12:56:45.905 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-22T12:56:45.905 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-22T12:56:45.905 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-22T12:56:45.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:45.905 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-22T12:56:45.905 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-22T12:56:45.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-22T12:56:45.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-22T12:56:45.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-22T12:56:45.906 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-22T12:56:46.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:46.625 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-22T12:56:46.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:46.666 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-22T12:56:46.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:46.751 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-22T12:56:46.751 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-22T12:56:46.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T12:56:46.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-22T12:56:46.751 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-22T12:56:46.924 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-22T12:56:46.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-22T12:56:46.924 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-22T12:56:47.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:47.167 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-22T12:56:47.167 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-22T12:56:47.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-22T12:56:47.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-03-22T12:56:47.167 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-22T12:56:47.319 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-22T12:56:47.319 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T12:56:47.319 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:47.357 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-22T12:56:47.358 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T12:56:47.358 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:47.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-22T12:56:47.567 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T12:56:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-22T12:56:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-22T12:56:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-22T12:56:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-22T12:56:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-22T12:56:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T12:56:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T12:56:47.609 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T12:56:47.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:47.610 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-22T12:56:47.696 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-22T12:56:47.697 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-22T12:56:47.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-22T12:56:47.825 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-22T12:56:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-22T12:56:47.869 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-22T12:56:47.870 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-22T12:56:47.870 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-22T12:56:47.870 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-22T12:56:47.870 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-22T12:56:47.870 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T12:56:47.870 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T12:56:47.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:47.871 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-22T12:56:48.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:2024-03-22T12:56:48.028+0000 7feffb40f480 -1 init, newargv = 0x559b8b998040 newargc=15 2024-03-22T12:56:48.028 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[42105]: starting ceph client 2024-03-22T12:56:48.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[42105]: starting fuse 2024-03-22T12:56:49.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-03-22T12:56:49.137 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-03-22T12:56:49.184 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-03-22T12:56:49.185 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-03-22T12:56:49.186 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T12:56:49.186 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T12:56:49.186 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:49.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:49.187 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-03-22T12:56:49.232 INFO:teuthology.orchestra.run.smithi006.stdout:42 2024-03-22T12:56:49.232 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-22T12:56:49.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-22T12:56:49.233 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.admin")) 2024-03-22T12:56:49.234 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-22T12:56:49.452 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.admin.42105.asok 2024-03-22T12:56:49.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T12:56:49.471 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.42105.asok 2024-03-22T12:56:49.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:49.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.42105.asok status 2024-03-22T12:56:49.700 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T12:56:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "admin", 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "42105", 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4372, 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-03-22T12:56:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4372 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3049217357 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:0", 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3049217357 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-22T12:56:49.702 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.4372 172.21.15.6:0/3049217357", 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "172.21.15.6:0/3049217357", 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 7, 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 28, 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-03-22T12:56:49.703 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T12:56:49.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:49.716 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-03-22T12:56:49.716 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:49.776 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-03-22T12:56:49.776 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:49.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:49.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:49.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:49.946 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-22T12:56:50.462 INFO:teuthology.orchestra.run.smithi006.stdout:# file: . 2024-03-22T12:56:50.462 INFO:teuthology.orchestra.run.smithi006.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-22T12:56:50.462 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:50.464 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:56:50.464 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T12:56:50.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.726+0000 7fd5f8800640 1 Processor -- start 2024-03-22T12:56:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.726+0000 7fd5f8800640 1 -- start start 2024-03-22T12:56:50.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.728+0000 7fd5f8800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5f006fb70 0x7fd5f0165bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:50.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.728+0000 7fd5f8800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f015b0b0 con 0x7fd5f01545c0 2024-03-22T12:56:50.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.728+0000 7fd5f8800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f006c3e0 con 0x7fd5f0070670 2024-03-22T12:56:50.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.728+0000 7fd5f8800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f01549a0 con 0x7fd5f006fb70 2024-03-22T12:56:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.729+0000 7fd5f5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd5f01545c0 0x7fd5f00c9530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41568/0 (socket says 172.21.15.6:41568) 2024-03-22T12:56:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.729+0000 7fd5f5e00640 1 -- 172.21.15.6:0/2074978843 learned_addr learned my addr 172.21.15.6:0/2074978843 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.730+0000 7fd5f6800640 1 --2- 172.21.15.6:0/2074978843 >> v2:172.21.15.6:3300/0 conn(0x7fd5f006fb70 0x7fd5f0165bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.735+0000 7fd5f6800640 1 -- 172.21.15.6:0/2074978843 >> v1:172.21.15.92:6789/0 conn(0x7fd5f0070670 legacy=0x7fd5f006f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.735+0000 7fd5f6800640 1 -- 172.21.15.6:0/2074978843 >> v1:172.21.15.6:6789/0 conn(0x7fd5f01545c0 legacy=0x7fd5f00c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:50.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.735+0000 7fd5f6800640 1 -- 172.21.15.6:0/2074978843 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f006c6c0 con 0x7fd5f006fb70 2024-03-22T12:56:50.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.761+0000 7fd5f6800640 1 --2- 172.21.15.6:0/2074978843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f006fb70 0x7fd5f0165bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009d90 tx=0x7fd5ec02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bcae01cb991fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:50.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.780+0000 7fd5f4a00640 1 -- 172.21.15.6:0/2074978843 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec039070 con 0x7fd5f006fb70 2024-03-22T12:56:50.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.781+0000 7fd5f4a00640 1 -- 172.21.15.6:0/2074978843 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ec02cbd0 con 0x7fd5f006fb70 2024-03-22T12:56:50.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.781+0000 7fd5f4a00640 1 -- 172.21.15.6:0/2074978843 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec034440 con 0x7fd5f006fb70 2024-03-22T12:56:50.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.782+0000 7fd5f8800640 1 -- 172.21.15.6:0/2074978843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f006fb70 msgr2=0x7fd5f0165bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:50.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.782+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2074978843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f006fb70 0x7fd5f0165bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009d90 tx=0x7fd5ec02be20 comp rx=0 tx=0).stop 2024-03-22T12:56:50.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.782+0000 7fd5f8800640 1 -- 172.21.15.6:0/2074978843 shutdown_connections 2024-03-22T12:56:50.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.782+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2074978843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f006fb70 0x7fd5f0165bf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:50.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.782+0000 7fd5f8800640 1 -- 172.21.15.6:0/2074978843 >> 172.21.15.6:0/2074978843 conn(0x7fd5f0067470 msgr2=0x7fd5f0071430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:50.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.782+0000 7fd5f8800640 1 -- 172.21.15.6:0/2074978843 shutdown_connections 2024-03-22T12:56:50.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.782+0000 7fd5f8800640 1 -- 172.21.15.6:0/2074978843 wait complete. 2024-03-22T12:56:50.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.785+0000 7fd5f8800640 1 Processor -- start 2024-03-22T12:56:50.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.786+0000 7fd5f8800640 1 -- start start 2024-03-22T12:56:50.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.787+0000 7fd5f8800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5f006fb70 0x7fd5f00c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:50.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.787+0000 7fd5f8800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f0070670 0x7fd5f01f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:50.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.787+0000 7fd5f8800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5f01545c0 0x7fd5f00c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:50.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.787+0000 7fd5f8800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f00c8860 con 0x7fd5f0070670 2024-03-22T12:56:50.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.787+0000 7fd5f8800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f006c190 con 0x7fd5f01545c0 2024-03-22T12:56:50.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.787+0000 7fd5f8800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f006cc80 con 0x7fd5f006fb70 2024-03-22T12:56:50.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.788+0000 7fd5f5e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5f006fb70 0x7fd5f00c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:50.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.788+0000 7fd5f5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f0070670 0x7fd5f01f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:50.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.788+0000 7fd5f6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5f01545c0 0x7fd5f00c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:50.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.788+0000 7fd5f5e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5f006fb70 0x7fd5f00c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47506/0 (socket says 172.21.15.6:47506) 2024-03-22T12:56:50.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.788+0000 7fd5f6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5f01545c0 0x7fd5f00c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51462/0 (socket says 172.21.15.6:51462) 2024-03-22T12:56:50.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.788+0000 7fd5f5e00640 1 -- 172.21.15.6:0/2031683396 learned_addr learned my addr 172.21.15.6:0/2031683396 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:50.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.792+0000 7fd5f5e00640 1 -- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5f01545c0 msgr2=0x7fd5f00c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:50.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.792+0000 7fd5f5e00640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5f01545c0 0x7fd5f00c6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:50.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.792+0000 7fd5f5e00640 1 -- 172.21.15.6:0/2031683396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f0070670 msgr2=0x7fd5f01f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:50.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.792+0000 7fd5f5e00640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f0070670 0x7fd5f01f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:50.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.792+0000 7fd5f5e00640 1 -- 172.21.15.6:0/2031683396 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f00c59f0 con 0x7fd5f006fb70 2024-03-22T12:56:50.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.794+0000 7fd5f5e00640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5f006fb70 0x7fd5f00c53b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c400 tx=0x7fd5e400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.798+0000 7fd5ea000640 1 -- 172.21.15.6:0/2031683396 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4017070 con 0x7fd5f006fb70 2024-03-22T12:56:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.798+0000 7fd5ea000640 1 -- 172.21.15.6:0/2031683396 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e4005ce0 con 0x7fd5f006fb70 2024-03-22T12:56:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.798+0000 7fd5ea000640 1 -- 172.21.15.6:0/2031683396 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40072f0 con 0x7fd5f006fb70 2024-03-22T12:56:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.798+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f00c5cd0 con 0x7fd5f006fb70 2024-03-22T12:56:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.798+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f01f3ac0 con 0x7fd5f006fb70 2024-03-22T12:56:50.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:50.800+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f00756c0 con 0x7fd5f006fb70 2024-03-22T12:56:51.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.054+0000 7fd5ea000640 1 -- 172.21.15.6:0/2031683396 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd5e401a070 con 0x7fd5f006fb70 2024-03-22T12:56:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.056+0000 7fd5ea000640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd5d406bc50 0x7fd5d406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.056+0000 7fd5ea000640 1 -- 172.21.15.6:0/2031683396 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd5e408a420 con 0x7fd5f006fb70 2024-03-22T12:56:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.056+0000 7fd5f5400640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd5d406bc50 0x7fd5d406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.057+0000 7fd5f5400640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd5d406bc50 0x7fd5d406e110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd5f01eadb0 tx=0x7fd5e0007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:51.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.066+0000 7fd5ea000640 1 -- 172.21.15.6:0/2031683396 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd5e4021020 con 0x7fd5f006fb70 2024-03-22T12:56:51.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.297+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5f014ae50 con 0x7fd5f006fb70 2024-03-22T12:56:51.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.347+0000 7fd5ea000640 1 -- 172.21.15.6:0/2031683396 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18418 (secure 0 0 0) 0x7fd5e40586f0 con 0x7fd5f006fb70 2024-03-22T12:56:51.347 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:51.347 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:37.855636+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd5d406bc50 msgr2=0x7fd5d406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd5d406bc50 0x7fd5d406e110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd5f01eadb0 tx=0x7fd5e0007e30 comp rx=0 tx=0).stop 2024-03-22T12:56:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5f006fb70 msgr2=0x7fd5f00c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5f006fb70 0x7fd5f00c53b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c400 tx=0x7fd5e400c8c0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 shutdown_connections 2024-03-22T12:56:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd5d406bc50 0x7fd5d406e110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5f01545c0 0x7fd5f00c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5f0070670 0x7fd5f01f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 --2- 172.21.15.6:0/2031683396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd5f006fb70 0x7fd5f00c53b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 >> 172.21.15.6:0/2031683396 conn(0x7fd5f0067470 msgr2=0x7fd5f0158d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:51.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 shutdown_connections 2024-03-22T12:56:51.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.350+0000 7fd5f8800640 1 -- 172.21.15.6:0/2031683396 wait complete. 2024-03-22T12:56:51.369 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T12:56:14.196164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T12:56:26.857509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T12:56:28.394044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T12:56:34.623081+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T12:56:51.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T12:56:51.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.571+0000 7f4c00e00640 1 Processor -- start 2024-03-22T12:56:51.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.571+0000 7f4c00e00640 1 -- start start 2024-03-22T12:56:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.572+0000 7f4c00e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4bfc154500 0x7f4bfc15e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.572+0000 7f4c00e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4bfc0c6c60 0x7f4bfc15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.572+0000 7f4c00e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c7600 0x7f4bfc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.572+0000 7f4c00e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc15a760 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.572+0000 7f4c00e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc15b130 con 0x7f4bfc154500 2024-03-22T12:56:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.572+0000 7f4c00e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc06b270 con 0x7f4bfc0c7600 2024-03-22T12:56:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.573+0000 7f4bfbe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c7600 0x7f4bfc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.573+0000 7f4bfb400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4bfc154500 0x7f4bfc15e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.573+0000 7f4bfaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4bfc0c6c60 0x7f4bfc15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.573+0000 7f4bfbe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c7600 0x7f4bfc0cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47514/0 (socket says 172.21.15.6:47514) 2024-03-22T12:56:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.574+0000 7f4bfbe00640 1 -- 172.21.15.6:0/2192539764 learned_addr learned my addr 172.21.15.6:0/2192539764 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:51.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.576+0000 7f4bfbe00640 1 -- 172.21.15.6:0/2192539764 >> v2:172.21.15.92:3300/0 conn(0x7f4bfc154500 msgr2=0x7f4bfc15e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:51.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.576+0000 7f4bfbe00640 1 --2- 172.21.15.6:0/2192539764 >> v2:172.21.15.92:3300/0 conn(0x7f4bfc154500 0x7f4bfc15e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.576+0000 7f4bfbe00640 1 -- 172.21.15.6:0/2192539764 >> v2:172.21.15.6:3300/0 conn(0x7f4bfc0c6c60 msgr2=0x7f4bfc15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:51.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.576+0000 7f4bfbe00640 1 --2- 172.21.15.6:0/2192539764 >> v2:172.21.15.6:3300/0 conn(0x7f4bfc0c6c60 0x7f4bfc15abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.576+0000 7f4bfbe00640 1 -- 172.21.15.6:0/2192539764 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc1548e0 con 0x7f4bfc0c7600 2024-03-22T12:56:51.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.576+0000 7f4bfb400640 1 --2- 172.21.15.6:0/2192539764 >> v2:172.21.15.92:3300/0 conn(0x7f4bfc154500 0x7f4bfc15e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:51.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.578+0000 7f4bfbe00640 1 --2- 172.21.15.6:0/2192539764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c7600 0x7f4bfc0cc1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc06e610 tx=0x7f4bf402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d3a9d63917c1af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.582+0000 7f4bfa000640 1 -- 172.21.15.6:0/2192539764 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf402cb00 con 0x7f4bfc0c7600 2024-03-22T12:56:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.582+0000 7f4bfa000640 1 -- 172.21.15.6:0/2192539764 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf402cca0 con 0x7f4bfc0c7600 2024-03-22T12:56:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.582+0000 7f4bfa000640 1 -- 172.21.15.6:0/2192539764 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4034610 con 0x7f4bfc0c7600 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 -- 172.21.15.6:0/2192539764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c7600 msgr2=0x7f4bfc0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 --2- 172.21.15.6:0/2192539764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c7600 0x7f4bfc0cc1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc06e610 tx=0x7f4bf402bf60 comp rx=0 tx=0).stop 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 -- 172.21.15.6:0/2192539764 shutdown_connections 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 --2- 172.21.15.6:0/2192539764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c7600 0x7f4bfc0cc1f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 --2- 172.21.15.6:0/2192539764 >> v2:172.21.15.6:3300/0 conn(0x7f4bfc0c6c60 0x7f4bfc15abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 --2- 172.21.15.6:0/2192539764 >> v2:172.21.15.92:3300/0 conn(0x7f4bfc154500 0x7f4bfc15e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 -- 172.21.15.6:0/2192539764 >> 172.21.15.6:0/2192539764 conn(0x7f4bfc06db00 msgr2=0x7f4bfc06dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 -- 172.21.15.6:0/2192539764 shutdown_connections 2024-03-22T12:56:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.583+0000 7f4c00e00640 1 -- 172.21.15.6:0/2192539764 wait complete. 2024-03-22T12:56:51.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.585+0000 7f4c00e00640 1 Processor -- start 2024-03-22T12:56:51.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.585+0000 7f4c00e00640 1 -- start start 2024-03-22T12:56:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4c00e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c6c60 0x7f4bfc14b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4c00e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bfc0c7600 0x7f4bfc14b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4c00e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bfc154500 0x7f4bfc1f3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4c00e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc06bef0 con 0x7f4bfc154500 2024-03-22T12:56:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4c00e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc15a3f0 con 0x7f4bfc0c7600 2024-03-22T12:56:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4c00e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc06c070 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4bfbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bfc154500 0x7f4bfc1f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.586+0000 7f4bfb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c6c60 0x7f4bfc14b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.587+0000 7f4bfaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bfc0c7600 0x7f4bfc14b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.587+0000 7f4bfbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bfc154500 0x7f4bfc1f3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37098/0 (socket says 172.21.15.6:37098) 2024-03-22T12:56:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.587+0000 7f4bfbe00640 1 -- 172.21.15.6:0/577694973 learned_addr learned my addr 172.21.15.6:0/577694973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.590+0000 7f4bfb400640 1 -- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bfc0c7600 msgr2=0x7f4bfc14b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.590+0000 7f4bfb400640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bfc0c7600 0x7f4bfc14b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.590+0000 7f4bfb400640 1 -- 172.21.15.6:0/577694973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bfc154500 msgr2=0x7f4bfc1f3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.590+0000 7f4bfb400640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bfc154500 0x7f4bfc1f3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.590+0000 7f4bfb400640 1 -- 172.21.15.6:0/577694973 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc159670 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.592+0000 7f4bfb400640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c6c60 0x7f4bfc14b1c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00e8a0 tx=0x7f4bec00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.596+0000 7f4bf3400640 1 -- 172.21.15.6:0/577694973 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec017070 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.596+0000 7f4bf3400640 1 -- 172.21.15.6:0/577694973 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bec00c940 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.596+0000 7f4bf3400640 1 -- 172.21.15.6:0/577694973 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec00cd30 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.596+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bfc158f00 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.596+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bfc15b890 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.597+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bfc075650 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.854+0000 7f4bf3400640 1 -- 172.21.15.6:0/577694973 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4bec01a070 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.856+0000 7f4bf3400640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4bdc06bc50 0x7f4bdc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.856+0000 7f4bf3400640 1 -- 172.21.15.6:0/577694973 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4bec08b2c0 con 0x7f4bfc0c6c60 2024-03-22T12:56:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.856+0000 7f4bfaa00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4bdc06bc50 0x7f4bdc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.858+0000 7f4bfaa00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4bdc06bc50 0x7f4bdc06e110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc1f4550 tx=0x7f4be8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:51.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:51.864+0000 7f4bf3400640 1 -- 172.21.15.6:0/577694973 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4bec059400 con 0x7f4bfc0c6c60 2024-03-22T12:56:52.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.068+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4bfc14ade0 con 0x7f4bfc0c6c60 2024-03-22T12:56:52.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.074+0000 7f4bf3400640 1 -- 172.21.15.6:0/577694973 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4bec05d4a0 con 0x7f4bfc0c6c60 2024-03-22T12:56:52.074 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-22T12:56:52.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4bdc06bc50 msgr2=0x7f4bdc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4bdc06bc50 0x7f4bdc06e110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc1f4550 tx=0x7f4be8009190 comp rx=0 tx=0).stop 2024-03-22T12:56:52.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c6c60 msgr2=0x7f4bfc14b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c6c60 0x7f4bfc14b1c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4bec00e8a0 tx=0x7f4bec00ed60 comp rx=0 tx=0).stop 2024-03-22T12:56:52.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 shutdown_connections 2024-03-22T12:56:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bfc154500 0x7f4bfc1f3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4bdc06bc50 0x7f4bdc06e110 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4bfc0c7600 0x7f4bfc14b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 --2- 172.21.15.6:0/577694973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4bfc0c6c60 0x7f4bfc14b1c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.076+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 >> 172.21.15.6:0/577694973 conn(0x7f4bfc06db00 msgr2=0x7f4bfc0c48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.077+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 shutdown_connections 2024-03-22T12:56:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.077+0000 7f4c00e00640 1 -- 172.21.15.6:0/577694973 wait complete. 2024-03-22T12:56:52.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T12:56:52.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.294+0000 7fe153000640 1 Processor -- start 2024-03-22T12:56:52.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.294+0000 7fe153000640 1 -- start start 2024-03-22T12:56:52.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.296+0000 7fe153000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe14c0cc7e0 0x7fe14c0ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:52.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.296+0000 7fe153000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe14c0cef10 0x7fe14c164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:52.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.296+0000 7fe153000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c0cf360 con 0x7fe14c0cc400 2024-03-22T12:56:52.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.296+0000 7fe153000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14c06dd30 con 0x7fe14c0cc7e0 2024-03-22T12:56:52.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.296+0000 7fe153000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14c15a670 con 0x7fe14c0cef10 2024-03-22T12:56:52.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.297+0000 7fe152600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe14c0cef10 0x7fe14c164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:52.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.298+0000 7fe151c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe14c0cc400 0x7fe14c0cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:38418/0 (socket says 172.21.15.6:38418) 2024-03-22T12:56:52.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.298+0000 7fe151c00640 1 -- 172.21.15.6:0/2996152985 learned_addr learned my addr 172.21.15.6:0/2996152985 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.302+0000 7fe152600640 1 -- 172.21.15.6:0/2996152985 >> v2:172.21.15.6:3300/0 conn(0x7fe14c0cc7e0 msgr2=0x7fe14c0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.302+0000 7fe152600640 1 --2- 172.21.15.6:0/2996152985 >> v2:172.21.15.6:3300/0 conn(0x7fe14c0cc7e0 0x7fe14c0ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.302+0000 7fe152600640 1 -- 172.21.15.6:0/2996152985 >> v1:172.21.15.92:6789/0 conn(0x7fe14c0cc400 legacy=0x7fe14c0cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.302+0000 7fe152600640 1 -- 172.21.15.6:0/2996152985 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c15a930 con 0x7fe14c0cef10 2024-03-22T12:56:52.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.306+0000 7fe152600640 1 --2- 172.21.15.6:0/2996152985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cef10 0x7fe14c164c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0c33c0 tx=0x7fe14402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c329e20758c843b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.309+0000 7fe14be00640 1 -- 172.21.15.6:0/2996152985 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144039070 con 0x7fe14c0cef10 2024-03-22T12:56:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.309+0000 7fe14be00640 1 -- 172.21.15.6:0/2996152985 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14402cbf0 con 0x7fe14c0cef10 2024-03-22T12:56:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.309+0000 7fe14be00640 1 -- 172.21.15.6:0/2996152985 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144034440 con 0x7fe14c0cef10 2024-03-22T12:56:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.310+0000 7fe153000640 1 -- 172.21.15.6:0/2996152985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cef10 msgr2=0x7fe14c164c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.310+0000 7fe153000640 1 --2- 172.21.15.6:0/2996152985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cef10 0x7fe14c164c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0c33c0 tx=0x7fe14402be40 comp rx=0 tx=0).stop 2024-03-22T12:56:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.311+0000 7fe153000640 1 -- 172.21.15.6:0/2996152985 shutdown_connections 2024-03-22T12:56:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.311+0000 7fe153000640 1 --2- 172.21.15.6:0/2996152985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cef10 0x7fe14c164c00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.311+0000 7fe153000640 1 --2- 172.21.15.6:0/2996152985 >> v2:172.21.15.6:3300/0 conn(0x7fe14c0cc7e0 0x7fe14c0ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.311+0000 7fe153000640 1 -- 172.21.15.6:0/2996152985 >> 172.21.15.6:0/2996152985 conn(0x7fe14c0cf9f0 msgr2=0x7fe14c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.311+0000 7fe153000640 1 -- 172.21.15.6:0/2996152985 shutdown_connections 2024-03-22T12:56:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.311+0000 7fe153000640 1 -- 172.21.15.6:0/2996152985 wait complete. 2024-03-22T12:56:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.314+0000 7fe153000640 1 Processor -- start 2024-03-22T12:56:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.315+0000 7fe153000640 1 -- start start 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.317+0000 7fe153000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cc400 0x7fe14c1f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.317+0000 7fe153000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe14c0cc7e0 0x7fe14c0b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.317+0000 7fe153000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe14c0cef10 0x7fe14c0b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.317+0000 7fe153000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c15b1d0 con 0x7fe14c0cc7e0 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.317+0000 7fe153000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c15aef0 con 0x7fe14c0cc400 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.317+0000 7fe153000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0ce1d0 con 0x7fe14c0cef10 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.318+0000 7fe151c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cc400 0x7fe14c1f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.318+0000 7fe152600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe14c0cef10 0x7fe14c0b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.318+0000 7fe151c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cc400 0x7fe14c1f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51498/0 (socket says 172.21.15.6:51498) 2024-03-22T12:56:52.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.318+0000 7fe151c00640 1 -- 172.21.15.6:0/616968405 learned_addr learned my addr 172.21.15.6:0/616968405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:52.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.318+0000 7fe151200640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe14c0cc7e0 0x7fe14c0b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.322+0000 7fe152600640 1 -- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cc400 msgr2=0x7fe14c1f0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.322+0000 7fe152600640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cc400 0x7fe14c1f0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.322+0000 7fe152600640 1 -- 172.21.15.6:0/616968405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe14c0cc7e0 msgr2=0x7fe14c0b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.322+0000 7fe152600640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe14c0cc7e0 0x7fe14c0b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.322+0000 7fe152600640 1 -- 172.21.15.6:0/616968405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c0b4fb0 con 0x7fe14c0cef10 2024-03-22T12:56:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.322+0000 7fe151c00640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cc400 0x7fe14c1f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.324+0000 7fe152600640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe14c0cef10 0x7fe14c0b2610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe144033000 tx=0x7fe144004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.328+0000 7fe149600640 1 -- 172.21.15.6:0/616968405 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144039050 con 0x7fe14c0cef10 2024-03-22T12:56:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.328+0000 7fe149600640 1 -- 172.21.15.6:0/616968405 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14403a040 con 0x7fe14c0cef10 2024-03-22T12:56:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.328+0000 7fe149600640 1 -- 172.21.15.6:0/616968405 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14403e610 con 0x7fe14c0cef10 2024-03-22T12:56:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.328+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c0b5270 con 0x7fe14c0cef10 2024-03-22T12:56:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.328+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c15b660 con 0x7fe14c0cef10 2024-03-22T12:56:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.329+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c075650 con 0x7fe14c0cef10 2024-03-22T12:56:52.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.578+0000 7fe149600640 1 -- 172.21.15.6:0/616968405 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe144047020 con 0x7fe14c0cef10 2024-03-22T12:56:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.581+0000 7fe149600640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe12c06bc20 0x7fe12c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:52.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.581+0000 7fe149600640 1 -- 172.21.15.6:0/616968405 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe1440adc10 con 0x7fe14c0cef10 2024-03-22T12:56:52.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.581+0000 7fe151c00640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe12c06bc20 0x7fe12c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:52.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.582+0000 7fe151c00640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe12c06bc20 0x7fe12c06e0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe140005880 tx=0x7fe140009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.590+0000 7fe149600640 1 -- 172.21.15.6:0/616968405 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe14407c4a0 con 0x7fe14c0cef10 2024-03-22T12:56:52.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.814+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe14c0c5c90 con 0x7fe14c0cef10 2024-03-22T12:56:52.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.849+0000 7fe149600640 1 -- 172.21.15.6:0/616968405 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe14407bda0 con 0x7fe14c0cef10 2024-03-22T12:56:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.852+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe12c06bc20 msgr2=0x7fe12c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.852+0000 7fe153000640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe12c06bc20 0x7fe12c06e0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe140005880 tx=0x7fe140009040 comp rx=0 tx=0).stop 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe14c0cef10 msgr2=0x7fe14c0b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe14c0cef10 0x7fe14c0b2610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe144033000 tx=0x7fe144004a40 comp rx=0 tx=0).stop 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 shutdown_connections 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe14c0cef10 0x7fe14c0b2610 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe14c0cc7e0 0x7fe14c0b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe12c06bc20 0x7fe12c06e0e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 --2- 172.21.15.6:0/616968405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe14c0cc400 0x7fe14c1f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 >> 172.21.15.6:0/616968405 conn(0x7fe14c0cf9f0 msgr2=0x7fe14c154070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 shutdown_connections 2024-03-22T12:56:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:52.853+0000 7fe153000640 1 -- 172.21.15.6:0/616968405 wait complete. 2024-03-22T12:56:52.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T12:56:53.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.068+0000 7f1308000640 1 Processor -- start 2024-03-22T12:56:53.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.069+0000 7f1308000640 1 -- start start 2024-03-22T12:56:53.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.070+0000 7f1308000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13001522c0 0x7f130006fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.070+0000 7f1308000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13000c84b0 0x7f13000c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.070+0000 7f1308000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000c7fc0 con 0x7f1300151ee0 2024-03-22T12:56:53.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.070+0000 7f1308000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13000b30b0 con 0x7f13000c84b0 2024-03-22T12:56:53.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.070+0000 7f1308000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130006fbe0 con 0x7f13001522c0 2024-03-22T12:56:53.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.071+0000 7f1306000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13000c84b0 0x7f13000c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.071+0000 7f1304c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13001522c0 0x7f130006fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.071+0000 7f1304c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13001522c0 0x7f130006fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47522/0 (socket says 172.21.15.6:47522) 2024-03-22T12:56:53.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.071+0000 7f1304c00640 1 -- 172.21.15.6:0/1039418607 learned_addr learned my addr 172.21.15.6:0/1039418607 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:53.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.074+0000 7f1304c00640 1 -- 172.21.15.6:0/1039418607 >> v2:172.21.15.6:3300/0 conn(0x7f13000c84b0 msgr2=0x7f13000c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:53.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.074+0000 7f1304c00640 1 --2- 172.21.15.6:0/1039418607 >> v2:172.21.15.6:3300/0 conn(0x7f13000c84b0 0x7f13000c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.074+0000 7f1304c00640 1 -- 172.21.15.6:0/1039418607 >> v1:172.21.15.6:6789/0 conn(0x7f1300151ee0 legacy=0x7f13000c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.074+0000 7f1304c00640 1 -- 172.21.15.6:0/1039418607 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300158e30 con 0x7f13001522c0 2024-03-22T12:56:53.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.076+0000 7f1304c00640 1 --2- 172.21.15.6:0/1039418607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f130006fe40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009870 tx=0x7f12ec02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60024b956d58c2cb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:53.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.080+0000 7f12ffe00640 1 -- 172.21.15.6:0/1039418607 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec02f850 con 0x7f13001522c0 2024-03-22T12:56:53.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.080+0000 7f12ffe00640 1 -- 172.21.15.6:0/1039418607 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec02f9f0 con 0x7f13001522c0 2024-03-22T12:56:53.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.080+0000 7f12ffe00640 1 -- 172.21.15.6:0/1039418607 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec02fdc0 con 0x7f13001522c0 2024-03-22T12:56:53.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 -- 172.21.15.6:0/1039418607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 msgr2=0x7f130006fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 --2- 172.21.15.6:0/1039418607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f130006fe40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009870 tx=0x7f12ec02ba30 comp rx=0 tx=0).stop 2024-03-22T12:56:53.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 -- 172.21.15.6:0/1039418607 shutdown_connections 2024-03-22T12:56:53.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 --2- 172.21.15.6:0/1039418607 >> v2:172.21.15.6:3300/0 conn(0x7f13000c84b0 0x7f13000c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 --2- 172.21.15.6:0/1039418607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f130006fe40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 -- 172.21.15.6:0/1039418607 >> 172.21.15.6:0/1039418607 conn(0x7f1300071480 msgr2=0x7f1300071890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:53.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 -- 172.21.15.6:0/1039418607 shutdown_connections 2024-03-22T12:56:53.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.081+0000 7f1308000640 1 -- 172.21.15.6:0/1039418607 wait complete. 2024-03-22T12:56:53.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.082+0000 7f1308000640 1 Processor -- start 2024-03-22T12:56:53.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.083+0000 7f1308000640 1 -- start start 2024-03-22T12:56:53.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.083+0000 7f1308000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13000c84b0 0x7f130006c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1308000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1300151ee0 0x7f1300156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1308000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f1300156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1308000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130006f3e0 con 0x7f1300151ee0 2024-03-22T12:56:53.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1308000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130006f100 con 0x7f13001522c0 2024-03-22T12:56:53.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1308000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130006f6c0 con 0x7f13000c84b0 2024-03-22T12:56:53.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1304c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1300151ee0 0x7f1300156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1306000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f1300156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1305600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13000c84b0 0x7f130006c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1304c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1300151ee0 0x7f1300156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50034/0 (socket says 172.21.15.6:50034) 2024-03-22T12:56:53.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.084+0000 7f1304c00640 1 -- 172.21.15.6:0/1368590736 learned_addr learned my addr 172.21.15.6:0/1368590736 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:53.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.088+0000 7f1306000640 1 -- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13000c84b0 msgr2=0x7f130006c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.088+0000 7f1306000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13000c84b0 0x7f130006c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.088+0000 7f1306000640 1 -- 172.21.15.6:0/1368590736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1300151ee0 msgr2=0x7f1300156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.088+0000 7f1306000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1300151ee0 0x7f1300156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.088+0000 7f1306000640 1 -- 172.21.15.6:0/1368590736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300157dc0 con 0x7f13001522c0 2024-03-22T12:56:53.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.090+0000 7f1306000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f1300156690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e7f0 tx=0x7f12f800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:53.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.094+0000 7f12fd600640 1 -- 172.21.15.6:0/1368590736 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8017070 con 0x7f13001522c0 2024-03-22T12:56:53.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.094+0000 7f12fd600640 1 -- 172.21.15.6:0/1368590736 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f800c940 con 0x7f13001522c0 2024-03-22T12:56:53.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.094+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13001f0c90 con 0x7f13001522c0 2024-03-22T12:56:53.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.094+0000 7f12fd600640 1 -- 172.21.15.6:0/1368590736 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f800cd10 con 0x7f13001522c0 2024-03-22T12:56:53.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.094+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13001e27b0 con 0x7f13001522c0 2024-03-22T12:56:53.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.095+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13001e2da0 con 0x7f13001522c0 2024-03-22T12:56:53.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.114+0000 7f12fd600640 1 -- 172.21.15.6:0/1368590736 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f12f801a070 con 0x7f13001522c0 2024-03-22T12:56:53.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.116+0000 7f12fd600640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12e006bc00 0x7f12e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.116+0000 7f12fd600640 1 -- 172.21.15.6:0/1368590736 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f12f808ab60 con 0x7f13001522c0 2024-03-22T12:56:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.116+0000 7f1305600640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12e006bc00 0x7f12e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.117+0000 7f1305600640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12e006bc00 0x7f12e006e0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f12f4001f10 tx=0x7f12f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:53.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.356+0000 7f12fd600640 1 -- 172.21.15.6:0/1368590736 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f12f8058da0 con 0x7f13001522c0 2024-03-22T12:56:53.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.573+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13000757d0 con 0x7f13001522c0 2024-03-22T12:56:53.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.578+0000 7f12fd600640 1 -- 172.21.15.6:0/1368590736 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f12f805ce40 con 0x7f13001522c0 2024-03-22T12:56:53.578 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-22T12:56:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12e006bc00 msgr2=0x7f12e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12e006bc00 0x7f12e006e0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f12f4001f10 tx=0x7f12f4009040 comp rx=0 tx=0).stop 2024-03-22T12:56:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 msgr2=0x7f1300156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f1300156690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e7f0 tx=0x7f12f800ecb0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 shutdown_connections 2024-03-22T12:56:53.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12e006bc00 0x7f12e006e0c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13001522c0 0x7f1300156690 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1300151ee0 0x7f1300156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 --2- 172.21.15.6:0/1368590736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13000c84b0 0x7f130006c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 >> 172.21.15.6:0/1368590736 conn(0x7f1300071480 msgr2=0x7f13000c9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:53.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 shutdown_connections 2024-03-22T12:56:53.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.582+0000 7f1308000640 1 -- 172.21.15.6:0/1368590736 wait complete. 2024-03-22T12:56:53.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T12:56:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.806+0000 7fadfde00640 1 Processor -- start 2024-03-22T12:56:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.806+0000 7fadfde00640 1 -- start start 2024-03-22T12:56:53.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.807+0000 7fadfde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150980 0x7fadf8071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.807+0000 7fadfde00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fadf8150d60 0x7fadf806b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.807+0000 7fadfde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fadf80731f0 0x7fadf806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.807+0000 7fadfde00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf80c65d0 con 0x7fadf80731f0 2024-03-22T12:56:53.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.807+0000 7fadfde00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf80c94b0 con 0x7fadf8150d60 2024-03-22T12:56:53.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.807+0000 7fadfde00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8156820 con 0x7fadf8150980 2024-03-22T12:56:53.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.863+0000 7fadf7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fadf80731f0 0x7fadf806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.864+0000 7fadf7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fadf80731f0 0x7fadf806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50042/0 (socket says 172.21.15.6:50042) 2024-03-22T12:56:53.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.864+0000 7fadf7400640 1 -- 172.21.15.6:0/3590836284 learned_addr learned my addr 172.21.15.6:0/3590836284 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:53.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.869+0000 7fadf7400640 1 -- 172.21.15.6:0/3590836284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150980 msgr2=0x7fadf8071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:53.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.869+0000 7fadf7400640 1 --2- 172.21.15.6:0/3590836284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150980 0x7fadf8071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.869+0000 7fadf7400640 1 -- 172.21.15.6:0/3590836284 >> v2:172.21.15.92:3300/0 conn(0x7fadf8150d60 msgr2=0x7fadf806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:53.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.869+0000 7fadf7400640 1 --2- 172.21.15.6:0/3590836284 >> v2:172.21.15.92:3300/0 conn(0x7fadf8150d60 0x7fadf806b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.869+0000 7fadf7400640 1 -- 172.21.15.6:0/3590836284 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8071c60 con 0x7fadf80731f0 2024-03-22T12:56:53.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.873+0000 7fadf7400640 1 --2- 172.21.15.6:0/3590836284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf80731f0 0x7fadf806b5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fadf806fea0 tx=0x7fadf002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a764c9c2f7dabf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:53.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.882+0000 7fadf5600640 1 -- 172.21.15.6:0/3590836284 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0035020 con 0x7fadf80731f0 2024-03-22T12:56:53.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.882+0000 7fadf5600640 1 -- 172.21.15.6:0/3590836284 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0031880 con 0x7fadf80731f0 2024-03-22T12:56:53.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.882+0000 7fadf5600640 1 -- 172.21.15.6:0/3590836284 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0031c50 con 0x7fadf80731f0 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 -- 172.21.15.6:0/3590836284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf80731f0 msgr2=0x7fadf806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 --2- 172.21.15.6:0/3590836284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf80731f0 0x7fadf806b5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fadf806fea0 tx=0x7fadf002b920 comp rx=0 tx=0).stop 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 -- 172.21.15.6:0/3590836284 shutdown_connections 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 --2- 172.21.15.6:0/3590836284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf80731f0 0x7fadf806b5c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 --2- 172.21.15.6:0/3590836284 >> v2:172.21.15.92:3300/0 conn(0x7fadf8150d60 0x7fadf806b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 --2- 172.21.15.6:0/3590836284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150980 0x7fadf8071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 -- 172.21.15.6:0/3590836284 >> 172.21.15.6:0/3590836284 conn(0x7fadf806e310 msgr2=0x7fadf806f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:53.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.883+0000 7fadfde00640 1 -- 172.21.15.6:0/3590836284 shutdown_connections 2024-03-22T12:56:53.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.884+0000 7fadfde00640 1 -- 172.21.15.6:0/3590836284 wait complete. 2024-03-22T12:56:53.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.886+0000 7fadfde00640 1 Processor -- start 2024-03-22T12:56:53.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.886+0000 7fadfde00640 1 -- start start 2024-03-22T12:56:53.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.887+0000 7fadfde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadf80731f0 0x7fadf80c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.887+0000 7fadfde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf8150980 0x7fadf81f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.887+0000 7fadfde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150d60 0x7fadf81f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.887+0000 7fadfde00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8157080 con 0x7fadf8150980 2024-03-22T12:56:53.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.887+0000 7fadfde00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8156da0 con 0x7fadf80731f0 2024-03-22T12:56:53.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.887+0000 7fadfde00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8157360 con 0x7fadf8150d60 2024-03-22T12:56:53.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.888+0000 7fadf6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadf80731f0 0x7fadf80c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.889+0000 7fadf6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadf80731f0 0x7fadf80c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47538/0 (socket says 172.21.15.6:47538) 2024-03-22T12:56:53.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.889+0000 7fadf6a00640 1 -- 172.21.15.6:0/2264778939 learned_addr learned my addr 172.21.15.6:0/2264778939 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:53.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.890+0000 7fadf7400640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150d60 0x7fadf81f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.893+0000 7fadf6a00640 1 -- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150d60 msgr2=0x7fadf81f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.893+0000 7fadf6a00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150d60 0x7fadf81f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.893+0000 7fadf6a00640 1 -- 172.21.15.6:0/2264778939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf8150980 msgr2=0x7fadf81f13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:53.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.893+0000 7fadf6a00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf8150980 0x7fadf81f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:53.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.893+0000 7fadf6a00640 1 -- 172.21.15.6:0/2264778939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80c4b60 con 0x7fadf80731f0 2024-03-22T12:56:53.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.895+0000 7fadf6a00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadf80731f0 0x7fadf80c4200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fade800ba50 tx=0x7fade800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.898+0000 7fadef400640 1 -- 172.21.15.6:0/2264778939 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8019070 con 0x7fadf80731f0 2024-03-22T12:56:53.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.899+0000 7fadef400640 1 -- 172.21.15.6:0/2264778939 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade800cc00 con 0x7fadf80731f0 2024-03-22T12:56:53.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.899+0000 7fadef400640 1 -- 172.21.15.6:0/2264778939 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8014440 con 0x7fadf80731f0 2024-03-22T12:56:53.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.899+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf814a490 con 0x7fadf80731f0 2024-03-22T12:56:53.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.899+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf81491b0 con 0x7fadf80731f0 2024-03-22T12:56:53.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.900+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf80757d0 con 0x7fadf80731f0 2024-03-22T12:56:53.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.920+0000 7fadef400640 1 -- 172.21.15.6:0/2264778939 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fade801c070 con 0x7fadf80731f0 2024-03-22T12:56:53.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.921+0000 7fadef400640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadd406bc50 0x7fadd406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:53.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.921+0000 7fadf6000640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadd406bc50 0x7fadd406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:53.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.922+0000 7fadf6000640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadd406bc50 0x7fadd406e110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fadf80ccc00 tx=0x7fade0006bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:53.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:53.922+0000 7fadef400640 1 -- 172.21.15.6:0/2264778939 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fade8057120 con 0x7fadf80731f0 2024-03-22T12:56:54.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.214+0000 7fadef400640 1 -- 172.21.15.6:0/2264778939 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fade8023020 con 0x7fadf80731f0 2024-03-22T12:56:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.414+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fadf814ad30 con 0x7fadf80731f0 2024-03-22T12:56:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.420+0000 7fadef400640 1 -- 172.21.15.6:0/2264778939 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fade8059080 con 0x7fadf80731f0 2024-03-22T12:56:54.420 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadd406bc50 msgr2=0x7fadd406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadd406bc50 0x7fadd406e110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fadf80ccc00 tx=0x7fade0006bf0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadf80731f0 msgr2=0x7fadf80c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadf80731f0 0x7fadf80c4200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fade800ba50 tx=0x7fade800bf10 comp rx=0 tx=0).stop 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 shutdown_connections 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadf8150d60 0x7fadf81f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadf8150980 0x7fadf81f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadd406bc50 0x7fadd406e110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 --2- 172.21.15.6:0/2264778939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadf80731f0 0x7fadf80c4200 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 >> 172.21.15.6:0/2264778939 conn(0x7fadf806e310 msgr2=0x7fadf80744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:54.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 shutdown_connections 2024-03-22T12:56:54.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.423+0000 7fadfde00640 1 -- 172.21.15.6:0/2264778939 wait complete. 2024-03-22T12:56:54.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T12:56:54.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.640+0000 7fe7be400640 1 Processor -- start 2024-03-22T12:56:54.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.640+0000 7fe7be400640 1 -- start start 2024-03-22T12:56:54.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.642+0000 7fe7be400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe7b8152280 0x7fe7b8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:54.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.642+0000 7fe7be400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7b80c0ea0 0x7fe7b8158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:54.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.642+0000 7fe7be400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b80c1450 con 0x7fe7b806b6f0 2024-03-22T12:56:54.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.642+0000 7fe7be400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b8158790 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.642+0000 7fe7be400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b80c1730 con 0x7fe7b8152280 2024-03-22T12:56:54.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.643+0000 7fe7b7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe7b8152280 0x7fe7b8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:54.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.643+0000 7fe7b7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe7b8152280 0x7fe7b8071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47542/0 (socket says 172.21.15.6:47542) 2024-03-22T12:56:54.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.643+0000 7fe7b7400640 1 -- 172.21.15.6:0/1936402332 learned_addr learned my addr 172.21.15.6:0/1936402332 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:54.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.643+0000 7fe7b6a00640 1 --2- 172.21.15.6:0/1936402332 >> v2:172.21.15.6:3300/0 conn(0x7fe7b80c0ea0 0x7fe7b8158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.645+0000 7fe7b7400640 1 -- 172.21.15.6:0/1936402332 >> v2:172.21.15.6:3300/0 conn(0x7fe7b80c0ea0 msgr2=0x7fe7b8158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.645+0000 7fe7b7400640 1 --2- 172.21.15.6:0/1936402332 >> v2:172.21.15.6:3300/0 conn(0x7fe7b80c0ea0 0x7fe7b8158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.645+0000 7fe7b7400640 1 -- 172.21.15.6:0/1936402332 >> v1:172.21.15.6:6789/0 conn(0x7fe7b806b6f0 legacy=0x7fe7b80c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.645+0000 7fe7b7400640 1 -- 172.21.15.6:0/1936402332 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b80c1a10 con 0x7fe7b8152280 2024-03-22T12:56:54.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.648+0000 7fe7b7400640 1 --2- 172.21.15.6:0/1936402332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b8152280 0x7fe7b8071920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80c27a0 tx=0x7fe7a802b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9b798d7c24bd431 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:54.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.651+0000 7fe7b6000640 1 -- 172.21.15.6:0/1936402332 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8035020 con 0x7fe7b8152280 2024-03-22T12:56:54.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.651+0000 7fe7b6000640 1 -- 172.21.15.6:0/1936402332 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a802d040 con 0x7fe7b8152280 2024-03-22T12:56:54.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.651+0000 7fe7b6000640 1 -- 172.21.15.6:0/1936402332 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8031a80 con 0x7fe7b8152280 2024-03-22T12:56:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 -- 172.21.15.6:0/1936402332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b8152280 msgr2=0x7fe7b8071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 --2- 172.21.15.6:0/1936402332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b8152280 0x7fe7b8071920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80c27a0 tx=0x7fe7a802b900 comp rx=0 tx=0).stop 2024-03-22T12:56:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 -- 172.21.15.6:0/1936402332 shutdown_connections 2024-03-22T12:56:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 --2- 172.21.15.6:0/1936402332 >> v2:172.21.15.6:3300/0 conn(0x7fe7b80c0ea0 0x7fe7b8158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 --2- 172.21.15.6:0/1936402332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b8152280 0x7fe7b8071920 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 -- 172.21.15.6:0/1936402332 >> 172.21.15.6:0/1936402332 conn(0x7fe7b806e310 msgr2=0x7fe7b806f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 -- 172.21.15.6:0/1936402332 shutdown_connections 2024-03-22T12:56:54.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.652+0000 7fe7be400640 1 -- 172.21.15.6:0/1936402332 wait complete. 2024-03-22T12:56:54.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.654+0000 7fe7be400640 1 Processor -- start 2024-03-22T12:56:54.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.655+0000 7fe7be400640 1 -- start start 2024-03-22T12:56:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.655+0000 7fe7be400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7b806b6f0 0x7fe7b8147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.655+0000 7fe7be400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b80c0ea0 0x7fe7b815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.655+0000 7fe7be400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7b8152280 0x7fe7b80c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.655+0000 7fe7be400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b8152660 con 0x7fe7b806b6f0 2024-03-22T12:56:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.655+0000 7fe7be400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b81580d0 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.655+0000 7fe7be400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b81527e0 con 0x7fe7b8152280 2024-03-22T12:56:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.656+0000 7fe7b7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7b806b6f0 0x7fe7b8147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.656+0000 7fe7b6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b80c0ea0 0x7fe7b815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:54.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.656+0000 7fe7b7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7b806b6f0 0x7fe7b8147a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50064/0 (socket says 172.21.15.6:50064) 2024-03-22T12:56:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.656+0000 7fe7b7400640 1 -- 172.21.15.6:0/2877242427 learned_addr learned my addr 172.21.15.6:0/2877242427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.656+0000 7fe7b7e00640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7b8152280 0x7fe7b80c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.660+0000 7fe7b6a00640 1 -- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7b8152280 msgr2=0x7fe7b80c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.660+0000 7fe7b6a00640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7b8152280 0x7fe7b80c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.660+0000 7fe7b6a00640 1 -- 172.21.15.6:0/2877242427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7b806b6f0 msgr2=0x7fe7b8147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.660+0000 7fe7b6a00640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7b806b6f0 0x7fe7b8147a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.660+0000 7fe7b6a00640 1 -- 172.21.15.6:0/2877242427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b815c020 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.662+0000 7fe7b6a00640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b80c0ea0 0x7fe7b815a2d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e850 tx=0x7fe7a000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:54.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.666+0000 7fe7af400640 1 -- 172.21.15.6:0/2877242427 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0017070 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.666+0000 7fe7af400640 1 -- 172.21.15.6:0/2877242427 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a000c940 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.666+0000 7fe7af400640 1 -- 172.21.15.6:0/2877242427 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000cd10 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.666+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b815bd80 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.666+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b8159680 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.667+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b80757d0 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.685+0000 7fe7af400640 1 -- 172.21.15.6:0/2877242427 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe7a001a070 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.686+0000 7fe7af400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe79406bc20 0x7fe79406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:54.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.686+0000 7fe7b7400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe79406bc20 0x7fe79406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:54.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.687+0000 7fe7b7400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe79406bc20 0x7fe79406e0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe7a802b940 tx=0x7fe7a803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.688+0000 7fe7af400640 1 -- 172.21.15.6:0/2877242427 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe7a0057610 con 0x7fe7b80c0ea0 2024-03-22T12:56:54.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:54.942+0000 7fe7af400640 1 -- 172.21.15.6:0/2877242427 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe7a0059570 con 0x7fe7b80c0ea0 2024-03-22T12:56:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.165+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe7b8071b00 con 0x7fe7b80c0ea0 2024-03-22T12:56:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.171+0000 7fe7af400640 1 -- 172.21.15.6:0/2877242427 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe7a005d610 con 0x7fe7b80c0ea0 2024-03-22T12:56:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-22T12:56:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe79406bc20 msgr2=0x7fe79406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe79406bc20 0x7fe79406e0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe7a802b940 tx=0x7fe7a803c040 comp rx=0 tx=0).stop 2024-03-22T12:56:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b80c0ea0 msgr2=0x7fe7b815a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b80c0ea0 0x7fe7b815a2d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e850 tx=0x7fe7a000ed10 comp rx=0 tx=0).stop 2024-03-22T12:56:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 shutdown_connections 2024-03-22T12:56:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7b8152280 0x7fe7b80c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe79406bc20 0x7fe79406e0e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7b80c0ea0 0x7fe7b815a2d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 --2- 172.21.15.6:0/2877242427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7b806b6f0 0x7fe7b8147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 >> 172.21.15.6:0/2877242427 conn(0x7fe7b806e310 msgr2=0x7fe7b80744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 shutdown_connections 2024-03-22T12:56:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.173+0000 7fe7be400640 1 -- 172.21.15.6:0/2877242427 wait complete. 2024-03-22T12:56:55.192 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-22T12:56:55.192 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T12:56:55.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:56:55.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.487+0000 7f5642400640 1 Processor -- start 2024-03-22T12:56:55.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.487+0000 7f5642400640 1 -- start start 2024-03-22T12:56:55.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.489+0000 7f5642400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f563c0ca590 0x7f563c0ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.489+0000 7f5642400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0cc5f0 0x7f563c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.489+0000 7f5642400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f563c0cdfd0 0x7f563c0ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.489+0000 7f5642400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c0cdb40 con 0x7f563c0ca590 2024-03-22T12:56:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.489+0000 7f5642400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c0ce9a0 con 0x7f563c0cdfd0 2024-03-22T12:56:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.489+0000 7f5642400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0ccf30 con 0x7f563c0cc5f0 2024-03-22T12:56:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.490+0000 7f563b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f563c0ca590 0x7f563c0ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.489+0000 7f563be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f563c0cdfd0 0x7f563c0ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.490+0000 7f563aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0cc5f0 0x7f563c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.490+0000 7f563aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0cc5f0 0x7f563c0cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59314/0 (socket says 172.21.15.6:59314) 2024-03-22T12:56:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.490+0000 7f563aa00640 1 -- 172.21.15.6:0/3855855730 learned_addr learned my addr 172.21.15.6:0/3855855730 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.490+0000 7f563be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f563c0cdfd0 0x7f563c0ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47556/0 (socket says 172.21.15.6:47556) 2024-03-22T12:56:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.492+0000 7f563aa00640 1 -- 172.21.15.6:0/3855855730 >> v2:172.21.15.92:3300/0 conn(0x7f563c0cdfd0 msgr2=0x7f563c0ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.492+0000 7f563aa00640 1 --2- 172.21.15.6:0/3855855730 >> v2:172.21.15.92:3300/0 conn(0x7f563c0cdfd0 0x7f563c0ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.492+0000 7f563aa00640 1 -- 172.21.15.6:0/3855855730 >> v2:172.21.15.6:3300/0 conn(0x7f563c0ca590 msgr2=0x7f563c0ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.492+0000 7f563aa00640 1 --2- 172.21.15.6:0/3855855730 >> v2:172.21.15.6:3300/0 conn(0x7f563c0ca590 0x7f563c0ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.492+0000 7f563aa00640 1 -- 172.21.15.6:0/3855855730 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c15a790 con 0x7f563c0cc5f0 2024-03-22T12:56:55.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.494+0000 7f563aa00640 1 --2- 172.21.15.6:0/3855855730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0cc5f0 0x7f563c0cc9f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5628009870 tx=0x7f562802b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4570f3895f46ce6d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.498+0000 7f563a000640 1 -- 172.21.15.6:0/3855855730 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628034070 con 0x7f563c0cc5f0 2024-03-22T12:56:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.498+0000 7f563a000640 1 -- 172.21.15.6:0/3855855730 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5628030850 con 0x7f563c0cc5f0 2024-03-22T12:56:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.498+0000 7f563a000640 1 -- 172.21.15.6:0/3855855730 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628030c20 con 0x7f563c0cc5f0 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 -- 172.21.15.6:0/3855855730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0cc5f0 msgr2=0x7f563c0cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 --2- 172.21.15.6:0/3855855730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0cc5f0 0x7f563c0cc9f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5628009870 tx=0x7f562802b990 comp rx=0 tx=0).stop 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 -- 172.21.15.6:0/3855855730 shutdown_connections 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 --2- 172.21.15.6:0/3855855730 >> v2:172.21.15.92:3300/0 conn(0x7f563c0cdfd0 0x7f563c0ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 --2- 172.21.15.6:0/3855855730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0cc5f0 0x7f563c0cc9f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 --2- 172.21.15.6:0/3855855730 >> v2:172.21.15.6:3300/0 conn(0x7f563c0ca590 0x7f563c0ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 -- 172.21.15.6:0/3855855730 >> 172.21.15.6:0/3855855730 conn(0x7f563c0b1720 msgr2=0x7f563c0b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 -- 172.21.15.6:0/3855855730 shutdown_connections 2024-03-22T12:56:55.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.499+0000 7f5642400640 1 -- 172.21.15.6:0/3855855730 wait complete. 2024-03-22T12:56:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.501+0000 7f5642400640 1 Processor -- start 2024-03-22T12:56:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.501+0000 7f5642400640 1 -- start start 2024-03-22T12:56:55.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.502+0000 7f5642400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0ca590 0x7f563c158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:55.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.502+0000 7f5642400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f563c0cc5f0 0x7f563c15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:55.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.502+0000 7f5642400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f563c0cdfd0 0x7f563c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:55.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.502+0000 7f5642400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c15b030 con 0x7f563c0cc5f0 2024-03-22T12:56:55.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.502+0000 7f5642400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c15ad50 con 0x7f563c0cdfd0 2024-03-22T12:56:55.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.502+0000 7f5642400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c15b310 con 0x7f563c0ca590 2024-03-22T12:56:55.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.503+0000 7f563b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0ca590 0x7f563c158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:55.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.503+0000 7f563aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f563c0cc5f0 0x7f563c15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:55.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.503+0000 7f563b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0ca590 0x7f563c158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59322/0 (socket says 172.21.15.6:59322) 2024-03-22T12:56:55.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.503+0000 7f563b400640 1 -- 172.21.15.6:0/2489733428 learned_addr learned my addr 172.21.15.6:0/2489733428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:55.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.507+0000 7f563b400640 1 -- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f563c0cdfd0 msgr2=0x7f563c15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:55.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.507+0000 7f563b400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f563c0cdfd0 0x7f563c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.507+0000 7f563b400640 1 -- 172.21.15.6:0/2489733428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f563c0cc5f0 msgr2=0x7f563c15b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.507+0000 7f563b400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f563c0cc5f0 0x7f563c15b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:55.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.507+0000 7f563b400640 1 -- 172.21.15.6:0/2489733428 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0c74a0 con 0x7f563c0ca590 2024-03-22T12:56:55.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.509+0000 7f563b400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0ca590 0x7f563c158540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f562c00e9b0 tx=0x7f562c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.512+0000 7f5633400640 1 -- 172.21.15.6:0/2489733428 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c017070 con 0x7f563c0ca590 2024-03-22T12:56:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.513+0000 7f5633400640 1 -- 172.21.15.6:0/2489733428 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c00c920 con 0x7f563c0ca590 2024-03-22T12:56:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.513+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c1e4fa0 con 0x7f563c0ca590 2024-03-22T12:56:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.513+0000 7f5633400640 1 -- 172.21.15.6:0/2489733428 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c00cd10 con 0x7f563c0ca590 2024-03-22T12:56:55.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.513+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c15d8d0 con 0x7f563c0ca590 2024-03-22T12:56:55.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.514+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c0756c0 con 0x7f563c0ca590 2024-03-22T12:56:55.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.532+0000 7f5633400640 1 -- 172.21.15.6:0/2489733428 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f562c01a070 con 0x7f563c0ca590 2024-03-22T12:56:55.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.534+0000 7f5633400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f561c06bc50 0x7f561c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:55.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.535+0000 7f5633400640 1 -- 172.21.15.6:0/2489733428 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f562c057680 con 0x7f563c0ca590 2024-03-22T12:56:55.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.535+0000 7f563aa00640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f561c06bc50 0x7f561c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:55.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.536+0000 7f563aa00640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f561c06bc50 0x7f561c06e110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5628036000 tx=0x7f5628008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:55.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.775+0000 7f5633400640 1 -- 172.21.15.6:0/2489733428 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f563c1e4fa0 con 0x7f563c0ca590 2024-03-22T12:56:55.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:55.984+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f563c0ce460 con 0x7f563c0ca590 2024-03-22T12:56:56.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.021+0000 7f5633400640 1 -- 172.21.15.6:0/2489733428 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18418 (secure 0 0 0) 0x7f562c0595e0 con 0x7f563c0ca590 2024-03-22T12:56:56.022 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:56:56.022 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:56:37.855636+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:56:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f561c06bc50 msgr2=0x7f561c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f561c06bc50 0x7f561c06e110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5628036000 tx=0x7f5628008030 comp rx=0 tx=0).stop 2024-03-22T12:56:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0ca590 msgr2=0x7f563c158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0ca590 0x7f563c158540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f562c00e9b0 tx=0x7f562c00ee70 comp rx=0 tx=0).stop 2024-03-22T12:56:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 shutdown_connections 2024-03-22T12:56:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f561c06bc50 0x7f561c06e110 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f563c0cdfd0 0x7f563c15bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f563c0cc5f0 0x7f563c15b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 --2- 172.21.15.6:0/2489733428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563c0ca590 0x7f563c158540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 >> 172.21.15.6:0/2489733428 conn(0x7f563c0b1720 msgr2=0x7f563c1541e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 shutdown_connections 2024-03-22T12:56:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.024+0000 7f5642400640 1 -- 172.21.15.6:0/2489733428 wait complete. 2024-03-22T12:56:56.042 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T12:56:14.196164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T12:56:26.857509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T12:56:28.394044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T12:56:34.623081+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T12:56:56.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T12:56:56.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.231+0000 7fda2e800640 1 Processor -- start 2024-03-22T12:56:56.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.231+0000 7fda2e800640 1 -- start start 2024-03-22T12:56:56.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.233+0000 7fda2e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda28073360 0x7fda280c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.233+0000 7fda2e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda28157480 con 0x7fda280c3eb0 2024-03-22T12:56:56.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.233+0000 7fda2e800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda28157760 con 0x7fda28151050 2024-03-22T12:56:56.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.233+0000 7fda2e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda28158000 con 0x7fda28073360 2024-03-22T12:56:56.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.234+0000 7fda26a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda28073360 0x7fda280c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.234+0000 7fda27e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fda280c3eb0 0x7fda280c6760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47722/0 (socket says 172.21.15.6:47722) 2024-03-22T12:56:56.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.234+0000 7fda27e00640 1 -- 172.21.15.6:0/296023475 learned_addr learned my addr 172.21.15.6:0/296023475 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:56.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.238+0000 7fda26a00640 1 -- 172.21.15.6:0/296023475 >> v1:172.21.15.92:6789/0 conn(0x7fda28151050 legacy=0x7fda2815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:56.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.238+0000 7fda26a00640 1 -- 172.21.15.6:0/296023475 >> v1:172.21.15.6:6789/0 conn(0x7fda280c3eb0 legacy=0x7fda280c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.238+0000 7fda26a00640 1 -- 172.21.15.6:0/296023475 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda28157a40 con 0x7fda28073360 2024-03-22T12:56:56.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.245+0000 7fda26a00640 1 --2- 172.21.15.6:0/296023475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28073360 0x7fda280c6220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda140097a0 tx=0x7fda1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdde999bcb0c9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:56.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.248+0000 7fda26000640 1 -- 172.21.15.6:0/296023475 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14035020 con 0x7fda28073360 2024-03-22T12:56:56.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.248+0000 7fda26000640 1 -- 172.21.15.6:0/296023475 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda14031830 con 0x7fda28073360 2024-03-22T12:56:56.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.248+0000 7fda26000640 1 -- 172.21.15.6:0/296023475 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14031c00 con 0x7fda28073360 2024-03-22T12:56:56.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.249+0000 7fda2e800640 1 -- 172.21.15.6:0/296023475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28073360 msgr2=0x7fda280c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.249+0000 7fda2e800640 1 --2- 172.21.15.6:0/296023475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28073360 0x7fda280c6220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda140097a0 tx=0x7fda1402b8d0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.249+0000 7fda2e800640 1 -- 172.21.15.6:0/296023475 shutdown_connections 2024-03-22T12:56:56.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.249+0000 7fda2e800640 1 --2- 172.21.15.6:0/296023475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28073360 0x7fda280c6220 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.249+0000 7fda2e800640 1 -- 172.21.15.6:0/296023475 >> 172.21.15.6:0/296023475 conn(0x7fda28072790 msgr2=0x7fda28072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:56.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.249+0000 7fda2e800640 1 -- 172.21.15.6:0/296023475 shutdown_connections 2024-03-22T12:56:56.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.249+0000 7fda2e800640 1 -- 172.21.15.6:0/296023475 wait complete. 2024-03-22T12:56:56.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.251+0000 7fda2e800640 1 Processor -- start 2024-03-22T12:56:56.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.251+0000 7fda2e800640 1 -- start start 2024-03-22T12:56:56.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.252+0000 7fda2e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda28073360 0x7fda280cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.252+0000 7fda2e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda280c3eb0 0x7fda280c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.252+0000 7fda2e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28151050 0x7fda280c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.252+0000 7fda2e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2815a760 con 0x7fda28151050 2024-03-22T12:56:56.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.252+0000 7fda2e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda280c6e70 con 0x7fda280c3eb0 2024-03-22T12:56:56.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.252+0000 7fda2e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda28151430 con 0x7fda28073360 2024-03-22T12:56:56.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.253+0000 7fda27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28151050 0x7fda280c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.253+0000 7fda27400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda28073360 0x7fda280cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.253+0000 7fda26a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda280c3eb0 0x7fda280c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.253+0000 7fda27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28151050 0x7fda280c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50090/0 (socket says 172.21.15.6:50090) 2024-03-22T12:56:56.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.253+0000 7fda27e00640 1 -- 172.21.15.6:0/1392890424 learned_addr learned my addr 172.21.15.6:0/1392890424 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:56.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.257+0000 7fda26a00640 1 -- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda28073360 msgr2=0x7fda280cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.257+0000 7fda26a00640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda28073360 0x7fda280cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.257+0000 7fda26a00640 1 -- 172.21.15.6:0/1392890424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28151050 msgr2=0x7fda280c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.257+0000 7fda26a00640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28151050 0x7fda280c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.257+0000 7fda26a00640 1 -- 172.21.15.6:0/1392890424 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda280c76d0 con 0x7fda280c3eb0 2024-03-22T12:56:56.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.260+0000 7fda26a00640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda280c3eb0 0x7fda280c9020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda14033000 tx=0x7fda140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.264+0000 7fda1f400640 1 -- 172.21.15.6:0/1392890424 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14041070 con 0x7fda280c3eb0 2024-03-22T12:56:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.264+0000 7fda1f400640 1 -- 172.21.15.6:0/1392890424 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1403aa30 con 0x7fda280c3eb0 2024-03-22T12:56:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.264+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda281f4930 con 0x7fda280c3eb0 2024-03-22T12:56:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.264+0000 7fda1f400640 1 -- 172.21.15.6:0/1392890424 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14035040 con 0x7fda280c3eb0 2024-03-22T12:56:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.264+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda281f4c10 con 0x7fda280c3eb0 2024-03-22T12:56:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.265+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda28070680 con 0x7fda280c3eb0 2024-03-22T12:56:56.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.284+0000 7fda1f400640 1 -- 172.21.15.6:0/1392890424 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fda1404c060 con 0x7fda280c3eb0 2024-03-22T12:56:56.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.285+0000 7fda1f400640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda0806bc20 0x7fda0806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.285+0000 7fda27400640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda0806bc20 0x7fda0806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.286+0000 7fda27400640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda0806bc20 0x7fda0806e0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda18006fd0 tx=0x7fda18009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:56.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.286+0000 7fda1f400640 1 -- 172.21.15.6:0/1392890424 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fda14046df0 con 0x7fda280c3eb0 2024-03-22T12:56:56.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.529+0000 7fda1f400640 1 -- 172.21.15.6:0/1392890424 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fda28070680 con 0x7fda280c3eb0 2024-03-22T12:56:56.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.730+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda280757d0 con 0x7fda280c3eb0 2024-03-22T12:56:56.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.739+0000 7fda1f400640 1 -- 172.21.15.6:0/1392890424 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda1407acd0 con 0x7fda280c3eb0 2024-03-22T12:56:56.739 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-22T12:56:56.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda0806bc20 msgr2=0x7fda0806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda0806bc20 0x7fda0806e0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda18006fd0 tx=0x7fda18009060 comp rx=0 tx=0).stop 2024-03-22T12:56:56.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda280c3eb0 msgr2=0x7fda280c9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda280c3eb0 0x7fda280c9020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda14033000 tx=0x7fda140046c0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 shutdown_connections 2024-03-22T12:56:56.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda28151050 0x7fda280c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda0806bc20 0x7fda0806e0e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda280c3eb0 0x7fda280c9020 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 --2- 172.21.15.6:0/1392890424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda28073360 0x7fda280cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 >> 172.21.15.6:0/1392890424 conn(0x7fda28072790 msgr2=0x7fda28158db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:56.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 shutdown_connections 2024-03-22T12:56:56.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.742+0000 7fda2e800640 1 -- 172.21.15.6:0/1392890424 wait complete. 2024-03-22T12:56:56.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T12:56:56.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.954+0000 7f17d8c00640 1 Processor -- start 2024-03-22T12:56:56.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.954+0000 7f17d8c00640 1 -- start start 2024-03-22T12:56:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.956+0000 7f17d8c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f17d41544e0 0x7f17d415d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.956+0000 7f17d8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17d41548c0 0x7f17d415de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.956+0000 7f17d8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d4073520 con 0x7f17d40c80a0 2024-03-22T12:56:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.956+0000 7f17d8c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d415a550 con 0x7f17d41548c0 2024-03-22T12:56:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.956+0000 7f17d8c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d415a830 con 0x7f17d41544e0 2024-03-22T12:56:56.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.957+0000 7f17d1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f17d41544e0 0x7f17d415d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.957+0000 7f17c9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17d41548c0 0x7f17d415de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.957+0000 7f17d1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f17d41544e0 0x7f17d415d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47592/0 (socket says 172.21.15.6:47592) 2024-03-22T12:56:56.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.957+0000 7f17d1e00640 1 -- 172.21.15.6:0/3395690051 learned_addr learned my addr 172.21.15.6:0/3395690051 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:56.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.959+0000 7f17d1e00640 1 -- 172.21.15.6:0/3395690051 >> v2:172.21.15.6:3300/0 conn(0x7f17d41548c0 msgr2=0x7f17d415de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:56.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.959+0000 7f17d1e00640 1 --2- 172.21.15.6:0/3395690051 >> v2:172.21.15.6:3300/0 conn(0x7f17d41548c0 0x7f17d415de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.959+0000 7f17d1e00640 1 -- 172.21.15.6:0/3395690051 >> v1:172.21.15.6:6789/0 conn(0x7f17d40c80a0 legacy=0x7f17d40c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:56.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.959+0000 7f17d1e00640 1 -- 172.21.15.6:0/3395690051 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d415ab10 con 0x7f17d41544e0 2024-03-22T12:56:56.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.961+0000 7f17d1e00640 1 --2- 172.21.15.6:0/3395690051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d41544e0 0x7f17d415d930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17d40c9230 tx=0x7f17c402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad84d7d322da1b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:56.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.965+0000 7f17d1400640 1 -- 172.21.15.6:0/3395690051 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4034070 con 0x7f17d41544e0 2024-03-22T12:56:56.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.965+0000 7f17d1400640 1 -- 172.21.15.6:0/3395690051 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c4030860 con 0x7f17d41544e0 2024-03-22T12:56:56.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.965+0000 7f17d1400640 1 -- 172.21.15.6:0/3395690051 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4030c30 con 0x7f17d41544e0 2024-03-22T12:56:56.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.965+0000 7f17d8c00640 1 -- 172.21.15.6:0/3395690051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d41544e0 msgr2=0x7f17d415d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.965+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3395690051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d41544e0 0x7f17d415d930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f17d40c9230 tx=0x7f17c402b9a0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.966+0000 7f17d8c00640 1 -- 172.21.15.6:0/3395690051 shutdown_connections 2024-03-22T12:56:56.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.966+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3395690051 >> v2:172.21.15.6:3300/0 conn(0x7f17d41548c0 0x7f17d415de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.966+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3395690051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d41544e0 0x7f17d415d930 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.966+0000 7f17d8c00640 1 -- 172.21.15.6:0/3395690051 >> 172.21.15.6:0/3395690051 conn(0x7f17d406e310 msgr2=0x7f17d406f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:56.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.966+0000 7f17d8c00640 1 -- 172.21.15.6:0/3395690051 shutdown_connections 2024-03-22T12:56:56.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.966+0000 7f17d8c00640 1 -- 172.21.15.6:0/3395690051 wait complete. 2024-03-22T12:56:56.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.967+0000 7f17d8c00640 1 Processor -- start 2024-03-22T12:56:56.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.967+0000 7f17d8c00640 1 -- start start 2024-03-22T12:56:56.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.968+0000 7f17d8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d40c80a0 0x7f17d41f4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.968+0000 7f17d8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d41544e0 0x7f17d41f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.968+0000 7f17d8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d4150de0 0x7f17d4159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:56.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.968+0000 7f17d8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d40c7240 con 0x7f17d41544e0 2024-03-22T12:56:56.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.968+0000 7f17d8c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d40c70c0 con 0x7f17d40c80a0 2024-03-22T12:56:56.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.968+0000 7f17d8c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d4072900 con 0x7f17d4150de0 2024-03-22T12:56:56.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.969+0000 7f17c9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d41544e0 0x7f17d41f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.969+0000 7f17d1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d40c80a0 0x7f17d41f4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.969+0000 7f17c9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d41544e0 0x7f17d41f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50108/0 (socket says 172.21.15.6:50108) 2024-03-22T12:56:56.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.969+0000 7f17c9400640 1 -- 172.21.15.6:0/3707080721 learned_addr learned my addr 172.21.15.6:0/3707080721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:56.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.969+0000 7f17d2800640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d4150de0 0x7f17d4159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:56.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.973+0000 7f17d1e00640 1 -- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d4150de0 msgr2=0x7f17d4159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.973+0000 7f17d1e00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d4150de0 0x7f17d4159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.973+0000 7f17d1e00640 1 -- 172.21.15.6:0/3707080721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d41544e0 msgr2=0x7f17d41f5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.973+0000 7f17d1e00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d41544e0 0x7f17d41f5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.973+0000 7f17d1e00640 1 -- 172.21.15.6:0/3707080721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d41f5e60 con 0x7f17d40c80a0 2024-03-22T12:56:56.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.976+0000 7f17d1e00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d40c80a0 0x7f17d41f4f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17c4033000 tx=0x7f17c4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.981+0000 7f17caa00640 1 -- 172.21.15.6:0/3707080721 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4048070 con 0x7f17d40c80a0 2024-03-22T12:56:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.982+0000 7f17caa00640 1 -- 172.21.15.6:0/3707080721 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c4030660 con 0x7f17d40c80a0 2024-03-22T12:56:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.982+0000 7f17caa00640 1 -- 172.21.15.6:0/3707080721 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4034040 con 0x7f17d40c80a0 2024-03-22T12:56:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.982+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d4151de0 con 0x7f17d40c80a0 2024-03-22T12:56:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.982+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d4152390 con 0x7f17d40c80a0 2024-03-22T12:56:56.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:56.983+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d40757d0 con 0x7f17d40c80a0 2024-03-22T12:56:57.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.003+0000 7f17caa00640 1 -- 172.21.15.6:0/3707080721 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f17c404b070 con 0x7f17d40c80a0 2024-03-22T12:56:57.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.004+0000 7f17caa00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f17b006bc00 0x7f17b006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:57.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.004+0000 7f17c9400640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f17b006bc00 0x7f17b006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:57.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.005+0000 7f17c9400640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f17b006bc00 0x7f17b006e0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17d40c2b50 tx=0x7f17bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:57.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.005+0000 7f17caa00640 1 -- 172.21.15.6:0/3707080721 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f17c4030810 con 0x7f17d40c80a0 2024-03-22T12:56:57.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.244+0000 7f17caa00640 1 -- 172.21.15.6:0/3707080721 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f17c407b210 con 0x7f17d40c80a0 2024-03-22T12:56:57.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.465+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f17d4153220 con 0x7f17d40c80a0 2024-03-22T12:56:57.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.470+0000 7f17caa00640 1 -- 172.21.15.6:0/3707080721 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f17c407f2b0 con 0x7f17d40c80a0 2024-03-22T12:56:57.470 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f17b006bc00 msgr2=0x7f17b006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f17b006bc00 0x7f17b006e0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17d40c2b50 tx=0x7f17bc008040 comp rx=0 tx=0).stop 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d40c80a0 msgr2=0x7f17d41f4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d40c80a0 0x7f17d41f4f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17c4033000 tx=0x7f17c4004730 comp rx=0 tx=0).stop 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 shutdown_connections 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d4150de0 0x7f17d4159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17d41544e0 0x7f17d41f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f17b006bc00 0x7f17b006e0c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 --2- 172.21.15.6:0/3707080721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d40c80a0 0x7f17d41f4f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 >> 172.21.15.6:0/3707080721 conn(0x7f17d406e310 msgr2=0x7f17d40744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 shutdown_connections 2024-03-22T12:56:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.473+0000 7f17d8c00640 1 -- 172.21.15.6:0/3707080721 wait complete. 2024-03-22T12:56:57.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T12:56:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.689+0000 7f6e83000640 1 Processor -- start 2024-03-22T12:56:57.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.689+0000 7f6e83000640 1 -- start start 2024-03-22T12:56:57.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.691+0000 7f6e83000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6e7c1544e0 0x7f6e7c15d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.691+0000 7f6e83000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c1548c0 0x7f6e7c15de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.691+0000 7f6e83000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7c073520 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.691+0000 7f6e83000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7c15a550 con 0x7f6e7c1544e0 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.691+0000 7f6e83000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c15a830 con 0x7f6e7c1548c0 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.691+0000 7f6e7be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6e7c1544e0 0x7f6e7c15d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.691+0000 7f6e73e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c1548c0 0x7f6e7c15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.692+0000 7f6e81000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6e7c0c80a0 0x7f6e7c0c69b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47736/0 (socket says 172.21.15.6:47736) 2024-03-22T12:56:57.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.692+0000 7f6e81000640 1 -- 172.21.15.6:0/4053563463 learned_addr learned my addr 172.21.15.6:0/4053563463 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:57.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.695+0000 7f6e73e00640 1 -- 172.21.15.6:0/4053563463 >> v2:172.21.15.92:3300/0 conn(0x7f6e7c1544e0 msgr2=0x7f6e7c15d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:57.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.695+0000 7f6e73e00640 1 --2- 172.21.15.6:0/4053563463 >> v2:172.21.15.92:3300/0 conn(0x7f6e7c1544e0 0x7f6e7c15d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.695+0000 7f6e73e00640 1 -- 172.21.15.6:0/4053563463 >> v1:172.21.15.6:6789/0 conn(0x7f6e7c0c80a0 legacy=0x7f6e7c0c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:57.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.695+0000 7f6e73e00640 1 -- 172.21.15.6:0/4053563463 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c15ab10 con 0x7f6e7c1548c0 2024-03-22T12:56:57.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.695+0000 7f6e7be00640 1 --2- 172.21.15.6:0/4053563463 >> v2:172.21.15.92:3300/0 conn(0x7f6e7c1544e0 0x7f6e7c15d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:56:57.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.698+0000 7f6e73e00640 1 --2- 172.21.15.6:0/4053563463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c1548c0 0x7f6e7c15de70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6e640097a0 tx=0x7f6e6402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dabb2074fbda988 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:57.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.701+0000 7f6e7b400640 1 -- 172.21.15.6:0/4053563463 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64034070 con 0x7f6e7c1548c0 2024-03-22T12:56:57.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.701+0000 7f6e7b400640 1 -- 172.21.15.6:0/4053563463 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e64030610 con 0x7f6e7c1548c0 2024-03-22T12:56:57.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.701+0000 7f6e7b400640 1 -- 172.21.15.6:0/4053563463 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640309e0 con 0x7f6e7c1548c0 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 -- 172.21.15.6:0/4053563463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c1548c0 msgr2=0x7f6e7c15de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 --2- 172.21.15.6:0/4053563463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c1548c0 0x7f6e7c15de70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6e640097a0 tx=0x7f6e6402b960 comp rx=0 tx=0).stop 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 -- 172.21.15.6:0/4053563463 shutdown_connections 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 --2- 172.21.15.6:0/4053563463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c1548c0 0x7f6e7c15de70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 --2- 172.21.15.6:0/4053563463 >> v2:172.21.15.92:3300/0 conn(0x7f6e7c1544e0 0x7f6e7c15d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 -- 172.21.15.6:0/4053563463 >> 172.21.15.6:0/4053563463 conn(0x7f6e7c06e310 msgr2=0x7f6e7c06f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 -- 172.21.15.6:0/4053563463 shutdown_connections 2024-03-22T12:56:57.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.702+0000 7f6e83000640 1 -- 172.21.15.6:0/4053563463 wait complete. 2024-03-22T12:56:57.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.704+0000 7f6e83000640 1 Processor -- start 2024-03-22T12:56:57.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.704+0000 7f6e83000640 1 -- start start 2024-03-22T12:56:57.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.705+0000 7f6e83000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c0c80a0 0x7f6e7c14f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:57.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.705+0000 7f6e83000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e7c1544e0 0x7f6e7c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:57.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.705+0000 7f6e83000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7c0c5c00 0x7f6e7c0c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:57.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.705+0000 7f6e83000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c7240 con 0x7f6e7c0c5c00 2024-03-22T12:56:57.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.705+0000 7f6e83000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c0c70c0 con 0x7f6e7c1544e0 2024-03-22T12:56:57.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.705+0000 7f6e83000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c072900 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.706+0000 7f6e7be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c0c80a0 0x7f6e7c14f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:57.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.706+0000 7f6e73e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e7c1544e0 0x7f6e7c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:57.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.706+0000 7f6e7be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c0c80a0 0x7f6e7c14f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59354/0 (socket says 172.21.15.6:59354) 2024-03-22T12:56:57.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.706+0000 7f6e7be00640 1 -- 172.21.15.6:0/4170577382 learned_addr learned my addr 172.21.15.6:0/4170577382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:57.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.710+0000 7f6e7be00640 1 -- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e7c1544e0 msgr2=0x7f6e7c0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:57.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.710+0000 7f6e7be00640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e7c1544e0 0x7f6e7c0c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.710+0000 7f6e7be00640 1 -- 172.21.15.6:0/4170577382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7c0c5c00 msgr2=0x7f6e7c0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:57.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.710+0000 7f6e7be00640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7c0c5c00 0x7f6e7c0c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:57.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.710+0000 7f6e7be00640 1 -- 172.21.15.6:0/4170577382 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7c152d80 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.713+0000 7f6e7be00640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c0c80a0 0x7f6e7c14f7e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00ec40 tx=0x7f6e6c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.717+0000 7f6e78c00640 1 -- 172.21.15.6:0/4170577382 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c017070 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.717+0000 7f6e78c00640 1 -- 172.21.15.6:0/4170577382 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c00cbd0 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.717+0000 7f6e78c00640 1 -- 172.21.15.6:0/4170577382 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c013470 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.717+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c0c9940 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.717+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c0c3170 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.718+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7c0757d0 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.737+0000 7f6e78c00640 1 -- 172.21.15.6:0/4170577382 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e6c01a070 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.738+0000 7f6e78c00640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e5806bc50 0x7f6e5806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:57.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.738+0000 7f6e73e00640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e5806bc50 0x7f6e5806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.739+0000 7f6e73e00640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e5806bc50 0x7f6e5806e110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6e64036000 tx=0x7f6e6403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.739+0000 7f6e78c00640 1 -- 172.21.15.6:0/4170577382 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6e6c057660 con 0x7f6e7c0c80a0 2024-03-22T12:56:57.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:57.975+0000 7f6e78c00640 1 -- 172.21.15.6:0/4170577382 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e6c0595c0 con 0x7f6e7c0c80a0 2024-03-22T12:56:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.179+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6e7c15e7d0 con 0x7f6e7c0c80a0 2024-03-22T12:56:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.186+0000 7f6e78c00640 1 -- 172.21.15.6:0/4170577382 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6e6c05d660 con 0x7f6e7c0c80a0 2024-03-22T12:56:58.187 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-22T12:56:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e5806bc50 msgr2=0x7f6e5806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e5806bc50 0x7f6e5806e110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6e64036000 tx=0x7f6e6403b040 comp rx=0 tx=0).stop 2024-03-22T12:56:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c0c80a0 msgr2=0x7f6e7c14f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c0c80a0 0x7f6e7c14f7e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00ec40 tx=0x7f6e6c00c520 comp rx=0 tx=0).stop 2024-03-22T12:56:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 shutdown_connections 2024-03-22T12:56:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7c0c5c00 0x7f6e7c0c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e5806bc50 0x7f6e5806e110 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e7c1544e0 0x7f6e7c0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 --2- 172.21.15.6:0/4170577382 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e7c0c80a0 0x7f6e7c14f7e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 >> 172.21.15.6:0/4170577382 conn(0x7f6e7c06e310 msgr2=0x7f6e7c0c8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 shutdown_connections 2024-03-22T12:56:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.189+0000 7f6e83000640 1 -- 172.21.15.6:0/4170577382 wait complete. 2024-03-22T12:56:58.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T12:56:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.425+0000 7f4132600640 1 Processor -- start 2024-03-22T12:56:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.425+0000 7f4132600640 1 -- start start 2024-03-22T12:56:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.427+0000 7f4132600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c1417f0 0x7f412c0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.427+0000 7f4132600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f412c06b080 0x7f412c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.427+0000 7f4132600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c06b930 con 0x7f412c06b4a0 2024-03-22T12:56:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.427+0000 7f4132600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c06bc10 con 0x7f412c06b080 2024-03-22T12:56:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.427+0000 7f4132600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c158500 con 0x7f412c1417f0 2024-03-22T12:56:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.465+0000 7f412be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f412c06b4a0 0x7f412c0c60c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:54484/0 (socket says 172.21.15.6:54484) 2024-03-22T12:56:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.465+0000 7f412be00640 1 -- 172.21.15.6:0/1604261471 learned_addr learned my addr 172.21.15.6:0/1604261471 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:58.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.471+0000 7f412b400640 1 --2- 172.21.15.6:0/1604261471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c1417f0 0x7f412c0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.473+0000 7f412b400640 1 -- 172.21.15.6:0/1604261471 >> v2:172.21.15.6:3300/0 conn(0x7f412c06b080 msgr2=0x7f412c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:56:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.473+0000 7f412b400640 1 --2- 172.21.15.6:0/1604261471 >> v2:172.21.15.6:3300/0 conn(0x7f412c06b080 0x7f412c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.473+0000 7f412b400640 1 -- 172.21.15.6:0/1604261471 >> v1:172.21.15.92:6789/0 conn(0x7f412c06b4a0 legacy=0x7f412c0c60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.473+0000 7f412b400640 1 -- 172.21.15.6:0/1604261471 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c1587e0 con 0x7f412c1417f0 2024-03-22T12:56:58.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.477+0000 7f412b400640 1 --2- 172.21.15.6:0/1604261471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c1417f0 0x7f412c0c7dc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f411c0097a0 tx=0x7f411c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771160dd4f991ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.479+0000 7f412a000640 1 -- 172.21.15.6:0/1604261471 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c0315a0 con 0x7f412c1417f0 2024-03-22T12:56:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.480+0000 7f412a000640 1 -- 172.21.15.6:0/1604261471 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c037070 con 0x7f412c1417f0 2024-03-22T12:56:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.480+0000 7f412a000640 1 -- 172.21.15.6:0/1604261471 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c031960 con 0x7f412c1417f0 2024-03-22T12:56:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.481+0000 7f4132600640 1 -- 172.21.15.6:0/1604261471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c1417f0 msgr2=0x7f412c0c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.481+0000 7f4132600640 1 --2- 172.21.15.6:0/1604261471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c1417f0 0x7f412c0c7dc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f411c0097a0 tx=0x7f411c02b840 comp rx=0 tx=0).stop 2024-03-22T12:56:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.482+0000 7f4132600640 1 -- 172.21.15.6:0/1604261471 shutdown_connections 2024-03-22T12:56:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.482+0000 7f4132600640 1 --2- 172.21.15.6:0/1604261471 >> v2:172.21.15.6:3300/0 conn(0x7f412c06b080 0x7f412c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.482+0000 7f4132600640 1 --2- 172.21.15.6:0/1604261471 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c1417f0 0x7f412c0c7dc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.482+0000 7f4132600640 1 -- 172.21.15.6:0/1604261471 >> 172.21.15.6:0/1604261471 conn(0x7f412c072790 msgr2=0x7f412c072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.482+0000 7f4132600640 1 -- 172.21.15.6:0/1604261471 shutdown_connections 2024-03-22T12:56:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.482+0000 7f4132600640 1 -- 172.21.15.6:0/1604261471 wait complete. 2024-03-22T12:56:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.484+0000 7f4132600640 1 Processor -- start 2024-03-22T12:56:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.484+0000 7f4132600640 1 -- start start 2024-03-22T12:56:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.485+0000 7f4132600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c06b080 0x7f412c1f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.485+0000 7f4132600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f412c1417f0 0x7f412c1382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.485+0000 7f4132600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c151240 0x7f412c1516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.485+0000 7f4132600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0707a0 con 0x7f412c06b080 2024-03-22T12:56:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.485+0000 7f4132600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c1581f0 con 0x7f412c1417f0 2024-03-22T12:56:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.485+0000 7f4132600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c137e20 con 0x7f412c151240 2024-03-22T12:56:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.485+0000 7f412aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f412c1417f0 0x7f412c1382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.486+0000 7f412aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f412c1417f0 0x7f412c1382b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47606/0 (socket says 172.21.15.6:47606) 2024-03-22T12:56:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.486+0000 7f412aa00640 1 -- 172.21.15.6:0/2710300823 learned_addr learned my addr 172.21.15.6:0/2710300823 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.486+0000 7f412be00640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c151240 0x7f412c1516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:58.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.488+0000 7f412b400640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c06b080 0x7f412c1f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:58.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.489+0000 7f412aa00640 1 -- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c151240 msgr2=0x7f412c1516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.489+0000 7f412aa00640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c151240 0x7f412c1516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.489+0000 7f412aa00640 1 -- 172.21.15.6:0/2710300823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c06b080 msgr2=0x7f412c1f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.489+0000 7f412aa00640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c06b080 0x7f412c1f0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.489+0000 7f412aa00640 1 -- 172.21.15.6:0/2710300823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c1f0950 con 0x7f412c1417f0 2024-03-22T12:56:58.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.489+0000 7f412b400640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c06b080 0x7f412c1f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T12:56:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.492+0000 7f412aa00640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f412c1417f0 0x7f412c1382b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f411800b4d0 tx=0x7f411800b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.495+0000 7f4123400640 1 -- 172.21.15.6:0/2710300823 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118014070 con 0x7f412c1417f0 2024-03-22T12:56:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.495+0000 7f4123400640 1 -- 172.21.15.6:0/2710300823 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118010870 con 0x7f412c1417f0 2024-03-22T12:56:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.495+0000 7f4123400640 1 -- 172.21.15.6:0/2710300823 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118010c60 con 0x7f412c1417f0 2024-03-22T12:56:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.495+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c138eb0 con 0x7f412c1417f0 2024-03-22T12:56:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.496+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c1f0ea0 con 0x7f412c1417f0 2024-03-22T12:56:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.497+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c0757d0 con 0x7f412c1417f0 2024-03-22T12:56:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.515+0000 7f4123400640 1 -- 172.21.15.6:0/2710300823 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4118017070 con 0x7f412c1417f0 2024-03-22T12:56:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.516+0000 7f4123400640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f410c06bc20 0x7f410c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.516+0000 7f412b400640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f410c06bc20 0x7f410c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.517+0000 7f412b400640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f410c06bc20 0x7f410c06e0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f411c036000 tx=0x7f411c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.518+0000 7f4123400640 1 -- 172.21.15.6:0/2710300823 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4118057110 con 0x7f412c1417f0 2024-03-22T12:56:58.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.769+0000 7f4123400640 1 -- 172.21.15.6:0/2710300823 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f412c138eb0 con 0x7f412c1417f0 2024-03-22T12:56:58.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.991+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f412c0c6dd0 con 0x7f412c1417f0 2024-03-22T12:56:58.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.996+0000 7f4123400640 1 -- 172.21.15.6:0/2710300823 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4118059070 con 0x7f412c1417f0 2024-03-22T12:56:58.996 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f410c06bc20 msgr2=0x7f410c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f410c06bc20 0x7f410c06e0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f411c036000 tx=0x7f411c008030 comp rx=0 tx=0).stop 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f412c1417f0 msgr2=0x7f412c1382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f412c1417f0 0x7f412c1382b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f411800b4d0 tx=0x7f411800b990 comp rx=0 tx=0).stop 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 shutdown_connections 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f412c151240 0x7f412c1516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f410c06bc20 0x7f410c06e0e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:58.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f412c1417f0 0x7f412c1382b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 --2- 172.21.15.6:0/2710300823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f412c06b080 0x7f412c1f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 >> 172.21.15.6:0/2710300823 conn(0x7f412c072790 msgr2=0x7f412c0c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:59.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 shutdown_connections 2024-03-22T12:56:59.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:58.999+0000 7f4132600640 1 -- 172.21.15.6:0/2710300823 wait complete. 2024-03-22T12:56:59.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T12:56:59.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.224+0000 7f8c02000640 1 Processor -- start 2024-03-22T12:56:59.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.225+0000 7f8c02000640 1 -- start start 2024-03-22T12:56:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.226+0000 7f8c02000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0ceae0 0x7f8bfc0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:59.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.226+0000 7f8c02000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0ccf20 con 0x7f8bfc0caa80 2024-03-22T12:56:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.226+0000 7f8c02000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc15a6b0 con 0x7f8bfc0cf660 2024-03-22T12:56:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.226+0000 7f8c02000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0cfa40 con 0x7f8bfc0ceae0 2024-03-22T12:56:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.226+0000 7f8bf2a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0ceae0 0x7f8bfc0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.227+0000 7f8bfb400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8bfc0cf660 0x7f8bfc0cd9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:54496/0 (socket says 172.21.15.6:54496) 2024-03-22T12:56:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.227+0000 7f8bfb400640 1 -- 172.21.15.6:0/1898345505 learned_addr learned my addr 172.21.15.6:0/1898345505 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:56:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.229+0000 7f8bfaa00640 1 -- 172.21.15.6:0/1898345505 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227459543 0 0) 0x7f8bfc15a6b0 con 0x7f8bfc0cf660 2024-03-22T12:56:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.229+0000 7f8bfaa00640 1 -- 172.21.15.6:0/1898345505 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be0003520 con 0x7f8bfc0cf660 2024-03-22T12:56:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.229+0000 7f8bf2a00640 1 -- 172.21.15.6:0/1898345505 >> v1:172.21.15.92:6789/0 conn(0x7f8bfc0cf660 legacy=0x7f8bfc0cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.229+0000 7f8bf2a00640 1 -- 172.21.15.6:0/1898345505 >> v1:172.21.15.6:6789/0 conn(0x7f8bfc0caa80 legacy=0x7f8bfc0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.229+0000 7f8bf2a00640 1 -- 172.21.15.6:0/1898345505 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc15a990 con 0x7f8bfc0ceae0 2024-03-22T12:56:59.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.232+0000 7f8bf2a00640 1 --2- 172.21.15.6:0/1898345505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0ceae0 0x7f8bfc0ca540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8be40097d0 tx=0x7f8be402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f1be52e248f852 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.235+0000 7f8bfaa00640 1 -- 172.21.15.6:0/1898345505 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4034070 con 0x7f8bfc0ceae0 2024-03-22T12:56:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.235+0000 7f8bfaa00640 1 -- 172.21.15.6:0/1898345505 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be4030880 con 0x7f8bfc0ceae0 2024-03-22T12:56:59.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.235+0000 7f8bfaa00640 1 -- 172.21.15.6:0/1898345505 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4030c50 con 0x7f8bfc0ceae0 2024-03-22T12:56:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.236+0000 7f8c02000640 1 -- 172.21.15.6:0/1898345505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0ceae0 msgr2=0x7f8bfc0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.236+0000 7f8c02000640 1 --2- 172.21.15.6:0/1898345505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0ceae0 0x7f8bfc0ca540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8be40097d0 tx=0x7f8be402b9c0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.236+0000 7f8c02000640 1 -- 172.21.15.6:0/1898345505 shutdown_connections 2024-03-22T12:56:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.236+0000 7f8c02000640 1 --2- 172.21.15.6:0/1898345505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0ceae0 0x7f8bfc0ca540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.236+0000 7f8c02000640 1 -- 172.21.15.6:0/1898345505 >> 172.21.15.6:0/1898345505 conn(0x7f8bfc0b16d0 msgr2=0x7f8bfc0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.236+0000 7f8c02000640 1 -- 172.21.15.6:0/1898345505 shutdown_connections 2024-03-22T12:56:59.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.236+0000 7f8c02000640 1 -- 172.21.15.6:0/1898345505 wait complete. 2024-03-22T12:56:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.238+0000 7f8c02000640 1 Processor -- start 2024-03-22T12:56:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.238+0000 7f8c02000640 1 -- start start 2024-03-22T12:56:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.239+0000 7f8c02000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0caa80 0x7f8bfc151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.239+0000 7f8c02000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0ceae0 0x7f8bfc0b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.239+0000 7f8c02000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cf660 0x7f8bfc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.239+0000 7f8c02000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc15afe0 con 0x7f8bfc0ceae0 2024-03-22T12:56:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.239+0000 7f8c02000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc15b470 con 0x7f8bfc0cf660 2024-03-22T12:56:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.239+0000 7f8c02000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0ce0d0 con 0x7f8bfc0caa80 2024-03-22T12:56:59.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.240+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0caa80 0x7f8bfc151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:59.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.240+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cf660 0x7f8bfc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:59.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.240+0000 7f8bf2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0ceae0 0x7f8bfc0b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:59.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.240+0000 7f8bfb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0caa80 0x7f8bfc151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59372/0 (socket says 172.21.15.6:59372) 2024-03-22T12:56:59.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.240+0000 7f8bfb400640 1 -- 172.21.15.6:0/2482611031 learned_addr learned my addr 172.21.15.6:0/2482611031 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:56:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.243+0000 7f8bfb400640 1 -- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cf660 msgr2=0x7f8bfc0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.243+0000 7f8bfb400640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cf660 0x7f8bfc0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.243+0000 7f8bfb400640 1 -- 172.21.15.6:0/2482611031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0ceae0 msgr2=0x7f8bfc0b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.243+0000 7f8bfb400640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0ceae0 0x7f8bfc0b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.243+0000 7f8bfb400640 1 -- 172.21.15.6:0/2482611031 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc1f03f0 con 0x7f8bfc0caa80 2024-03-22T12:56:59.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.246+0000 7f8bfb400640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0caa80 0x7f8bfc151830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00cad0 tx=0x7f8bec00cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.249+0000 7f8bf3e00640 1 -- 172.21.15.6:0/2482611031 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec017070 con 0x7f8bfc0caa80 2024-03-22T12:56:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.249+0000 7f8bf3e00640 1 -- 172.21.15.6:0/2482611031 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec00ac30 con 0x7f8bfc0caa80 2024-03-22T12:56:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.249+0000 7f8bf3e00640 1 -- 172.21.15.6:0/2482611031 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec012400 con 0x7f8bfc0caa80 2024-03-22T12:56:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.249+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc1f06d0 con 0x7f8bfc0caa80 2024-03-22T12:56:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.249+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc1f0c90 con 0x7f8bfc0caa80 2024-03-22T12:56:59.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.251+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc158c00 con 0x7f8bfc0caa80 2024-03-22T12:56:59.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.269+0000 7f8bf3e00640 1 -- 172.21.15.6:0/2482611031 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8bec01a070 con 0x7f8bfc0caa80 2024-03-22T12:56:59.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.270+0000 7f8bf3e00640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bd806bc20 0x7f8bd806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:56:59.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.271+0000 7f8bf2a00640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bd806bc20 0x7f8bd806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:56:59.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.271+0000 7f8bf3e00640 1 -- 172.21.15.6:0/2482611031 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8bec057130 con 0x7f8bfc0caa80 2024-03-22T12:56:59.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.271+0000 7f8bf2a00640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bd806bc20 0x7f8bd806e0e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8be4036000 tx=0x7f8be403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:56:59.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:56:59.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:56:59.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.515+0000 7f8bf3e00640 1 -- 172.21.15.6:0/2482611031 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8bfc158c00 con 0x7f8bfc0caa80 2024-03-22T12:56:59.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.761+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8bfc0757d0 con 0x7f8bfc0caa80 2024-03-22T12:56:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.766+0000 7f8bf3e00640 1 -- 172.21.15.6:0/2482611031 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8bec021020 con 0x7f8bfc0caa80 2024-03-22T12:56:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bd806bc20 msgr2=0x7f8bd806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bd806bc20 0x7f8bd806e0e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8be4036000 tx=0x7f8be403b040 comp rx=0 tx=0).stop 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0caa80 msgr2=0x7f8bfc151830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0caa80 0x7f8bfc151830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00cad0 tx=0x7f8bec00cf90 comp rx=0 tx=0).stop 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 shutdown_connections 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bd806bc20 0x7f8bd806e0e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cf660 0x7f8bfc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0ceae0 0x7f8bfc0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 --2- 172.21.15.6:0/2482611031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0caa80 0x7f8bfc151830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 >> 172.21.15.6:0/2482611031 conn(0x7f8bfc0b16d0 msgr2=0x7f8bfc0c7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:56:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 shutdown_connections 2024-03-22T12:56:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:56:59.770+0000 7f8c02000640 1 -- 172.21.15.6:0/2482611031 wait complete. 2024-03-22T12:56:59.789 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-03-22T12:56:59.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:56:59.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-22T12:56:59.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[42105]: fuse finished with error 0 and tester_r 0 2024-03-22T12:56:59.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T12:57:05.972 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T12:57:05.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T12:57:05.973 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-22T12:57:06.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:06.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.230+0000 7f6019a00640 1 Processor -- start 2024-03-22T12:57:06.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.230+0000 7f6019a00640 1 -- start start 2024-03-22T12:57:06.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.231+0000 7f6019a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6014150f50 0x7f60140c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:06.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.231+0000 7f6019a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:06.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.232+0000 7f6019a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60140c3bf0 con 0x7f6014157410 2024-03-22T12:57:06.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.232+0000 7f6019a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60140c3760 con 0x7f6014150f50 2024-03-22T12:57:06.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.232+0000 7f6019a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6014071e70 con 0x7f60140c5c20 2024-03-22T12:57:06.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.232+0000 7f600be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:06.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.232+0000 7f6013400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6014157410 0x7f601406b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48780/0 (socket says 172.21.15.6:48780) 2024-03-22T12:57:06.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.232+0000 7f6013400640 1 -- 172.21.15.6:0/735230177 learned_addr learned my addr 172.21.15.6:0/735230177 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:06.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.232+0000 7f6012a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6014150f50 0x7f60140c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:06.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.235+0000 7f600be00640 1 -- 172.21.15.6:0/735230177 >> v2:172.21.15.6:3300/0 conn(0x7f6014150f50 msgr2=0x7f60140c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:06.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.235+0000 7f600be00640 1 --2- 172.21.15.6:0/735230177 >> v2:172.21.15.6:3300/0 conn(0x7f6014150f50 0x7f60140c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.235+0000 7f600be00640 1 -- 172.21.15.6:0/735230177 >> v1:172.21.15.92:6789/0 conn(0x7f6014157410 legacy=0x7f601406b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:06.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.235+0000 7f600be00640 1 -- 172.21.15.6:0/735230177 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6014072150 con 0x7f60140c5c20 2024-03-22T12:57:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.237+0000 7f600be00640 1 --2- 172.21.15.6:0/735230177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014156ed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0097a0 tx=0x7f5ffc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab62a1bb0385366 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:06.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.241+0000 7f6012000640 1 -- 172.21.15.6:0/735230177 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc034070 con 0x7f60140c5c20 2024-03-22T12:57:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.241+0000 7f6012000640 1 -- 172.21.15.6:0/735230177 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc030610 con 0x7f60140c5c20 2024-03-22T12:57:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.241+0000 7f6012000640 1 -- 172.21.15.6:0/735230177 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0309e0 con 0x7f60140c5c20 2024-03-22T12:57:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.241+0000 7f6019a00640 1 -- 172.21.15.6:0/735230177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 msgr2=0x7f6014156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.241+0000 7f6019a00640 1 --2- 172.21.15.6:0/735230177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014156ed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0097a0 tx=0x7f5ffc02b960 comp rx=0 tx=0).stop 2024-03-22T12:57:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.242+0000 7f6019a00640 1 -- 172.21.15.6:0/735230177 shutdown_connections 2024-03-22T12:57:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.242+0000 7f6019a00640 1 --2- 172.21.15.6:0/735230177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014156ed0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.242+0000 7f6019a00640 1 --2- 172.21.15.6:0/735230177 >> v2:172.21.15.6:3300/0 conn(0x7f6014150f50 0x7f60140c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.242+0000 7f6019a00640 1 -- 172.21.15.6:0/735230177 >> 172.21.15.6:0/735230177 conn(0x7f60140cb930 msgr2=0x7f60140cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.242+0000 7f6019a00640 1 -- 172.21.15.6:0/735230177 shutdown_connections 2024-03-22T12:57:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.242+0000 7f6019a00640 1 -- 172.21.15.6:0/735230177 wait complete. 2024-03-22T12:57:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.244+0000 7f6019a00640 1 Processor -- start 2024-03-22T12:57:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.244+0000 7f6019a00640 1 -- start start 2024-03-22T12:57:06.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.245+0000 7f6019a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:06.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.245+0000 7f6019a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6014150f50 0x7f601414efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:06.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.245+0000 7f6019a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6014157410 0x7f6014140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.245+0000 7f6019a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60140c33f0 con 0x7f6014157410 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.245+0000 7f6019a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60140c3110 con 0x7f6014150f50 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.245+0000 7f6019a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6014151330 con 0x7f60140c5c20 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.246+0000 7f6013400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6014157410 0x7f6014140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.246+0000 7f6012a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.246+0000 7f600be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6014150f50 0x7f601414efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.246+0000 7f6013400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6014157410 0x7f6014140040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55780/0 (socket says 172.21.15.6:55780) 2024-03-22T12:57:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.246+0000 7f6013400640 1 -- 172.21.15.6:0/2102705201 learned_addr learned my addr 172.21.15.6:0/2102705201 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.249+0000 7f6012a00640 1 -- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6014150f50 msgr2=0x7f601414efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.249+0000 7f6012a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6014150f50 0x7f601414efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.249+0000 7f6012a00640 1 -- 172.21.15.6:0/2102705201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6014157410 msgr2=0x7f6014140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.249+0000 7f6012a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6014157410 0x7f6014140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.249+0000 7f6012a00640 1 -- 172.21.15.6:0/2102705201 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60141ec180 con 0x7f60140c5c20 2024-03-22T12:57:06.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.252+0000 7f6012a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014142700 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6004009db0 tx=0x7f600400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:06.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.255+0000 7f600aa00640 1 -- 172.21.15.6:0/2102705201 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004018020 con 0x7f60140c5c20 2024-03-22T12:57:06.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.255+0000 7f600aa00640 1 -- 172.21.15.6:0/2102705201 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6004014480 con 0x7f60140c5c20 2024-03-22T12:57:06.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.255+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60141ec460 con 0x7f60140c5c20 2024-03-22T12:57:06.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.255+0000 7f600aa00640 1 -- 172.21.15.6:0/2102705201 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004014850 con 0x7f60140c5c20 2024-03-22T12:57:06.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.255+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60141eca20 con 0x7f60140c5c20 2024-03-22T12:57:06.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.257+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60140756c0 con 0x7f60140c5c20 2024-03-22T12:57:06.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.279+0000 7f600aa00640 1 -- 172.21.15.6:0/2102705201 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6004005d10 con 0x7f60140c5c20 2024-03-22T12:57:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.281+0000 7f600aa00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5ff006bc20 0x7f5ff006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.281+0000 7f600aa00640 1 -- 172.21.15.6:0/2102705201 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f600408c120 con 0x7f60140c5c20 2024-03-22T12:57:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.282+0000 7f600be00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5ff006bc20 0x7f5ff006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.283+0000 7f600be00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5ff006bc20 0x7f5ff006e0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc036000 tx=0x7f5ffc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.519+0000 7f600aa00640 1 -- 172.21.15.6:0/2102705201 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6004022030 con 0x7f60140c5c20 2024-03-22T12:57:06.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.732+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60140c2690 con 0x7f60140c5c20 2024-03-22T12:57:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.777+0000 7f600aa00640 1 -- 172.21.15.6:0/2102705201 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7f6004010040 con 0x7f60140c5c20 2024-03-22T12:57:06.777 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:06.777 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5ff006bc20 msgr2=0x7f5ff006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5ff006bc20 0x7f5ff006e0e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc036000 tx=0x7f5ffc03b040 comp rx=0 tx=0).stop 2024-03-22T12:57:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 msgr2=0x7f6014142700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014142700 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6004009db0 tx=0x7f600400c490 comp rx=0 tx=0).stop 2024-03-22T12:57:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 shutdown_connections 2024-03-22T12:57:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6014157410 0x7f6014140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5ff006bc20 0x7f5ff006e0e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6014150f50 0x7f601414efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 --2- 172.21.15.6:0/2102705201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60140c5c20 0x7f6014142700 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.779+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 >> 172.21.15.6:0/2102705201 conn(0x7f60140cb930 msgr2=0x7f60140cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.780+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 shutdown_connections 2024-03-22T12:57:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.780+0000 7f6019a00640 1 -- 172.21.15.6:0/2102705201 wait complete. 2024-03-22T12:57:06.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:06.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.997+0000 7fdbe5e00640 1 Processor -- start 2024-03-22T12:57:06.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.998+0000 7fdbe5e00640 1 -- start start 2024-03-22T12:57:06.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.999+0000 7fdbe5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdbe00cc380 0x7fdbe006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:06.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.999+0000 7fdbe5e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe015b240 con 0x7fdbe00ce8a0 2024-03-22T12:57:06.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.999+0000 7fdbe5e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00cee00 con 0x7fdbe015ae30 2024-03-22T12:57:06.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:06.999+0000 7fdbe5e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe00cf0c0 con 0x7fdbe00cc380 2024-03-22T12:57:07.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.000+0000 7fdbe4a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdbe00cc380 0x7fdbe006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.000+0000 7fdbd7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdbe00ce8a0 0x7fdbe00cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58092/0 (socket says 172.21.15.6:58092) 2024-03-22T12:57:07.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.000+0000 7fdbd7e00640 1 -- 172.21.15.6:0/4183062229 learned_addr learned my addr 172.21.15.6:0/4183062229 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:07.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.004+0000 7fdbdfe00640 1 -- 172.21.15.6:0/4183062229 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39851471 0 0) 0x7fdbe00cee00 con 0x7fdbe015ae30 2024-03-22T12:57:07.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.004+0000 7fdbdfe00640 1 -- 172.21.15.6:0/4183062229 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc4003520 con 0x7fdbe015ae30 2024-03-22T12:57:07.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.004+0000 7fdbdfe00640 1 -- 172.21.15.6:0/4183062229 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710281294 0 0) 0x7fdbe015b240 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.004+0000 7fdbdfe00640 1 -- 172.21.15.6:0/4183062229 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe00cee00 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.007+0000 7fdbe4a00640 1 -- 172.21.15.6:0/4183062229 >> v1:172.21.15.92:6789/0 conn(0x7fdbe015ae30 legacy=0x7fdbe0164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.007+0000 7fdbe4a00640 1 -- 172.21.15.6:0/4183062229 >> v1:172.21.15.6:6789/0 conn(0x7fdbe00ce8a0 legacy=0x7fdbe00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.007+0000 7fdbe4a00640 1 -- 172.21.15.6:0/4183062229 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00cf3a0 con 0x7fdbe00cc380 2024-03-22T12:57:07.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.012+0000 7fdbe4a00640 1 --2- 172.21.15.6:0/4183062229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe00cc380 0x7fdbe006c230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00097a0 tx=0x7fdbd002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab4b598912ce7ff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:07.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.020+0000 7fdbdfe00640 1 -- 172.21.15.6:0/4183062229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0034070 con 0x7fdbe00cc380 2024-03-22T12:57:07.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.020+0000 7fdbdfe00640 1 -- 172.21.15.6:0/4183062229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd0030610 con 0x7fdbe00cc380 2024-03-22T12:57:07.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.020+0000 7fdbdfe00640 1 -- 172.21.15.6:0/4183062229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00309e0 con 0x7fdbe00cc380 2024-03-22T12:57:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.021+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4183062229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe00cc380 msgr2=0x7fdbe006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.021+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4183062229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe00cc380 0x7fdbe006c230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00097a0 tx=0x7fdbd002b960 comp rx=0 tx=0).stop 2024-03-22T12:57:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.021+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4183062229 shutdown_connections 2024-03-22T12:57:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.021+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4183062229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe00cc380 0x7fdbe006c230 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.021+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4183062229 >> 172.21.15.6:0/4183062229 conn(0x7fdbe00b1480 msgr2=0x7fdbe00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.021+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4183062229 shutdown_connections 2024-03-22T12:57:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.021+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4183062229 wait complete. 2024-03-22T12:57:07.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.023+0000 7fdbe5e00640 1 Processor -- start 2024-03-22T12:57:07.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.023+0000 7fdbe5e00640 1 -- start start 2024-03-22T12:57:07.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.024+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdbe00cc380 0x7fdbe01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.024+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe00ce8a0 0x7fdbe0156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.024+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe015ae30 0x7fdbe01e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.024+0000 7fdbe5e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe015a910 con 0x7fdbe015ae30 2024-03-22T12:57:07.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.024+0000 7fdbe5e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe015a630 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.024+0000 7fdbe5e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe01549d0 con 0x7fdbe00cc380 2024-03-22T12:57:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.025+0000 7fdbe4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdbe00cc380 0x7fdbe01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.025+0000 7fdbd7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe00ce8a0 0x7fdbe0156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.025+0000 7fdbe4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdbe00cc380 0x7fdbe01566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57252/0 (socket says 172.21.15.6:57252) 2024-03-22T12:57:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.025+0000 7fdbe4a00640 1 -- 172.21.15.6:0/4281567695 learned_addr learned my addr 172.21.15.6:0/4281567695 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.029+0000 7fdbd7e00640 1 -- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdbe00cc380 msgr2=0x7fdbe01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.029+0000 7fdbd7e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdbe00cc380 0x7fdbe01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.029+0000 7fdbd7e00640 1 -- 172.21.15.6:0/4281567695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe015ae30 msgr2=0x7fdbe01e4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.029+0000 7fdbd7e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe015ae30 0x7fdbe01e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.029+0000 7fdbd7e00640 1 -- 172.21.15.6:0/4281567695 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00c4390 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.029+0000 7fdbe4a00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdbe00cc380 0x7fdbe01566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:57:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.031+0000 7fdbd7e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe00ce8a0 0x7fdbe0156be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c430 tx=0x7fdbc800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.036+0000 7fdbdd600640 1 -- 172.21.15.6:0/4281567695 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8017070 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.036+0000 7fdbdd600640 1 -- 172.21.15.6:0/4281567695 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc8005ce0 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.036+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe01e3a00 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.036+0000 7fdbdd600640 1 -- 172.21.15.6:0/4281567695 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc80072f0 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.036+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe00c3640 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.038+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe0156380 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.294+0000 7fdbdd600640 1 -- 172.21.15.6:0/4281567695 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdbc801a070 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.296+0000 7fdbdd600640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbbc06bc20 0x7fdbbc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.297+0000 7fdbdd600640 1 -- 172.21.15.6:0/4281567695 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fdbc808a500 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.297+0000 7fdbe4a00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbbc06bc20 0x7fdbbc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.298+0000 7fdbe4a00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbbc06bc20 0x7fdbbc06e0e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009560 tx=0x7fdbd003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:07.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.306+0000 7fdbdd600640 1 -- 172.21.15.6:0/4281567695 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdbe0156380 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.532+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbe0075650 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.577+0000 7fdbdd600640 1 -- 172.21.15.6:0/4281567695 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7fdbc8021020 con 0x7fdbe00ce8a0 2024-03-22T12:57:07.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:07.577 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:07.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbbc06bc20 msgr2=0x7fdbbc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbbc06bc20 0x7fdbbc06e0e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009560 tx=0x7fdbd003b040 comp rx=0 tx=0).stop 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe00ce8a0 msgr2=0x7fdbe0156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe00ce8a0 0x7fdbe0156be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c430 tx=0x7fdbc800c8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 shutdown_connections 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbe015ae30 0x7fdbe01e4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbbc06bc20 0x7fdbbc06e0e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe00ce8a0 0x7fdbe0156be0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 --2- 172.21.15.6:0/4281567695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdbe00cc380 0x7fdbe01566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.579+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 >> 172.21.15.6:0/4281567695 conn(0x7fdbe00b1480 msgr2=0x7fdbe00cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.580+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 shutdown_connections 2024-03-22T12:57:07.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.580+0000 7fdbe5e00640 1 -- 172.21.15.6:0/4281567695 wait complete. 2024-03-22T12:57:07.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-22T12:57:07.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.791+0000 7f21c3e00640 1 Processor -- start 2024-03-22T12:57:07.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.791+0000 7f21c3e00640 1 -- start start 2024-03-22T12:57:07.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.792+0000 7f21c3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f21c4152090 0x7f21c406c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.793+0000 7f21c3e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f21c406c040 0x7f21c415b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.793+0000 7f21c3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c406b410 con 0x7f21c40c5970 2024-03-22T12:57:07.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.793+0000 7f21c3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c406ba00 con 0x7f21c4152090 2024-03-22T12:57:07.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.793+0000 7f21c3e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c406bce0 con 0x7f21c406c040 2024-03-22T12:57:07.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.793+0000 7f21c2000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f21c406c040 0x7f21c415b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.794+0000 7f21c2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f21c4152090 0x7f21c406c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.794+0000 7f21c2000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f21c406c040 0x7f21c415b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52306/0 (socket says 172.21.15.6:52306) 2024-03-22T12:57:07.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.794+0000 7f21c2000640 1 -- 172.21.15.6:0/3697458977 learned_addr learned my addr 172.21.15.6:0/3697458977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:07.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.796+0000 7f21c2000640 1 -- 172.21.15.6:0/3697458977 >> v2:172.21.15.6:3300/0 conn(0x7f21c4152090 msgr2=0x7f21c406c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:07.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.796+0000 7f21c2000640 1 --2- 172.21.15.6:0/3697458977 >> v2:172.21.15.6:3300/0 conn(0x7f21c4152090 0x7f21c406c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.796+0000 7f21c2000640 1 -- 172.21.15.6:0/3697458977 >> v1:172.21.15.6:6789/0 conn(0x7f21c40c5970 legacy=0x7f21c40cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.796+0000 7f21c2000640 1 -- 172.21.15.6:0/3697458977 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c40c6c60 con 0x7f21c406c040 2024-03-22T12:57:07.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.798+0000 7f21c2000640 1 --2- 172.21.15.6:0/3697458977 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c406c040 0x7f21c415b490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21c406f8e0 tx=0x7f21b002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=533ed87a38f29ee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c1600640 1 -- 172.21.15.6:0/3697458977 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b0034070 con 0x7f21c406c040 2024-03-22T12:57:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c1600640 1 -- 172.21.15.6:0/3697458977 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b0030860 con 0x7f21c406c040 2024-03-22T12:57:07.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c1600640 1 -- 172.21.15.6:0/3697458977 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b0030c30 con 0x7f21c406c040 2024-03-22T12:57:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c3e00640 1 -- 172.21.15.6:0/3697458977 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c406c040 msgr2=0x7f21c415b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c3e00640 1 --2- 172.21.15.6:0/3697458977 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c406c040 0x7f21c415b490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21c406f8e0 tx=0x7f21b002b9a0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c3e00640 1 -- 172.21.15.6:0/3697458977 shutdown_connections 2024-03-22T12:57:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c3e00640 1 --2- 172.21.15.6:0/3697458977 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c406c040 0x7f21c415b490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c3e00640 1 --2- 172.21.15.6:0/3697458977 >> v2:172.21.15.6:3300/0 conn(0x7f21c4152090 0x7f21c406c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.802+0000 7f21c3e00640 1 -- 172.21.15.6:0/3697458977 >> 172.21.15.6:0/3697458977 conn(0x7f21c406db00 msgr2=0x7f21c406dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.803+0000 7f21c3e00640 1 -- 172.21.15.6:0/3697458977 shutdown_connections 2024-03-22T12:57:07.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.803+0000 7f21c3e00640 1 -- 172.21.15.6:0/3697458977 wait complete. 2024-03-22T12:57:07.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.804+0000 7f21c3e00640 1 Processor -- start 2024-03-22T12:57:07.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.804+0000 7f21c3e00640 1 -- start start 2024-03-22T12:57:07.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.805+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c406c040 0x7f21c4147e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.805+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c40c5970 0x7f21c414b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.805+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c4152090 0x7f21c414bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.805+0000 7f21c3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40c9be0 con 0x7f21c4152090 2024-03-22T12:57:07.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.805+0000 7f21c3e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40c7220 con 0x7f21c40c5970 2024-03-22T12:57:07.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.805+0000 7f21c3e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40c7740 con 0x7f21c406c040 2024-03-22T12:57:07.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.806+0000 7f21c2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c40c5970 0x7f21c414b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.806+0000 7f21c2a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c406c040 0x7f21c4147e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.806+0000 7f21c3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c4152090 0x7f21c414bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.806+0000 7f21c2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c40c5970 0x7f21c414b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52312/0 (socket says 172.21.15.6:52312) 2024-03-22T12:57:07.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.806+0000 7f21c2000640 1 -- 172.21.15.6:0/2918381819 learned_addr learned my addr 172.21.15.6:0/2918381819 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:07.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.809+0000 7f21c2000640 1 -- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c406c040 msgr2=0x7f21c4147e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.809+0000 7f21c2000640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c406c040 0x7f21c4147e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.809+0000 7f21c2000640 1 -- 172.21.15.6:0/2918381819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c4152090 msgr2=0x7f21c414bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:07.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.809+0000 7f21c2000640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c4152090 0x7f21c414bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:07.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.809+0000 7f21c2000640 1 -- 172.21.15.6:0/2918381819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c41f4310 con 0x7f21c40c5970 2024-03-22T12:57:07.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.812+0000 7f21c2000640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c40c5970 0x7f21c414b590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f21b0009960 tx=0x7f21b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:07.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.815+0000 7f21bb400640 1 -- 172.21.15.6:0/2918381819 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b0034040 con 0x7f21c40c5970 2024-03-22T12:57:07.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.815+0000 7f21bb400640 1 -- 172.21.15.6:0/2918381819 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b0039d90 con 0x7f21c40c5970 2024-03-22T12:57:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.815+0000 7f21bb400640 1 -- 172.21.15.6:0/2918381819 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b0043610 con 0x7f21c40c5970 2024-03-22T12:57:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.815+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c414c1a0 con 0x7f21c40c5970 2024-03-22T12:57:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.816+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c4155070 con 0x7f21c40c5970 2024-03-22T12:57:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.817+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c4075650 con 0x7f21c40c5970 2024-03-22T12:57:07.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.837+0000 7f21bb400640 1 -- 172.21.15.6:0/2918381819 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f21b0037070 con 0x7f21c40c5970 2024-03-22T12:57:07.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.838+0000 7f21bb400640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a406bc20 0x7f21a406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:07.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.839+0000 7f21c2a00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a406bc20 0x7f21a406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:07.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:07.839+0000 7f21c2a00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a406bc20 0x7f21a406e0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009aa0 tx=0x7f21b4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:08.068+0000 7f21bb400640 1 -- 172.21.15.6:0/2918381819 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f21b0079810 con 0x7f21c40c5970 2024-03-22T12:57:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:08.080+0000 7f21bb400640 1 -- 172.21.15.6:0/2918381819 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f21b003e070 con 0x7f21c40c5970 2024-03-22T12:57:08.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:08.336+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f21c40c9750 con 0x7f21c40c5970 2024-03-22T12:57:09.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.559+0000 7f21bb400640 1 -- 172.21.15.6:0/2918381819 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f21b007b770 con 0x7f21c40c5970 2024-03-22T12:57:09.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a406bc20 msgr2=0x7f21a406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:09.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a406bc20 0x7f21a406e0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009aa0 tx=0x7f21b4009190 comp rx=0 tx=0).stop 2024-03-22T12:57:09.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c40c5970 msgr2=0x7f21c414b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:09.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c40c5970 0x7f21c414b590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f21b0009960 tx=0x7f21b00046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 shutdown_connections 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c4152090 0x7f21c414bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a406bc20 0x7f21a406e0e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c40c5970 0x7f21c414b590 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 --2- 172.21.15.6:0/2918381819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c406c040 0x7f21c4147e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.562+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 >> 172.21.15.6:0/2918381819 conn(0x7f21c406db00 msgr2=0x7f21c4149fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.563+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 shutdown_connections 2024-03-22T12:57:09.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.563+0000 7f21c3e00640 1 -- 172.21.15.6:0/2918381819 wait complete. 2024-03-22T12:57:09.580 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2024-03-22T12:57:09.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-22T12:57:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.776+0000 7f3725c00640 1 Processor -- start 2024-03-22T12:57:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.776+0000 7f3725c00640 1 -- start start 2024-03-22T12:57:09.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.778+0000 7f3725c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37200c5570 0x7f37200ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:09.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.778+0000 7f3725c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37200c5270 con 0x7f37201547f0 2024-03-22T12:57:09.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.778+0000 7f3725c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372015e380 con 0x7f37200c4b90 2024-03-22T12:57:09.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.778+0000 7f3725c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37200c5020 con 0x7f37200c5570 2024-03-22T12:57:09.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.779+0000 7f371ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f37201547f0 0x7f37200c6450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58118/0 (socket says 172.21.15.6:58118) 2024-03-22T12:57:09.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.779+0000 7f371ea00640 1 -- 172.21.15.6:0/1979673891 learned_addr learned my addr 172.21.15.6:0/1979673891 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:09.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.779+0000 7f371f400640 1 --2- 172.21.15.6:0/1979673891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37200c5570 0x7f37200ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:09.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.781+0000 7f371d600640 1 -- 172.21.15.6:0/1979673891 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199737211 0 0) 0x7f372015e380 con 0x7f37200c4b90 2024-03-22T12:57:09.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.782+0000 7f371d600640 1 -- 172.21.15.6:0/1979673891 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3704003520 con 0x7f37200c4b90 2024-03-22T12:57:09.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.782+0000 7f371f400640 1 -- 172.21.15.6:0/1979673891 >> v1:172.21.15.92:6789/0 conn(0x7f37200c4b90 legacy=0x7f37200cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:09.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.782+0000 7f371f400640 1 -- 172.21.15.6:0/1979673891 >> v1:172.21.15.6:6789/0 conn(0x7f37201547f0 legacy=0x7f37200c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:09.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.782+0000 7f371f400640 1 -- 172.21.15.6:0/1979673891 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372015e660 con 0x7f37200c5570 2024-03-22T12:57:09.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.785+0000 7f371f400640 1 --2- 172.21.15.6:0/1979673891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37200c5570 0x7f37200ccd30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37200c51a0 tx=0x7f371802be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ea10e6125054a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:09.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.787+0000 7f371d600640 1 -- 172.21.15.6:0/1979673891 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718039070 con 0x7f37200c5570 2024-03-22T12:57:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.788+0000 7f371d600640 1 -- 172.21.15.6:0/1979673891 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371802cc30 con 0x7f37200c5570 2024-03-22T12:57:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.788+0000 7f371d600640 1 -- 172.21.15.6:0/1979673891 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718034440 con 0x7f37200c5570 2024-03-22T12:57:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.789+0000 7f3725c00640 1 -- 172.21.15.6:0/1979673891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37200c5570 msgr2=0x7f37200ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.789+0000 7f3725c00640 1 --2- 172.21.15.6:0/1979673891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37200c5570 0x7f37200ccd30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37200c51a0 tx=0x7f371802be80 comp rx=0 tx=0).stop 2024-03-22T12:57:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.789+0000 7f3725c00640 1 -- 172.21.15.6:0/1979673891 shutdown_connections 2024-03-22T12:57:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.789+0000 7f3725c00640 1 --2- 172.21.15.6:0/1979673891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37200c5570 0x7f37200ccd30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.789+0000 7f3725c00640 1 -- 172.21.15.6:0/1979673891 >> 172.21.15.6:0/1979673891 conn(0x7f3720066e40 msgr2=0x7f3720067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.789+0000 7f3725c00640 1 -- 172.21.15.6:0/1979673891 shutdown_connections 2024-03-22T12:57:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.789+0000 7f3725c00640 1 -- 172.21.15.6:0/1979673891 wait complete. 2024-03-22T12:57:09.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.791+0000 7f3725c00640 1 Processor -- start 2024-03-22T12:57:09.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.791+0000 7f3725c00640 1 -- start start 2024-03-22T12:57:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f3725c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37200c4b90 0x7f3720158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f3725c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37200c5570 0x7f37201f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f3725c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37201547f0 0x7f37201f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f3725c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37200cd270 con 0x7f37200c4b90 2024-03-22T12:57:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f3725c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37200c5f30 con 0x7f37200c5570 2024-03-22T12:57:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f3725c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37200ca5d0 con 0x7f37201547f0 2024-03-22T12:57:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f371ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37200c4b90 0x7f3720158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f371f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37201547f0 0x7f37201f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.792+0000 7f371e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37200c5570 0x7f37201f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.793+0000 7f371f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37201547f0 0x7f37201f14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57286/0 (socket says 172.21.15.6:57286) 2024-03-22T12:57:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.793+0000 7f371f400640 1 -- 172.21.15.6:0/1967245910 learned_addr learned my addr 172.21.15.6:0/1967245910 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:09.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.796+0000 7f371f400640 1 -- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37200c5570 msgr2=0x7f37201f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:09.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.796+0000 7f371f400640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37200c5570 0x7f37201f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.796+0000 7f371f400640 1 -- 172.21.15.6:0/1967245910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37200c4b90 msgr2=0x7f3720158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:09.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.796+0000 7f371f400640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37200c4b90 0x7f3720158190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:09.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.796+0000 7f371f400640 1 -- 172.21.15.6:0/1967245910 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3720159610 con 0x7f37201547f0 2024-03-22T12:57:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.798+0000 7f371f400640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37201547f0 0x7f37201f14d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3718009eb0 tx=0x7f3718004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.802+0000 7f3716a00640 1 -- 172.21.15.6:0/1967245910 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718039050 con 0x7f37201547f0 2024-03-22T12:57:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.802+0000 7f3716a00640 1 -- 172.21.15.6:0/1967245910 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3718045070 con 0x7f37201547f0 2024-03-22T12:57:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.802+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37201598f0 con 0x7f37201547f0 2024-03-22T12:57:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.802+0000 7f3716a00640 1 -- 172.21.15.6:0/1967245910 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718009450 con 0x7f37201547f0 2024-03-22T12:57:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.803+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3720159eb0 con 0x7f37201547f0 2024-03-22T12:57:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:09.804+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37200757d0 con 0x7f37201547f0 2024-03-22T12:57:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.050+0000 7f3716a00640 1 -- 172.21.15.6:0/1967245910 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f371804a460 con 0x7f37201547f0 2024-03-22T12:57:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.052+0000 7f3716a00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f370006bc50 0x7f370006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.053+0000 7f3716a00640 1 -- 172.21.15.6:0/1967245910 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f37180ae530 con 0x7f37201547f0 2024-03-22T12:57:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.053+0000 7f371ea00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f370006bc50 0x7f370006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.054+0000 7f371ea00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f370006bc50 0x7f370006e110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3710007fd0 tx=0x7f3710007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.064+0000 7f3716a00640 1 -- 172.21.15.6:0/1967245910 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3718004820 con 0x7f37201547f0 2024-03-22T12:57:10.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.316+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f37201518f0 con 0x7f37201547f0 2024-03-22T12:57:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.826+0000 7f3716a00640 1 -- 172.21.15.6:0/1967245910 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v9) v1 ==== 105+0+0 (secure 0 0 0) 0x7f371807c5e0 con 0x7f37201547f0 2024-03-22T12:57:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f370006bc50 msgr2=0x7f370006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f370006bc50 0x7f370006e110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3710007fd0 tx=0x7f3710007e80 comp rx=0 tx=0).stop 2024-03-22T12:57:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37201547f0 msgr2=0x7f37201f14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37201547f0 0x7f37201f14d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3718009eb0 tx=0x7f3718004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 shutdown_connections 2024-03-22T12:57:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37201547f0 0x7f37201f14d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f370006bc50 0x7f370006e110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37200c5570 0x7f37201f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 --2- 172.21.15.6:0/1967245910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37200c4b90 0x7f3720158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 >> 172.21.15.6:0/1967245910 conn(0x7f3720066e40 msgr2=0x7f37200cec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 shutdown_connections 2024-03-22T12:57:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:10.830+0000 7f3725c00640 1 -- 172.21.15.6:0/1967245910 wait complete. 2024-03-22T12:57:10.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-22T12:57:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.054+0000 7fb216c00640 1 Processor -- start 2024-03-22T12:57:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.054+0000 7fb216c00640 1 -- start start 2024-03-22T12:57:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.056+0000 7fb216c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2100c0ea0 0x7fb210071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.056+0000 7fb216c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb210159070 con 0x7fb210158c20 2024-03-22T12:57:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.056+0000 7fb216c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2100c1450 con 0x7fb210152280 2024-03-22T12:57:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.056+0000 7fb216c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb210158790 con 0x7fb2100c0ea0 2024-03-22T12:57:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.057+0000 7fb20f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2100c0ea0 0x7fb210071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.057+0000 7fb20f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2100c0ea0 0x7fb210071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57294/0 (socket says 172.21.15.6:57294) 2024-03-22T12:57:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.057+0000 7fb20f400640 1 -- 172.21.15.6:0/3685624057 learned_addr learned my addr 172.21.15.6:0/3685624057 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:11.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.060+0000 7fb20f400640 1 -- 172.21.15.6:0/3685624057 >> v1:172.21.15.92:6789/0 conn(0x7fb210152280 legacy=0x7fb21006b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:11.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.060+0000 7fb20f400640 1 -- 172.21.15.6:0/3685624057 >> v1:172.21.15.6:6789/0 conn(0x7fb210158c20 legacy=0x7fb2100c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:11.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.060+0000 7fb20f400640 1 -- 172.21.15.6:0/3685624057 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100c1730 con 0x7fb2100c0ea0 2024-03-22T12:57:11.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.063+0000 7fb20f400640 1 --2- 172.21.15.6:0/3685624057 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2100c0ea0 0x7fb210071920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80098a0 tx=0x7fb1f802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8458e081bc957d3b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:11.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.068+0000 7fb20ea00640 1 -- 172.21.15.6:0/3685624057 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8035020 con 0x7fb2100c0ea0 2024-03-22T12:57:11.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.068+0000 7fb20ea00640 1 -- 172.21.15.6:0/3685624057 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f8031640 con 0x7fb2100c0ea0 2024-03-22T12:57:11.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.068+0000 7fb20ea00640 1 -- 172.21.15.6:0/3685624057 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f802d040 con 0x7fb2100c0ea0 2024-03-22T12:57:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.069+0000 7fb216c00640 1 -- 172.21.15.6:0/3685624057 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2100c0ea0 msgr2=0x7fb210071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.069+0000 7fb216c00640 1 --2- 172.21.15.6:0/3685624057 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2100c0ea0 0x7fb210071920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80098a0 tx=0x7fb1f802b8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.069+0000 7fb216c00640 1 -- 172.21.15.6:0/3685624057 shutdown_connections 2024-03-22T12:57:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.069+0000 7fb216c00640 1 --2- 172.21.15.6:0/3685624057 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2100c0ea0 0x7fb210071920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.069+0000 7fb216c00640 1 -- 172.21.15.6:0/3685624057 >> 172.21.15.6:0/3685624057 conn(0x7fb21006e310 msgr2=0x7fb21006f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.069+0000 7fb216c00640 1 -- 172.21.15.6:0/3685624057 shutdown_connections 2024-03-22T12:57:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.069+0000 7fb216c00640 1 -- 172.21.15.6:0/3685624057 wait complete. 2024-03-22T12:57:11.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.071+0000 7fb216c00640 1 Processor -- start 2024-03-22T12:57:11.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.071+0000 7fb216c00640 1 -- start start 2024-03-22T12:57:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.072+0000 7fb216c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2100c0ea0 0x7fb21015a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.072+0000 7fb216c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb210152280 0x7fb210154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.072+0000 7fb216c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb210158c20 0x7fb210144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.072+0000 7fb216c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2101580d0 con 0x7fb210152280 2024-03-22T12:57:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.072+0000 7fb216c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb210158420 con 0x7fb2100c0ea0 2024-03-22T12:57:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.072+0000 7fb216c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb210152660 con 0x7fb210158c20 2024-03-22T12:57:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.073+0000 7fb20fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2100c0ea0 0x7fb21015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.073+0000 7fb20f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb210152280 0x7fb210154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.073+0000 7fb20fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2100c0ea0 0x7fb21015a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52322/0 (socket says 172.21.15.6:52322) 2024-03-22T12:57:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.073+0000 7fb20fe00640 1 -- 172.21.15.6:0/3367339975 learned_addr learned my addr 172.21.15.6:0/3367339975 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:11.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.076+0000 7fb214c00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb210158c20 0x7fb210144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.077+0000 7fb20f400640 1 -- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb210158c20 msgr2=0x7fb210144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.077+0000 7fb20f400640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb210158c20 0x7fb210144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.077+0000 7fb20f400640 1 -- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2100c0ea0 msgr2=0x7fb21015a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.077+0000 7fb20f400640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2100c0ea0 0x7fb21015a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.077+0000 7fb20f400640 1 -- 172.21.15.6:0/3367339975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2100c3410 con 0x7fb210152280 2024-03-22T12:57:11.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.080+0000 7fb20f400640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb210152280 0x7fb210154740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb210070130 tx=0x7fb1f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:11.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.088+0000 7fb207e00640 1 -- 172.21.15.6:0/3367339975 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f803ac60 con 0x7fb210152280 2024-03-22T12:57:11.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.088+0000 7fb207e00640 1 -- 172.21.15.6:0/3367339975 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f803ae40 con 0x7fb210152280 2024-03-22T12:57:11.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.088+0000 7fb207e00640 1 -- 172.21.15.6:0/3367339975 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8044630 con 0x7fb210152280 2024-03-22T12:57:11.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.088+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2101594a0 con 0x7fb210152280 2024-03-22T12:57:11.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.088+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2100c3990 con 0x7fb210152280 2024-03-22T12:57:11.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.089+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2100757d0 con 0x7fb210152280 2024-03-22T12:57:11.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.115+0000 7fb207e00640 1 -- 172.21.15.6:0/3367339975 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb1f804b060 con 0x7fb210152280 2024-03-22T12:57:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.116+0000 7fb207e00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb1ec06bc20 0x7fb1ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.117+0000 7fb20fe00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb1ec06bc20 0x7fb1ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:11.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.117+0000 7fb20fe00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb1ec06bc20 0x7fb1ec06e0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb200007fd0 tx=0x7fb200009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:11.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.119+0000 7fb207e00640 1 -- 172.21.15.6:0/3367339975 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fb1f80315d0 con 0x7fb210152280 2024-03-22T12:57:11.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.427+0000 7fb207e00640 1 -- 172.21.15.6:0/3367339975 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb1f803f070 con 0x7fb210152280 2024-03-22T12:57:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:11.693+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb210072f70 con 0x7fb210152280 2024-03-22T12:57:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.105+0000 7fb207e00640 1 -- 172.21.15.6:0/3367339975 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb1f807aef0 con 0x7fb210152280 2024-03-22T12:57:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb1ec06bc20 msgr2=0x7fb1ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb1ec06bc20 0x7fb1ec06e0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb200007fd0 tx=0x7fb200009040 comp rx=0 tx=0).stop 2024-03-22T12:57:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb210152280 msgr2=0x7fb210154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb210152280 0x7fb210154740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb210070130 tx=0x7fb1f80046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 shutdown_connections 2024-03-22T12:57:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb210158c20 0x7fb210144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb210152280 0x7fb210154740 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb1ec06bc20 0x7fb1ec06e0e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 --2- 172.21.15.6:0/3367339975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2100c0ea0 0x7fb21015a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 >> 172.21.15.6:0/3367339975 conn(0x7fb21006e310 msgr2=0x7fb21006b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 shutdown_connections 2024-03-22T12:57:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.110+0000 7fb216c00640 1 -- 172.21.15.6:0/3367339975 wait complete. 2024-03-22T12:57:12.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-22T12:57:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.332+0000 7f1fc2400640 1 Processor -- start 2024-03-22T12:57:12.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.332+0000 7f1fc2400640 1 -- start start 2024-03-22T12:57:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.334+0000 7f1fc2400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc06c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.334+0000 7f1fc2400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1fbc066e40 0x7f1fbc15a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.334+0000 7f1fc2400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc06ffa0 con 0x7f1fbc154730 2024-03-22T12:57:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.334+0000 7f1fc2400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc06b080 con 0x7f1fbc066e40 2024-03-22T12:57:12.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.334+0000 7f1fc2400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc06b2d0 con 0x7f1fbc154350 2024-03-22T12:57:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.335+0000 7f1fbaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1fbc154730 0x7f1fbc06f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58134/0 (socket says 172.21.15.6:58134) 2024-03-22T12:57:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.335+0000 7f1fbaa00640 1 -- 172.21.15.6:0/712043327 learned_addr learned my addr 172.21.15.6:0/712043327 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:12.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.335+0000 7f1fbb400640 1 --2- 172.21.15.6:0/712043327 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc06c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.338+0000 7f1fbb400640 1 -- 172.21.15.6:0/712043327 >> v2:172.21.15.6:3300/0 conn(0x7f1fbc066e40 msgr2=0x7f1fbc15a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.338+0000 7f1fbb400640 1 --2- 172.21.15.6:0/712043327 >> v2:172.21.15.6:3300/0 conn(0x7f1fbc066e40 0x7f1fbc15a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.338+0000 7f1fbb400640 1 -- 172.21.15.6:0/712043327 >> v1:172.21.15.6:6789/0 conn(0x7f1fbc154730 legacy=0x7f1fbc06f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.338+0000 7f1fbb400640 1 -- 172.21.15.6:0/712043327 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc06b5b0 con 0x7f1fbc154350 2024-03-22T12:57:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.340+0000 7f1fbb400640 1 --2- 172.21.15.6:0/712043327 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc06c650 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097a0 tx=0x7f1fac02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb4451fd77b82f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.343+0000 7f1fba000640 1 -- 172.21.15.6:0/712043327 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac034070 con 0x7f1fbc154350 2024-03-22T12:57:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.343+0000 7f1fba000640 1 -- 172.21.15.6:0/712043327 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac030610 con 0x7f1fbc154350 2024-03-22T12:57:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.343+0000 7f1fba000640 1 -- 172.21.15.6:0/712043327 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac0309e0 con 0x7f1fbc154350 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 -- 172.21.15.6:0/712043327 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 msgr2=0x7f1fbc06c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 --2- 172.21.15.6:0/712043327 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc06c650 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097a0 tx=0x7f1fac02b960 comp rx=0 tx=0).stop 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 -- 172.21.15.6:0/712043327 shutdown_connections 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 --2- 172.21.15.6:0/712043327 >> v2:172.21.15.6:3300/0 conn(0x7f1fbc066e40 0x7f1fbc15a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 --2- 172.21.15.6:0/712043327 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc06c650 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 -- 172.21.15.6:0/712043327 >> 172.21.15.6:0/712043327 conn(0x7f1fbc0c8e00 msgr2=0x7f1fbc0c9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 -- 172.21.15.6:0/712043327 shutdown_connections 2024-03-22T12:57:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.344+0000 7f1fc2400640 1 -- 172.21.15.6:0/712043327 wait complete. 2024-03-22T12:57:12.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.346+0000 7f1fc2400640 1 Processor -- start 2024-03-22T12:57:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.346+0000 7f1fc2400640 1 -- start start 2024-03-22T12:57:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.347+0000 7f1fc2400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc066e40 0x7f1fbc0cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.347+0000 7f1fc2400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.347+0000 7f1fc2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fbc154730 0x7f1fbc15a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.347+0000 7f1fc2400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc06be50 con 0x7f1fbc154730 2024-03-22T12:57:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.347+0000 7f1fc2400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc06bb70 con 0x7f1fbc066e40 2024-03-22T12:57:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.347+0000 7f1fc2400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc06c130 con 0x7f1fbc154350 2024-03-22T12:57:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.348+0000 7f1fbbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fbc154730 0x7f1fbc15a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.348+0000 7f1fbaa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.348+0000 7f1fbb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc066e40 0x7f1fbc0cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.348+0000 7f1fbbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fbc154730 0x7f1fbc15a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55866/0 (socket says 172.21.15.6:55866) 2024-03-22T12:57:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.348+0000 7f1fbbe00640 1 -- 172.21.15.6:0/4254221950 learned_addr learned my addr 172.21.15.6:0/4254221950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:12.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.351+0000 7f1fbaa00640 1 -- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc066e40 msgr2=0x7f1fbc0cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:12.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.351+0000 7f1fbaa00640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc066e40 0x7f1fbc0cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.351+0000 7f1fbaa00640 1 -- 172.21.15.6:0/4254221950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fbc154730 msgr2=0x7f1fbc15a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:12.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.351+0000 7f1fbaa00640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fbc154730 0x7f1fbc15a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:12.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.351+0000 7f1fbaa00640 1 -- 172.21.15.6:0/4254221950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc14f770 con 0x7f1fbc154350 2024-03-22T12:57:12.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.354+0000 7f1fbaa00640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc14c960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc154b10 tx=0x7f1fa800c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.357+0000 7f1fb3400640 1 -- 172.21.15.6:0/4254221950 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8017070 con 0x7f1fbc154350 2024-03-22T12:57:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.357+0000 7f1fb3400640 1 -- 172.21.15.6:0/4254221950 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa8005ce0 con 0x7f1fbc154350 2024-03-22T12:57:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.357+0000 7f1fb3400640 1 -- 172.21.15.6:0/4254221950 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8007310 con 0x7f1fbc154350 2024-03-22T12:57:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.357+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc148de0 con 0x7f1fbc154350 2024-03-22T12:57:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.357+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc0cb310 con 0x7f1fbc154350 2024-03-22T12:57:12.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.358+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc0757d0 con 0x7f1fbc154350 2024-03-22T12:57:12.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.378+0000 7f1fb3400640 1 -- 172.21.15.6:0/4254221950 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1fa801a070 con 0x7f1fbc154350 2024-03-22T12:57:12.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.379+0000 7f1fb3400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9c06bc20 0x7f1f9c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:12.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.379+0000 7f1fbb400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9c06bc20 0x7f1f9c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:12.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.380+0000 7f1fb3400640 1 -- 172.21.15.6:0/4254221950 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f1fa8056920 con 0x7f1fbc154350 2024-03-22T12:57:12.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.380+0000 7f1fbb400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9c06bc20 0x7f1f9c06e0e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1fac036000 tx=0x7f1fac03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:12.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.627+0000 7f1fb3400640 1 -- 172.21.15.6:0/4254221950 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1fa8021070 con 0x7f1fbc154350 2024-03-22T12:57:12.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:12.866+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f1fbc151480 con 0x7f1fbc154350 2024-03-22T12:57:13.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.276+0000 7f1fb3400640 1 -- 172.21.15.6:0/4254221950 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1fbc06ffa0 con 0x7f1fbc154350 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.279+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9c06bc20 msgr2=0x7f1f9c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.279+0000 7f1fc2400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9c06bc20 0x7f1f9c06e0e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1fac036000 tx=0x7f1fac03b040 comp rx=0 tx=0).stop 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.279+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 msgr2=0x7f1fbc14c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.279+0000 7f1fc2400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc14c960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc154b10 tx=0x7f1fa800c760 comp rx=0 tx=0).stop 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 shutdown_connections 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fbc154730 0x7f1fbc15a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc154350 0x7f1fbc14c960 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9c06bc20 0x7f1f9c06e0e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 --2- 172.21.15.6:0/4254221950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc066e40 0x7f1fbc0cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 >> 172.21.15.6:0/4254221950 conn(0x7f1fbc0c8e00 msgr2=0x7f1fbc150680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 shutdown_connections 2024-03-22T12:57:13.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.280+0000 7f1fc2400640 1 -- 172.21.15.6:0/4254221950 wait complete. 2024-03-22T12:57:13.309 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-22T12:57:13.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:57:13.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.497+0000 7fd54f200640 1 Processor -- start 2024-03-22T12:57:13.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.497+0000 7fd54f200640 1 -- start start 2024-03-22T12:57:13.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.498+0000 7fd54f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd548154980 0x7fd5480ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:13.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.498+0000 7fd54f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd54815b0c0 0x7fd5480ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:13.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.498+0000 7fd54f200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480cc380 con 0x7fd5480cd300 2024-03-22T12:57:13.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.498+0000 7fd54f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54815a520 con 0x7fd548154980 2024-03-22T12:57:13.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.498+0000 7fd54f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480cc660 con 0x7fd54815b0c0 2024-03-22T12:57:13.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.499+0000 7fd547400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd5480cd300 0x7fd54815a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40436/0 (socket says 172.21.15.6:40436) 2024-03-22T12:57:13.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.499+0000 7fd547400640 1 -- 172.21.15.6:0/1704700408 learned_addr learned my addr 172.21.15.6:0/1704700408 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:13.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.499+0000 7fd54d200640 1 --2- 172.21.15.6:0/1704700408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd54815b0c0 0x7fd5480ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:13.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.499+0000 7fd547e00640 1 --2- 172.21.15.6:0/1704700408 >> v2:172.21.15.6:3300/0 conn(0x7fd548154980 0x7fd5480ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:13.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.502+0000 7fd546a00640 1 -- 172.21.15.6:0/1704700408 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173005797 0 0) 0x7fd5480cc380 con 0x7fd5480cd300 2024-03-22T12:57:13.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.502+0000 7fd546a00640 1 -- 172.21.15.6:0/1704700408 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd52c003520 con 0x7fd5480cd300 2024-03-22T12:57:13.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.502+0000 7fd54d200640 1 -- 172.21.15.6:0/1704700408 >> v2:172.21.15.6:3300/0 conn(0x7fd548154980 msgr2=0x7fd5480ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:13.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.502+0000 7fd54d200640 1 --2- 172.21.15.6:0/1704700408 >> v2:172.21.15.6:3300/0 conn(0x7fd548154980 0x7fd5480ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.502+0000 7fd54d200640 1 -- 172.21.15.6:0/1704700408 >> v1:172.21.15.92:6789/0 conn(0x7fd5480cd300 legacy=0x7fd54815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:13.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.502+0000 7fd54d200640 1 -- 172.21.15.6:0/1704700408 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5480cc940 con 0x7fd54815b0c0 2024-03-22T12:57:13.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.504+0000 7fd54d200640 1 --2- 172.21.15.6:0/1704700408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd54815b0c0 0x7fd5480ce840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd5400097a0 tx=0x7fd54002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efdd9996bbaa312e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:13.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.508+0000 7fd546a00640 1 -- 172.21.15.6:0/1704700408 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540035020 con 0x7fd54815b0c0 2024-03-22T12:57:13.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.508+0000 7fd546a00640 1 -- 172.21.15.6:0/1704700408 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd540031830 con 0x7fd54815b0c0 2024-03-22T12:57:13.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.508+0000 7fd546a00640 1 -- 172.21.15.6:0/1704700408 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540031c00 con 0x7fd54815b0c0 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 -- 172.21.15.6:0/1704700408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd54815b0c0 msgr2=0x7fd5480ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 --2- 172.21.15.6:0/1704700408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd54815b0c0 0x7fd5480ce840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd5400097a0 tx=0x7fd54002b8d0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 -- 172.21.15.6:0/1704700408 shutdown_connections 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 --2- 172.21.15.6:0/1704700408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd54815b0c0 0x7fd5480ce840 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 --2- 172.21.15.6:0/1704700408 >> v2:172.21.15.6:3300/0 conn(0x7fd548154980 0x7fd5480ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 -- 172.21.15.6:0/1704700408 >> 172.21.15.6:0/1704700408 conn(0x7fd5480b16c0 msgr2=0x7fd5480b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 -- 172.21.15.6:0/1704700408 shutdown_connections 2024-03-22T12:57:13.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.509+0000 7fd54f200640 1 -- 172.21.15.6:0/1704700408 wait complete. 2024-03-22T12:57:13.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.511+0000 7fd54f200640 1 Processor -- start 2024-03-22T12:57:13.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.511+0000 7fd54f200640 1 -- start start 2024-03-22T12:57:13.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd54f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5480cd300 0x7fd5480b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:13.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd54f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd548154980 0x7fd54814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd54f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd54815b0c0 0x7fd54814e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd54f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480ceea0 con 0x7fd54815b0c0 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd54f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480ccf00 con 0x7fd5480cd300 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd54f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480cf020 con 0x7fd548154980 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd547400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd548154980 0x7fd54814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.512+0000 7fd54d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd54815b0c0 0x7fd54814e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.513+0000 7fd547400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd548154980 0x7fd54814dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57172/0 (socket says 172.21.15.6:57172) 2024-03-22T12:57:13.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.513+0000 7fd547400640 1 -- 172.21.15.6:0/3744368258 learned_addr learned my addr 172.21.15.6:0/3744368258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.516+0000 7fd547400640 1 -- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5480cd300 msgr2=0x7fd5480b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.516+0000 7fd547400640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5480cd300 0x7fd5480b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.516+0000 7fd547400640 1 -- 172.21.15.6:0/3744368258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd54815b0c0 msgr2=0x7fd54814e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.516+0000 7fd547400640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd54815b0c0 0x7fd54814e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:13.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.516+0000 7fd547400640 1 -- 172.21.15.6:0/3744368258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5481f0220 con 0x7fd548154980 2024-03-22T12:57:13.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.519+0000 7fd547400640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd548154980 0x7fd54814dea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd53000c430 tx=0x7fd53000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:13.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.523+0000 7fd53fe00640 1 -- 172.21.15.6:0/3744368258 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530017070 con 0x7fd548154980 2024-03-22T12:57:13.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.523+0000 7fd53fe00640 1 -- 172.21.15.6:0/3744368258 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530005d00 con 0x7fd548154980 2024-03-22T12:57:13.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.523+0000 7fd53fe00640 1 -- 172.21.15.6:0/3744368258 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530007310 con 0x7fd548154980 2024-03-22T12:57:13.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.523+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5481f0500 con 0x7fd548154980 2024-03-22T12:57:13.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.523+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5481f0ac0 con 0x7fd548154980 2024-03-22T12:57:13.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.525+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd548073240 con 0x7fd548154980 2024-03-22T12:57:13.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.543+0000 7fd53fe00640 1 -- 172.21.15.6:0/3744368258 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd53001a070 con 0x7fd548154980 2024-03-22T12:57:13.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.544+0000 7fd53fe00640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd52406bc20 0x7fd52406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:13.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.544+0000 7fd547e00640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd52406bc20 0x7fd52406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:13.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.545+0000 7fd547e00640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd52406bc20 0x7fd52406e0e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd53800e190 tx=0x7fd53800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.546+0000 7fd53fe00640 1 -- 172.21.15.6:0/3744368258 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fd530056810 con 0x7fd548154980 2024-03-22T12:57:13.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:13.791+0000 7fd53fe00640 1 -- 172.21.15.6:0/3744368258 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd548073240 con 0x7fd548154980 2024-03-22T12:57:14.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.031+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5480cb8c0 con 0x7fd548154980 2024-03-22T12:57:14.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.056+0000 7fd53fe00640 1 -- 172.21.15.6:0/3744368258 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6582 (secure 0 0 0) 0x7fd5480cb8c0 con 0x7fd548154980 2024-03-22T12:57:14.057 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:14.057 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4252,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/2825293724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2825293724},{"type":"v1","addr":"172.21.15.92:6837","nonce":2825293724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:57:13.210608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:creating","state_seq":1,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4265]},"id":1}]} 2024-03-22T12:57:14.057 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 13 2024-03-22T12:57:14.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd52406bc20 msgr2=0x7fd52406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd52406bc20 0x7fd52406e0e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd53800e190 tx=0x7fd53800b040 comp rx=0 tx=0).stop 2024-03-22T12:57:14.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd548154980 msgr2=0x7fd54814dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd548154980 0x7fd54814dea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd53000c430 tx=0x7fd53000c8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 shutdown_connections 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd54815b0c0 0x7fd54814e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd548154980 0x7fd54814dea0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd52406bc20 0x7fd52406e0e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 --2- 172.21.15.6:0/3744368258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5480cd300 0x7fd5480b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.058+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 >> 172.21.15.6:0/3744368258 conn(0x7fd5480b16c0 msgr2=0x7fd5481521a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.060+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 shutdown_connections 2024-03-22T12:57:14.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.060+0000 7fd54f200640 1 -- 172.21.15.6:0/3744368258 wait complete. 2024-03-22T12:57:14.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:14.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.274+0000 7f2ee5000640 1 Processor -- start 2024-03-22T12:57:14.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.274+0000 7f2ee5000640 1 -- start start 2024-03-22T12:57:14.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.276+0000 7f2ee5000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca520 0x7f2ee006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:14.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.276+0000 7f2ee5000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2ee015b0e0 0x7f2ee015b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:14.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.276+0000 7f2ee5000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee006c7a0 con 0x7f2ee00ca900 2024-03-22T12:57:14.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.276+0000 7f2ee5000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee00cd000 con 0x7f2ee015b0e0 2024-03-22T12:57:14.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.276+0000 7f2ee5000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee006ca80 con 0x7f2ee00ca520 2024-03-22T12:57:14.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.277+0000 7f2edde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca520 0x7f2ee006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:14.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.277+0000 7f2ed5400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2ee015b0e0 0x7f2ee015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:14.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.277+0000 7f2ede800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2ee00ca900 0x7f2ee00cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55096/0 (socket says 172.21.15.6:55096) 2024-03-22T12:57:14.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.277+0000 7f2ede800640 1 -- 172.21.15.6:0/3679139231 learned_addr learned my addr 172.21.15.6:0/3679139231 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:14.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.279+0000 7f2ed5400640 1 -- 172.21.15.6:0/3679139231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca520 msgr2=0x7f2ee006d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:14.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.279+0000 7f2ed5400640 1 --2- 172.21.15.6:0/3679139231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca520 0x7f2ee006d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.279+0000 7f2ed5400640 1 -- 172.21.15.6:0/3679139231 >> v1:172.21.15.6:6789/0 conn(0x7f2ee00ca900 legacy=0x7f2ee00cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.279+0000 7f2ed5400640 1 -- 172.21.15.6:0/3679139231 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee006cd60 con 0x7f2ee015b0e0 2024-03-22T12:57:14.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.282+0000 7f2ed5400640 1 --2- 172.21.15.6:0/3679139231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee015b0e0 0x7f2ee015b980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097a0 tx=0x7f2ec802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=193e8a97cd00337c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.285+0000 7f2edd400640 1 -- 172.21.15.6:0/3679139231 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8035020 con 0x7f2ee015b0e0 2024-03-22T12:57:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.285+0000 7f2edd400640 1 -- 172.21.15.6:0/3679139231 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec8031810 con 0x7f2ee015b0e0 2024-03-22T12:57:14.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.285+0000 7f2edd400640 1 -- 172.21.15.6:0/3679139231 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8031be0 con 0x7f2ee015b0e0 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 -- 172.21.15.6:0/3679139231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee015b0e0 msgr2=0x7f2ee015b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3679139231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee015b0e0 0x7f2ee015b980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80097a0 tx=0x7f2ec802b8b0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 -- 172.21.15.6:0/3679139231 shutdown_connections 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3679139231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee015b0e0 0x7f2ee015b980 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3679139231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca520 0x7f2ee006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 -- 172.21.15.6:0/3679139231 >> 172.21.15.6:0/3679139231 conn(0x7f2ee00b16b0 msgr2=0x7f2ee00b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 -- 172.21.15.6:0/3679139231 shutdown_connections 2024-03-22T12:57:14.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.286+0000 7f2ee5000640 1 -- 172.21.15.6:0/3679139231 wait complete. 2024-03-22T12:57:14.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.288+0000 7f2ee5000640 1 Processor -- start 2024-03-22T12:57:14.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.288+0000 7f2ee5000640 1 -- start start 2024-03-22T12:57:14.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2ee5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee00ca520 0x7f2ee00b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:14.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2ee5000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca900 0x7f2ee00c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:14.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2ee5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ee015b0e0 0x7f2ee00c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:14.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2ee5000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00cad60 con 0x7f2ee015b0e0 2024-03-22T12:57:14.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2ee5000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee006d320 con 0x7f2ee00ca520 2024-03-22T12:57:14.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2ee5000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee0154c20 con 0x7f2ee00ca900 2024-03-22T12:57:14.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2ede800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ee015b0e0 0x7f2ee00c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:14.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.289+0000 7f2edde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee00ca520 0x7f2ee00b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:14.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.290+0000 7f2ed5400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca900 0x7f2ee00c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:14.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.290+0000 7f2ede800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ee015b0e0 0x7f2ee00c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33660/0 (socket says 172.21.15.6:33660) 2024-03-22T12:57:14.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.290+0000 7f2ede800640 1 -- 172.21.15.6:0/3060662502 learned_addr learned my addr 172.21.15.6:0/3060662502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:14.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.293+0000 7f2edde00640 1 -- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca900 msgr2=0x7f2ee00c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.293+0000 7f2edde00640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca900 0x7f2ee00c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.293+0000 7f2edde00640 1 -- 172.21.15.6:0/3060662502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ee015b0e0 msgr2=0x7f2ee00c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.293+0000 7f2edde00640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ee015b0e0 0x7f2ee00c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.293+0000 7f2edde00640 1 -- 172.21.15.6:0/3060662502 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee01f09d0 con 0x7f2ee00ca520 2024-03-22T12:57:14.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.293+0000 7f2ed5400640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca900 0x7f2ee00c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:57:14.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.296+0000 7f2edde00640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee00ca520 0x7f2ee00b4890 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e870 tx=0x7f2ed000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:14.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.299+0000 7f2ed6a00640 1 -- 172.21.15.6:0/3060662502 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0017070 con 0x7f2ee00ca520 2024-03-22T12:57:14.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.299+0000 7f2ed6a00640 1 -- 172.21.15.6:0/3060662502 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed000ee60 con 0x7f2ee00ca520 2024-03-22T12:57:14.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.299+0000 7f2ed6a00640 1 -- 172.21.15.6:0/3060662502 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000cb60 con 0x7f2ee00ca520 2024-03-22T12:57:14.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.299+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee01f0cb0 con 0x7f2ee00ca520 2024-03-22T12:57:14.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.299+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee01f1270 con 0x7f2ee00ca520 2024-03-22T12:57:14.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.301+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee00756c0 con 0x7f2ee00ca520 2024-03-22T12:57:14.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.325+0000 7f2ed6a00640 1 -- 172.21.15.6:0/3060662502 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2ed001a070 con 0x7f2ee00ca520 2024-03-22T12:57:14.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.326+0000 7f2ed6a00640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ebc06bc20 0x7f2ebc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:14.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.326+0000 7f2ed5400640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ebc06bc20 0x7f2ebc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:14.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.327+0000 7f2ed5400640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ebc06bc20 0x7f2ebc06e0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8036000 tx=0x7f2ec8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:14.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.327+0000 7f2ed6a00640 1 -- 172.21.15.6:0/3060662502 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f2ed00574f0 con 0x7f2ee00ca520 2024-03-22T12:57:14.400 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T12:57:14.387+0000 f5f9640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-22T12:57:14.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.592+0000 7f2ed6a00640 1 -- 172.21.15.6:0/3060662502 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2ed0059450 con 0x7f2ee00ca520 2024-03-22T12:57:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.824+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ee00c7c90 con 0x7f2ee00ca520 2024-03-22T12:57:14.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.953+0000 7f2ed6a00640 1 -- 172.21.15.6:0/3060662502 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7f2ed005d4f0 con 0x7f2ee00ca520 2024-03-22T12:57:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:14.954 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:14.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ebc06bc20 msgr2=0x7f2ebc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ebc06bc20 0x7f2ebc06e0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8036000 tx=0x7f2ec8008030 comp rx=0 tx=0).stop 2024-03-22T12:57:14.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee00ca520 msgr2=0x7f2ee00b4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee00ca520 0x7f2ee00b4890 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000e870 tx=0x7f2ed000ed30 comp rx=0 tx=0).stop 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 shutdown_connections 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ee015b0e0 0x7f2ee00c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2ee00ca900 0x7f2ee00c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ebc06bc20 0x7f2ebc06e0e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 --2- 172.21.15.6:0/3060662502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ee00ca520 0x7f2ee00b4890 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.955+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 >> 172.21.15.6:0/3060662502 conn(0x7f2ee00b16b0 msgr2=0x7f2ee00b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.956+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 shutdown_connections 2024-03-22T12:57:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:14.956+0000 7f2ee5000640 1 -- 172.21.15.6:0/3060662502 wait complete. 2024-03-22T12:57:14.975 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-22T12:57:14.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T12:57:15.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.155+0000 7fbd61000640 1 Processor -- start 2024-03-22T12:57:15.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.155+0000 7fbd61000640 1 -- start start 2024-03-22T12:57:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.156+0000 7fbd61000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbd5c06f8c0 0x7fbd5c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.156+0000 7fbd61000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbd5c071c10 0x7fbd5c0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.156+0000 7fbd61000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c1590b0 con 0x7fbd5c152400 2024-03-22T12:57:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.156+0000 7fbd61000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c0724d0 con 0x7fbd5c06f8c0 2024-03-22T12:57:15.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.156+0000 7fbd61000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c0721f0 con 0x7fbd5c071c10 2024-03-22T12:57:15.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.236+0000 7fbd59e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd5c152400 0x7fbd5c15b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55100/0 (socket says 172.21.15.6:55100) 2024-03-22T12:57:15.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.236+0000 7fbd59e00640 1 -- 172.21.15.6:0/3978183899 learned_addr learned my addr 172.21.15.6:0/3978183899 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:15.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.239+0000 7fbd59400640 1 --2- 172.21.15.6:0/3978183899 >> v2:172.21.15.6:3300/0 conn(0x7fbd5c06f8c0 0x7fbd5c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:15.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.243+0000 7fbd5a800640 1 --2- 172.21.15.6:0/3978183899 >> v2:172.21.15.92:3300/0 conn(0x7fbd5c071c10 0x7fbd5c0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:15.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.246+0000 7fbd5a800640 1 -- 172.21.15.6:0/3978183899 >> v2:172.21.15.6:3300/0 conn(0x7fbd5c06f8c0 msgr2=0x7fbd5c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:15.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.246+0000 7fbd5a800640 1 --2- 172.21.15.6:0/3978183899 >> v2:172.21.15.6:3300/0 conn(0x7fbd5c06f8c0 0x7fbd5c0716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.246+0000 7fbd5a800640 1 -- 172.21.15.6:0/3978183899 >> v1:172.21.15.6:6789/0 conn(0x7fbd5c152400 legacy=0x7fbd5c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:15.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.246+0000 7fbd5a800640 1 -- 172.21.15.6:0/3978183899 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c1582f0 con 0x7fbd5c071c10 2024-03-22T12:57:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.248+0000 7fbd5a800640 1 --2- 172.21.15.6:0/3978183899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 0x7fbd5c0c5040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c15c830 tx=0x7fbd5402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19d2e0c2602266f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.250+0000 7fbd58a00640 1 -- 172.21.15.6:0/3978183899 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54039070 con 0x7fbd5c071c10 2024-03-22T12:57:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.252+0000 7fbd58a00640 1 -- 172.21.15.6:0/3978183899 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5402cd10 con 0x7fbd5c071c10 2024-03-22T12:57:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.252+0000 7fbd58a00640 1 -- 172.21.15.6:0/3978183899 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54034660 con 0x7fbd5c071c10 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 -- 172.21.15.6:0/3978183899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 msgr2=0x7fbd5c0c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 --2- 172.21.15.6:0/3978183899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 0x7fbd5c0c5040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c15c830 tx=0x7fbd5402bf60 comp rx=0 tx=0).stop 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 -- 172.21.15.6:0/3978183899 shutdown_connections 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 --2- 172.21.15.6:0/3978183899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 0x7fbd5c0c5040 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 --2- 172.21.15.6:0/3978183899 >> v2:172.21.15.6:3300/0 conn(0x7fbd5c06f8c0 0x7fbd5c0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 -- 172.21.15.6:0/3978183899 >> 172.21.15.6:0/3978183899 conn(0x7fbd5c0cb930 msgr2=0x7fbd5c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 -- 172.21.15.6:0/3978183899 shutdown_connections 2024-03-22T12:57:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.254+0000 7fbd61000640 1 -- 172.21.15.6:0/3978183899 wait complete. 2024-03-22T12:57:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.258+0000 7fbd61000640 1 Processor -- start 2024-03-22T12:57:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.258+0000 7fbd61000640 1 -- start start 2024-03-22T12:57:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.260+0000 7fbd61000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd5c06f8c0 0x7fbd5c13bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.260+0000 7fbd61000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 0x7fbd5c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.260+0000 7fbd61000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd5c152400 0x7fbd5c0c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.260+0000 7fbd61000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c158b90 con 0x7fbd5c06f8c0 2024-03-22T12:57:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.260+0000 7fbd61000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c1588b0 con 0x7fbd5c071c10 2024-03-22T12:57:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.260+0000 7fbd61000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c1527e0 con 0x7fbd5c152400 2024-03-22T12:57:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.261+0000 7fbd5a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd5c152400 0x7fbd5c0c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.261+0000 7fbd59400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 0x7fbd5c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.262+0000 7fbd5a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd5c152400 0x7fbd5c0c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57196/0 (socket says 172.21.15.6:57196) 2024-03-22T12:57:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.262+0000 7fbd5a800640 1 -- 172.21.15.6:0/903894541 learned_addr learned my addr 172.21.15.6:0/903894541 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.262+0000 7fbd59e00640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd5c06f8c0 0x7fbd5c13bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.265+0000 7fbd5a800640 1 -- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 msgr2=0x7fbd5c0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.265+0000 7fbd5a800640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 0x7fbd5c0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.265+0000 7fbd5a800640 1 -- 172.21.15.6:0/903894541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd5c06f8c0 msgr2=0x7fbd5c13bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.265+0000 7fbd5a800640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd5c06f8c0 0x7fbd5c13bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.265+0000 7fbd5a800640 1 -- 172.21.15.6:0/903894541 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c0c41a0 con 0x7fbd5c152400 2024-03-22T12:57:15.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.268+0000 7fbd5a800640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd5c152400 0x7fbd5c0c8e20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd5402c570 tx=0x7fbd54004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:15.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.272+0000 7fbd52000640 1 -- 172.21.15.6:0/903894541 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54039050 con 0x7fbd5c152400 2024-03-22T12:57:15.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.272+0000 7fbd52000640 1 -- 172.21.15.6:0/903894541 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd54045070 con 0x7fbd5c152400 2024-03-22T12:57:15.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.273+0000 7fbd52000640 1 -- 172.21.15.6:0/903894541 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5403e830 con 0x7fbd5c152400 2024-03-22T12:57:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.273+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c0c4480 con 0x7fbd5c152400 2024-03-22T12:57:15.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.273+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c0c4a40 con 0x7fbd5c152400 2024-03-22T12:57:15.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.274+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c0756c0 con 0x7fbd5c152400 2024-03-22T12:57:15.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.295+0000 7fbd52000640 1 -- 172.21.15.6:0/903894541 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbd5403e9d0 con 0x7fbd5c152400 2024-03-22T12:57:15.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.296+0000 7fbd52000640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd3c06bc20 0x7fbd3c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:15.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.296+0000 7fbd59e00640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd3c06bc20 0x7fbd3c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:15.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.297+0000 7fbd52000640 1 -- 172.21.15.6:0/903894541 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fbd5407a6e0 con 0x7fbd5c152400 2024-03-22T12:57:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.297+0000 7fbd59e00640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd3c06bc20 0x7fbd3c06e0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c007fd0 tx=0x7fbd4c007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:15.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.573+0000 7fbd52000640 1 -- 172.21.15.6:0/903894541 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbd5407c640 con 0x7fbd5c152400 2024-03-22T12:57:15.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.803+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd5c0c3f50 con 0x7fbd3c06bc20 2024-03-22T12:57:15.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.813+0000 7fbd52000640 1 -- 172.21.15.6:0/903894541 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332709 (secure 0 0 0) 0x7fbd5c0c3f50 con 0x7fbd3c06bc20 2024-03-22T12:57:15.819 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:15.820 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T12:57:15.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd3c06bc20 msgr2=0x7fbd3c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:15.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd3c06bc20 0x7fbd3c06e0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c007fd0 tx=0x7fbd4c007e80 comp rx=0 tx=0).stop 2024-03-22T12:57:15.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd5c152400 msgr2=0x7fbd5c0c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:15.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd5c152400 0x7fbd5c0c8e20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd5402c570 tx=0x7fbd54004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 shutdown_connections 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbd5c152400 0x7fbd5c0c8e20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbd3c06bc20 0x7fbd3c06e0e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd5c071c10 0x7fbd5c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 --2- 172.21.15.6:0/903894541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd5c06f8c0 0x7fbd5c13bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 >> 172.21.15.6:0/903894541 conn(0x7fbd5c0cb930 msgr2=0x7fbd5c1505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 shutdown_connections 2024-03-22T12:57:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:15.819+0000 7fbd61000640 1 -- 172.21.15.6:0/903894541 wait complete. 2024-03-22T12:57:15.837 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-03-22T12:57:15.699563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218852,"kb_used_data":2724,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755868,"statfs":{"total":773094113280,"available":772870008832,"internally_reserved":0,"allocated":2789376,"data_stored":2246348,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.531428"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172444+0000","last_change":"2024-03-22T12:56:28.778294+0000","last_active":"2024-03-22T12:57:04.172444+0000","last_peered":"2024-03-22T12:57:04.172444+0000","last_clean":"2024-03-22T12:57:04.172444+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T12:57:04.172444+0000","last_undegraded":"2024-03-22T12:57:04.172444+0000","last_fullsized":"2024-03-22T12:57:04.172444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174773+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T12:57:04.174773+0000","last_peered":"2024-03-22T12:57:04.174773+0000","last_clean":"2024-03-22T12:57:04.174773+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T12:57:04.174773+0000","last_undegraded":"2024-03-22T12:57:04.174773+0000","last_fullsized":"2024-03-22T12:57:04.174773+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173401+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T12:57:04.173401+0000","last_peered":"2024-03-22T12:57:04.173401+0000","last_clean":"2024-03-22T12:57:04.173401+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T12:57:04.173401+0000","last_undegraded":"2024-03-22T12:57:04.173401+0000","last_fullsized":"2024-03-22T12:57:04.173401+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:35:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174963+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T12:57:04.174963+0000","last_peered":"2024-03-22T12:57:04.174963+0000","last_clean":"2024-03-22T12:57:04.174963+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T12:57:04.174963+0000","last_undegraded":"2024-03-22T12:57:04.174963+0000","last_fullsized":"2024-03-22T12:57:04.174963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172065+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T12:57:04.172065+0000","last_peered":"2024-03-22T12:57:04.172065+0000","last_clean":"2024-03-22T12:57:04.172065+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T12:57:04.172065+0000","last_undegraded":"2024-03-22T12:57:04.172065+0000","last_fullsized":"2024-03-22T12:57:04.172065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042070+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T12:57:05.042070+0000","last_peered":"2024-03-22T12:57:05.042070+0000","last_clean":"2024-03-22T12:57:05.042070+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T12:57:05.042070+0000","last_undegraded":"2024-03-22T12:57:05.042070+0000","last_fullsized":"2024-03-22T12:57:05.042070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:40.477732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"24'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355140+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T12:57:04.355140+0000","last_peered":"2024-03-22T12:57:04.355140+0000","last_clean":"2024-03-22T12:57:04.355140+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T12:57:04.355140+0000","last_undegraded":"2024-03-22T12:57:04.355140+0000","last_fullsized":"2024-03-22T12:57:04.355140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946408+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T12:57:04.946408+0000","last_peered":"2024-03-22T12:57:04.946408+0000","last_clean":"2024-03-22T12:57:04.946408+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T12:57:04.946408+0000","last_undegraded":"2024-03-22T12:57:04.946408+0000","last_fullsized":"2024-03-22T12:57:04.946408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042290+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T12:57:05.042290+0000","last_peered":"2024-03-22T12:57:05.042290+0000","last_clean":"2024-03-22T12:57:05.042290+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T12:57:05.042290+0000","last_undegraded":"2024-03-22T12:57:05.042290+0000","last_fullsized":"2024-03-22T12:57:05.042290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:29.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354571+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T12:57:04.354571+0000","last_peered":"2024-03-22T12:57:04.354571+0000","last_clean":"2024-03-22T12:57:04.354571+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T12:57:04.354571+0000","last_undegraded":"2024-03-22T12:57:04.354571+0000","last_fullsized":"2024-03-22T12:57:04.354571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946567+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T12:57:04.946567+0000","last_peered":"2024-03-22T12:57:04.946567+0000","last_clean":"2024-03-22T12:57:04.946567+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T12:57:04.946567+0000","last_undegraded":"2024-03-22T12:57:04.946567+0000","last_fullsized":"2024-03-22T12:57:04.946567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042408+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T12:57:05.042408+0000","last_peered":"2024-03-22T12:57:05.042408+0000","last_clean":"2024-03-22T12:57:05.042408+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T12:57:05.042408+0000","last_undegraded":"2024-03-22T12:57:05.042408+0000","last_fullsized":"2024-03-22T12:57:05.042408+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247053+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T12:57:04.247053+0000","last_peered":"2024-03-22T12:57:04.247053+0000","last_clean":"2024-03-22T12:57:04.247053+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T12:57:04.247053+0000","last_undegraded":"2024-03-22T12:57:04.247053+0000","last_fullsized":"2024-03-22T12:57:04.247053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.305489+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T12:57:12.305489+0000","last_peered":"2024-03-22T12:57:12.305489+0000","last_clean":"2024-03-22T12:57:12.305489+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T12:57:12.305489+0000","last_undegraded":"2024-03-22T12:57:12.305489+0000","last_fullsized":"2024-03-22T12:57:12.305489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042319+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T12:57:05.042319+0000","last_peered":"2024-03-22T12:57:05.042319+0000","last_clean":"2024-03-22T12:57:05.042319+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T12:57:05.042319+0000","last_undegraded":"2024-03-22T12:57:05.042319+0000","last_fullsized":"2024-03-22T12:57:05.042319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:29.130909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174615+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T12:57:04.174615+0000","last_peered":"2024-03-22T12:57:04.174615+0000","last_clean":"2024-03-22T12:57:04.174615+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T12:57:04.174615+0000","last_undegraded":"2024-03-22T12:57:04.174615+0000","last_fullsized":"2024-03-22T12:57:04.174615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185126+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T12:57:04.185126+0000","last_peered":"2024-03-22T12:57:04.185126+0000","last_clean":"2024-03-22T12:57:04.185126+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T12:57:04.185126+0000","last_undegraded":"2024-03-22T12:57:04.185126+0000","last_fullsized":"2024-03-22T12:57:04.185126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042296+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T12:57:05.042296+0000","last_peered":"2024-03-22T12:57:05.042296+0000","last_clean":"2024-03-22T12:57:05.042296+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T12:57:05.042296+0000","last_undegraded":"2024-03-22T12:57:05.042296+0000","last_fullsized":"2024-03-22T12:57:05.042296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:40.776689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173279+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T12:57:04.173279+0000","last_peered":"2024-03-22T12:57:04.173279+0000","last_clean":"2024-03-22T12:57:04.173279+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T12:57:04.173279+0000","last_undegraded":"2024-03-22T12:57:04.173279+0000","last_fullsized":"2024-03-22T12:57:04.173279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.413103+0000","last_change":"2024-03-22T12:56:28.757253+0000","last_active":"2024-03-22T12:57:12.413103+0000","last_peered":"2024-03-22T12:57:12.413103+0000","last_clean":"2024-03-22T12:57:12.413103+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T12:57:12.413103+0000","last_undegraded":"2024-03-22T12:57:12.413103+0000","last_fullsized":"2024-03-22T12:57:12.413103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:48.506816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185414+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T12:57:04.185414+0000","last_peered":"2024-03-22T12:57:04.185414+0000","last_clean":"2024-03-22T12:57:04.185414+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T12:57:04.185414+0000","last_undegraded":"2024-03-22T12:57:04.185414+0000","last_fullsized":"2024-03-22T12:57:04.185414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042646+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T12:57:05.042646+0000","last_peered":"2024-03-22T12:57:05.042646+0000","last_clean":"2024-03-22T12:57:05.042646+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T12:57:05.042646+0000","last_undegraded":"2024-03-22T12:57:05.042646+0000","last_fullsized":"2024-03-22T12:57:05.042646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:34.508555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.306479+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T12:57:12.306479+0000","last_peered":"2024-03-22T12:57:12.306479+0000","last_clean":"2024-03-22T12:57:12.306479+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T12:57:12.306479+0000","last_undegraded":"2024-03-22T12:57:12.306479+0000","last_fullsized":"2024-03-22T12:57:12.306479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042323+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T12:57:05.042323+0000","last_peered":"2024-03-22T12:57:05.042323+0000","last_clean":"2024-03-22T12:57:05.042323+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T12:57:05.042323+0000","last_undegraded":"2024-03-22T12:57:05.042323+0000","last_fullsized":"2024-03-22T12:57:05.042323+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:09.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172429+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T12:57:04.172429+0000","last_peered":"2024-03-22T12:57:04.172429+0000","last_clean":"2024-03-22T12:57:04.172429+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T12:57:04.172429+0000","last_undegraded":"2024-03-22T12:57:04.172429+0000","last_fullsized":"2024-03-22T12:57:04.172429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.306102+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T12:57:12.306102+0000","last_peered":"2024-03-22T12:57:12.306102+0000","last_clean":"2024-03-22T12:57:12.306102+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T12:57:12.306102+0000","last_undegraded":"2024-03-22T12:57:12.306102+0000","last_fullsized":"2024-03-22T12:57:12.306102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042166+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T12:57:05.042166+0000","last_peered":"2024-03-22T12:57:05.042166+0000","last_clean":"2024-03-22T12:57:05.042166+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T12:57:05.042166+0000","last_undegraded":"2024-03-22T12:57:05.042166+0000","last_fullsized":"2024-03-22T12:57:05.042166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:13.307688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176050+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T12:57:04.176050+0000","last_peered":"2024-03-22T12:57:04.176050+0000","last_clean":"2024-03-22T12:57:04.176050+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T12:57:04.176050+0000","last_undegraded":"2024-03-22T12:57:04.176050+0000","last_fullsized":"2024-03-22T12:57:04.176050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:06.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185611+0000","last_change":"2024-03-22T12:56:28.767925+0000","last_active":"2024-03-22T12:57:04.185611+0000","last_peered":"2024-03-22T12:57:04.185611+0000","last_clean":"2024-03-22T12:57:04.185611+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T12:57:04.185611+0000","last_undegraded":"2024-03-22T12:57:04.185611+0000","last_fullsized":"2024-03-22T12:57:04.185611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172825+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T12:57:04.172825+0000","last_peered":"2024-03-22T12:57:04.172825+0000","last_clean":"2024-03-22T12:57:04.172825+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T12:57:04.172825+0000","last_undegraded":"2024-03-22T12:57:04.172825+0000","last_fullsized":"2024-03-22T12:57:04.172825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:40:09.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185111+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T12:57:04.185111+0000","last_peered":"2024-03-22T12:57:04.185111+0000","last_clean":"2024-03-22T12:57:04.185111+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T12:57:04.185111+0000","last_undegraded":"2024-03-22T12:57:04.185111+0000","last_fullsized":"2024-03-22T12:57:04.185111+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355281+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T12:57:04.355281+0000","last_peered":"2024-03-22T12:57:04.355281+0000","last_clean":"2024-03-22T12:57:04.355281+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T12:57:04.355281+0000","last_undegraded":"2024-03-22T12:57:04.355281+0000","last_fullsized":"2024-03-22T12:57:04.355281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170966+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T12:57:04.170966+0000","last_peered":"2024-03-22T12:57:04.170966+0000","last_clean":"2024-03-22T12:57:04.170966+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T12:57:04.170966+0000","last_undegraded":"2024-03-22T12:57:04.170966+0000","last_fullsized":"2024-03-22T12:57:04.170966+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946629+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T12:57:04.946629+0000","last_peered":"2024-03-22T12:57:04.946629+0000","last_clean":"2024-03-22T12:57:04.946629+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T12:57:04.946629+0000","last_undegraded":"2024-03-22T12:57:04.946629+0000","last_fullsized":"2024-03-22T12:57:04.946629+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185202+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T12:57:04.185202+0000","last_peered":"2024-03-22T12:57:04.185202+0000","last_clean":"2024-03-22T12:57:04.185202+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T12:57:04.185202+0000","last_undegraded":"2024-03-22T12:57:04.185202+0000","last_fullsized":"2024-03-22T12:57:04.185202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045655+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T12:57:05.045655+0000","last_peered":"2024-03-22T12:57:05.045655+0000","last_clean":"2024-03-22T12:57:05.045655+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T12:57:05.045655+0000","last_undegraded":"2024-03-22T12:57:05.045655+0000","last_fullsized":"2024-03-22T12:57:05.045655+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:45.197847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042741+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T12:57:05.042741+0000","last_peered":"2024-03-22T12:57:05.042741+0000","last_clean":"2024-03-22T12:57:05.042741+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T12:57:05.042741+0000","last_undegraded":"2024-03-22T12:57:05.042741+0000","last_fullsized":"2024-03-22T12:57:05.042741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:23.839356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946541+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T12:57:04.946541+0000","last_peered":"2024-03-22T12:57:04.946541+0000","last_clean":"2024-03-22T12:57:04.946541+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T12:57:04.946541+0000","last_undegraded":"2024-03-22T12:57:04.946541+0000","last_fullsized":"2024-03-22T12:57:04.946541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184845+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T12:57:04.184845+0000","last_peered":"2024-03-22T12:57:04.184845+0000","last_clean":"2024-03-22T12:57:04.184845+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T12:57:04.184845+0000","last_undegraded":"2024-03-22T12:57:04.184845+0000","last_fullsized":"2024-03-22T12:57:04.184845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184792+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T12:57:04.184792+0000","last_peered":"2024-03-22T12:57:04.184792+0000","last_clean":"2024-03-22T12:57:04.184792+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T12:57:04.184792+0000","last_undegraded":"2024-03-22T12:57:04.184792+0000","last_fullsized":"2024-03-22T12:57:04.184792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.179893+0000","last_change":"2024-03-22T12:56:28.763384+0000","last_active":"2024-03-22T12:57:04.179893+0000","last_peered":"2024-03-22T12:57:04.179893+0000","last_clean":"2024-03-22T12:57:04.179893+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T12:57:04.179893+0000","last_undegraded":"2024-03-22T12:57:04.179893+0000","last_fullsized":"2024-03-22T12:57:04.179893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:15:45.463056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.952755+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T12:57:04.952755+0000","last_peered":"2024-03-22T12:57:04.952755+0000","last_clean":"2024-03-22T12:57:04.952755+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T12:57:04.952755+0000","last_undegraded":"2024-03-22T12:57:04.952755+0000","last_fullsized":"2024-03-22T12:57:04.952755+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:48.651995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.169659+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T12:57:04.169659+0000","last_peered":"2024-03-22T12:57:04.169659+0000","last_clean":"2024-03-22T12:57:04.169659+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T12:57:04.169659+0000","last_undegraded":"2024-03-22T12:57:04.169659+0000","last_fullsized":"2024-03-22T12:57:04.169659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:36:50.048241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354403+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T12:57:04.354403+0000","last_peered":"2024-03-22T12:57:04.354403+0000","last_clean":"2024-03-22T12:57:04.354403+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T12:57:04.354403+0000","last_undegraded":"2024-03-22T12:57:04.354403+0000","last_fullsized":"2024-03-22T12:57:04.354403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172286+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T12:57:04.172286+0000","last_peered":"2024-03-22T12:57:04.172286+0000","last_clean":"2024-03-22T12:57:04.172286+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T12:57:04.172286+0000","last_undegraded":"2024-03-22T12:57:04.172286+0000","last_fullsized":"2024-03-22T12:57:04.172286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:29:31.400078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174048+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T12:57:04.174048+0000","last_peered":"2024-03-22T12:57:04.174048+0000","last_clean":"2024-03-22T12:57:04.174048+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T12:57:04.174048+0000","last_undegraded":"2024-03-22T12:57:04.174048+0000","last_fullsized":"2024-03-22T12:57:04.174048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.459550+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T12:57:12.459550+0000","last_peered":"2024-03-22T12:57:12.459550+0000","last_clean":"2024-03-22T12:57:12.459550+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T12:57:12.459550+0000","last_undegraded":"2024-03-22T12:57:12.459550+0000","last_fullsized":"2024-03-22T12:57:12.459550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354430+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T12:57:04.354430+0000","last_peered":"2024-03-22T12:57:04.354430+0000","last_clean":"2024-03-22T12:57:04.354430+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T12:57:04.354430+0000","last_undegraded":"2024-03-22T12:57:04.354430+0000","last_fullsized":"2024-03-22T12:57:04.354430+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247232+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T12:57:04.247232+0000","last_peered":"2024-03-22T12:57:04.247232+0000","last_clean":"2024-03-22T12:57:04.247232+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T12:57:04.247232+0000","last_undegraded":"2024-03-22T12:57:04.247232+0000","last_fullsized":"2024-03-22T12:57:04.247232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.181294+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T12:57:04.181294+0000","last_peered":"2024-03-22T12:57:04.181294+0000","last_clean":"2024-03-22T12:57:04.181294+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T12:57:04.181294+0000","last_undegraded":"2024-03-22T12:57:04.181294+0000","last_fullsized":"2024-03-22T12:57:04.181294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247035+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T12:57:04.247035+0000","last_peered":"2024-03-22T12:57:04.247035+0000","last_clean":"2024-03-22T12:57:04.247035+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T12:57:04.247035+0000","last_undegraded":"2024-03-22T12:57:04.247035+0000","last_fullsized":"2024-03-22T12:57:04.247035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946466+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T12:57:04.946466+0000","last_peered":"2024-03-22T12:57:04.946466+0000","last_clean":"2024-03-22T12:57:04.946466+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T12:57:04.946466+0000","last_undegraded":"2024-03-22T12:57:04.946466+0000","last_fullsized":"2024-03-22T12:57:04.946466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.186063+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T12:57:04.186063+0000","last_peered":"2024-03-22T12:57:04.186063+0000","last_clean":"2024-03-22T12:57:04.186063+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T12:57:04.186063+0000","last_undegraded":"2024-03-22T12:57:04.186063+0000","last_fullsized":"2024-03-22T12:57:04.186063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:22.329365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171343+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T12:57:04.171343+0000","last_peered":"2024-03-22T12:57:04.171343+0000","last_clean":"2024-03-22T12:57:04.171343+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T12:57:04.171343+0000","last_undegraded":"2024-03-22T12:57:04.171343+0000","last_fullsized":"2024-03-22T12:57:04.171343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355029+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T12:57:04.355029+0000","last_peered":"2024-03-22T12:57:04.355029+0000","last_clean":"2024-03-22T12:57:04.355029+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T12:57:04.355029+0000","last_undegraded":"2024-03-22T12:57:04.355029+0000","last_fullsized":"2024-03-22T12:57:04.355029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.182677+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T12:57:04.182677+0000","last_peered":"2024-03-22T12:57:04.182677+0000","last_clean":"2024-03-22T12:57:04.182677+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T12:57:04.182677+0000","last_undegraded":"2024-03-22T12:57:04.182677+0000","last_fullsized":"2024-03-22T12:57:04.182677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946109+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T12:57:04.946109+0000","last_peered":"2024-03-22T12:57:04.946109+0000","last_clean":"2024-03-22T12:57:04.946109+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T12:57:04.946109+0000","last_undegraded":"2024-03-22T12:57:04.946109+0000","last_fullsized":"2024-03-22T12:57:04.946109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042539+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T12:57:05.042539+0000","last_peered":"2024-03-22T12:57:05.042539+0000","last_clean":"2024-03-22T12:57:05.042539+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T12:57:05.042539+0000","last_undegraded":"2024-03-22T12:57:05.042539+0000","last_fullsized":"2024-03-22T12:57:05.042539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:24:46.021828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042059+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T12:57:05.042059+0000","last_peered":"2024-03-22T12:57:05.042059+0000","last_clean":"2024-03-22T12:57:05.042059+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T12:57:05.042059+0000","last_undegraded":"2024-03-22T12:57:05.042059+0000","last_fullsized":"2024-03-22T12:57:05.042059+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:38.316319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354615+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T12:57:04.354615+0000","last_peered":"2024-03-22T12:57:04.354615+0000","last_clean":"2024-03-22T12:57:04.354615+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T12:57:04.354615+0000","last_undegraded":"2024-03-22T12:57:04.354615+0000","last_fullsized":"2024-03-22T12:57:04.354615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'10","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.403151+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T12:57:12.403151+0000","last_peered":"2024-03-22T12:57:12.403151+0000","last_clean":"2024-03-22T12:57:12.403151+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T12:57:12.403151+0000","last_undegraded":"2024-03-22T12:57:12.403151+0000","last_fullsized":"2024-03-22T12:57:12.403151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:20:57.722062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172194+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T12:57:04.172194+0000","last_peered":"2024-03-22T12:57:04.172194+0000","last_clean":"2024-03-22T12:57:04.172194+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T12:57:04.172194+0000","last_undegraded":"2024-03-22T12:57:04.172194+0000","last_fullsized":"2024-03-22T12:57:04.172194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247088+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T12:57:04.247088+0000","last_peered":"2024-03-22T12:57:04.247088+0000","last_clean":"2024-03-22T12:57:04.247088+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T12:57:04.247088+0000","last_undegraded":"2024-03-22T12:57:04.247088+0000","last_fullsized":"2024-03-22T12:57:04.247088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247103+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T12:57:04.247103+0000","last_peered":"2024-03-22T12:57:04.247103+0000","last_clean":"2024-03-22T12:57:04.247103+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T12:57:04.247103+0000","last_undegraded":"2024-03-22T12:57:04.247103+0000","last_fullsized":"2024-03-22T12:57:04.247103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176522+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T12:57:04.176522+0000","last_peered":"2024-03-22T12:57:04.176522+0000","last_clean":"2024-03-22T12:57:04.176522+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T12:57:04.176522+0000","last_undegraded":"2024-03-22T12:57:04.176522+0000","last_fullsized":"2024-03-22T12:57:04.176522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:51.788457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354739+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T12:57:04.354739+0000","last_peered":"2024-03-22T12:57:04.354739+0000","last_clean":"2024-03-22T12:57:04.354739+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T12:57:04.354739+0000","last_undegraded":"2024-03-22T12:57:04.354739+0000","last_fullsized":"2024-03-22T12:57:04.354739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.459262+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T12:57:12.459262+0000","last_peered":"2024-03-22T12:57:12.459262+0000","last_clean":"2024-03-22T12:57:12.459262+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T12:57:12.459262+0000","last_undegraded":"2024-03-22T12:57:12.459262+0000","last_fullsized":"2024-03-22T12:57:12.459262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.949816+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T12:57:04.949816+0000","last_peered":"2024-03-22T12:57:04.949816+0000","last_clean":"2024-03-22T12:57:04.949816+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T12:57:04.949816+0000","last_undegraded":"2024-03-22T12:57:04.949816+0000","last_fullsized":"2024-03-22T12:57:04.949816+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:08:12.860933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170220+0000","last_change":"2024-03-22T12:56:30.299969+0000","last_active":"2024-03-22T12:57:04.170220+0000","last_peered":"2024-03-22T12:57:04.170220+0000","last_clean":"2024-03-22T12:57:04.170220+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T12:57:04.170220+0000","last_undegraded":"2024-03-22T12:57:04.170220+0000","last_fullsized":"2024-03-22T12:57:04.170220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.300624+0000","last_change":"2024-03-22T12:56:28.777142+0000","last_active":"2024-03-22T12:57:12.300624+0000","last_peered":"2024-03-22T12:57:12.300624+0000","last_clean":"2024-03-22T12:57:12.300624+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T12:57:12.300624+0000","last_undegraded":"2024-03-22T12:57:12.300624+0000","last_fullsized":"2024-03-22T12:57:12.300624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:49.411808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946528+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T12:57:04.946528+0000","last_peered":"2024-03-22T12:57:04.946528+0000","last_clean":"2024-03-22T12:57:04.946528+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T12:57:04.946528+0000","last_undegraded":"2024-03-22T12:57:04.946528+0000","last_fullsized":"2024-03-22T12:57:04.946528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354901+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T12:57:04.354901+0000","last_peered":"2024-03-22T12:57:04.354901+0000","last_clean":"2024-03-22T12:57:04.354901+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T12:57:04.354901+0000","last_undegraded":"2024-03-22T12:57:04.354901+0000","last_fullsized":"2024-03-22T12:57:04.354901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'12","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.308529+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T12:57:12.308529+0000","last_peered":"2024-03-22T12:57:12.308529+0000","last_clean":"2024-03-22T12:57:12.308529+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T12:57:12.308529+0000","last_undegraded":"2024-03-22T12:57:12.308529+0000","last_fullsized":"2024-03-22T12:57:12.308529+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184655+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T12:57:04.184655+0000","last_peered":"2024-03-22T12:57:04.184655+0000","last_clean":"2024-03-22T12:57:04.184655+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T12:57:04.184655+0000","last_undegraded":"2024-03-22T12:57:04.184655+0000","last_fullsized":"2024-03-22T12:57:04.184655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042567+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T12:57:05.042567+0000","last_peered":"2024-03-22T12:57:05.042567+0000","last_clean":"2024-03-22T12:57:05.042567+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T12:57:05.042567+0000","last_undegraded":"2024-03-22T12:57:05.042567+0000","last_fullsized":"2024-03-22T12:57:05.042567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:49.776407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.186011+0000","last_change":"2024-03-22T12:56:28.769041+0000","last_active":"2024-03-22T12:57:04.186011+0000","last_peered":"2024-03-22T12:57:04.186011+0000","last_clean":"2024-03-22T12:57:04.186011+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T12:57:04.186011+0000","last_undegraded":"2024-03-22T12:57:04.186011+0000","last_fullsized":"2024-03-22T12:57:04.186011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042461+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T12:57:05.042461+0000","last_peered":"2024-03-22T12:57:05.042461+0000","last_clean":"2024-03-22T12:57:05.042461+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T12:57:05.042461+0000","last_undegraded":"2024-03-22T12:57:05.042461+0000","last_fullsized":"2024-03-22T12:57:05.042461+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:56.815720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172275+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T12:57:04.172275+0000","last_peered":"2024-03-22T12:57:04.172275+0000","last_clean":"2024-03-22T12:57:04.172275+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T12:57:04.172275+0000","last_undegraded":"2024-03-22T12:57:04.172275+0000","last_fullsized":"2024-03-22T12:57:04.172275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.457584+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T12:57:12.457584+0000","last_peered":"2024-03-22T12:57:12.457584+0000","last_clean":"2024-03-22T12:57:12.457584+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T12:57:12.457584+0000","last_undegraded":"2024-03-22T12:57:12.457584+0000","last_fullsized":"2024-03-22T12:57:12.457584+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173079+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T12:57:04.173079+0000","last_peered":"2024-03-22T12:57:04.173079+0000","last_clean":"2024-03-22T12:57:04.173079+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T12:57:04.173079+0000","last_undegraded":"2024-03-22T12:57:04.173079+0000","last_fullsized":"2024-03-22T12:57:04.173079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:06:47.716384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946555+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T12:57:04.946555+0000","last_peered":"2024-03-22T12:57:04.946555+0000","last_clean":"2024-03-22T12:57:04.946555+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T12:57:04.946555+0000","last_undegraded":"2024-03-22T12:57:04.946555+0000","last_fullsized":"2024-03-22T12:57:04.946555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.307509+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T12:57:12.307509+0000","last_peered":"2024-03-22T12:57:12.307509+0000","last_clean":"2024-03-22T12:57:12.307509+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T12:57:12.307509+0000","last_undegraded":"2024-03-22T12:57:12.307509+0000","last_fullsized":"2024-03-22T12:57:12.307509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174378+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T12:57:04.174378+0000","last_peered":"2024-03-22T12:57:04.174378+0000","last_clean":"2024-03-22T12:57:04.174378+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T12:57:04.174378+0000","last_undegraded":"2024-03-22T12:57:04.174378+0000","last_fullsized":"2024-03-22T12:57:04.174378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172090+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T12:57:04.172090+0000","last_peered":"2024-03-22T12:57:04.172090+0000","last_clean":"2024-03-22T12:57:04.172090+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T12:57:04.172090+0000","last_undegraded":"2024-03-22T12:57:04.172090+0000","last_fullsized":"2024-03-22T12:57:04.172090+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184400+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T12:57:04.184400+0000","last_peered":"2024-03-22T12:57:04.184400+0000","last_clean":"2024-03-22T12:57:04.184400+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T12:57:04.184400+0000","last_undegraded":"2024-03-22T12:57:04.184400+0000","last_fullsized":"2024-03-22T12:57:04.184400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946306+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T12:57:04.946306+0000","last_peered":"2024-03-22T12:57:04.946306+0000","last_clean":"2024-03-22T12:57:04.946306+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T12:57:04.946306+0000","last_undegraded":"2024-03-22T12:57:04.946306+0000","last_fullsized":"2024-03-22T12:57:04.946306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247413+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T12:57:04.247413+0000","last_peered":"2024-03-22T12:57:04.247413+0000","last_clean":"2024-03-22T12:57:04.247413+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T12:57:04.247413+0000","last_undegraded":"2024-03-22T12:57:04.247413+0000","last_fullsized":"2024-03-22T12:57:04.247413+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173181+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T12:57:04.173181+0000","last_peered":"2024-03-22T12:57:04.173181+0000","last_clean":"2024-03-22T12:57:04.173181+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T12:57:04.173181+0000","last_undegraded":"2024-03-22T12:57:04.173181+0000","last_fullsized":"2024-03-22T12:57:04.173181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174556+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T12:57:04.174556+0000","last_peered":"2024-03-22T12:57:04.174556+0000","last_clean":"2024-03-22T12:57:04.174556+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T12:57:04.174556+0000","last_undegraded":"2024-03-22T12:57:04.174556+0000","last_fullsized":"2024-03-22T12:57:04.174556+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:31:41.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247241+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T12:57:04.247241+0000","last_peered":"2024-03-22T12:57:04.247241+0000","last_clean":"2024-03-22T12:57:04.247241+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T12:57:04.247241+0000","last_undegraded":"2024-03-22T12:57:04.247241+0000","last_fullsized":"2024-03-22T12:57:04.247241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.047312+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T12:57:05.047312+0000","last_peered":"2024-03-22T12:57:05.047312+0000","last_clean":"2024-03-22T12:57:05.047312+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T12:57:05.047312+0000","last_undegraded":"2024-03-22T12:57:05.047312+0000","last_fullsized":"2024-03-22T12:57:05.047312+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.180347+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T12:57:04.180347+0000","last_peered":"2024-03-22T12:57:04.180347+0000","last_clean":"2024-03-22T12:57:04.180347+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T12:57:04.180347+0000","last_undegraded":"2024-03-22T12:57:04.180347+0000","last_fullsized":"2024-03-22T12:57:04.180347+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:43.256977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.481695+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T12:57:12.481695+0000","last_peered":"2024-03-22T12:57:12.481695+0000","last_clean":"2024-03-22T12:57:12.481695+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T12:57:12.481695+0000","last_undegraded":"2024-03-22T12:57:12.481695+0000","last_fullsized":"2024-03-22T12:57:12.481695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042643+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T12:57:05.042643+0000","last_peered":"2024-03-22T12:57:05.042643+0000","last_clean":"2024-03-22T12:57:05.042643+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T12:57:05.042643+0000","last_undegraded":"2024-03-22T12:57:05.042643+0000","last_fullsized":"2024-03-22T12:57:05.042643+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:48.882251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174116+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T12:57:04.174116+0000","last_peered":"2024-03-22T12:57:04.174116+0000","last_clean":"2024-03-22T12:57:04.174116+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T12:57:04.174116+0000","last_undegraded":"2024-03-22T12:57:04.174116+0000","last_fullsized":"2024-03-22T12:57:04.174116+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247032+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T12:57:04.247032+0000","last_peered":"2024-03-22T12:57:04.247032+0000","last_clean":"2024-03-22T12:57:04.247032+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T12:57:04.247032+0000","last_undegraded":"2024-03-22T12:57:04.247032+0000","last_fullsized":"2024-03-22T12:57:04.247032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.188189+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T12:57:04.188189+0000","last_peered":"2024-03-22T12:57:04.188189+0000","last_clean":"2024-03-22T12:57:04.188189+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T12:57:04.188189+0000","last_undegraded":"2024-03-22T12:57:04.188189+0000","last_fullsized":"2024-03-22T12:57:04.188189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946357+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T12:57:04.946357+0000","last_peered":"2024-03-22T12:57:04.946357+0000","last_clean":"2024-03-22T12:57:04.946357+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T12:57:04.946357+0000","last_undegraded":"2024-03-22T12:57:04.946357+0000","last_fullsized":"2024-03-22T12:57:04.946357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171397+0000","last_change":"2024-03-22T12:56:30.300244+0000","last_active":"2024-03-22T12:57:04.171397+0000","last_peered":"2024-03-22T12:57:04.171397+0000","last_clean":"2024-03-22T12:57:04.171397+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T12:57:04.171397+0000","last_undegraded":"2024-03-22T12:57:04.171397+0000","last_fullsized":"2024-03-22T12:57:04.171397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174764+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T12:57:04.174764+0000","last_peered":"2024-03-22T12:57:04.174764+0000","last_clean":"2024-03-22T12:57:04.174764+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T12:57:04.174764+0000","last_undegraded":"2024-03-22T12:57:04.174764+0000","last_fullsized":"2024-03-22T12:57:04.174764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185804+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T12:57:04.185804+0000","last_peered":"2024-03-22T12:57:04.185804+0000","last_clean":"2024-03-22T12:57:04.185804+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T12:57:04.185804+0000","last_undegraded":"2024-03-22T12:57:04.185804+0000","last_fullsized":"2024-03-22T12:57:04.185804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247526+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T12:57:04.247526+0000","last_peered":"2024-03-22T12:57:04.247526+0000","last_clean":"2024-03-22T12:57:04.247526+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T12:57:04.247526+0000","last_undegraded":"2024-03-22T12:57:04.247526+0000","last_fullsized":"2024-03-22T12:57:04.247526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.175763+0000","last_change":"2024-03-22T12:56:30.308903+0000","last_active":"2024-03-22T12:57:04.175763+0000","last_peered":"2024-03-22T12:57:04.175763+0000","last_clean":"2024-03-22T12:57:04.175763+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T12:57:04.175763+0000","last_undegraded":"2024-03-22T12:57:04.175763+0000","last_fullsized":"2024-03-22T12:57:04.175763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185929+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T12:57:04.185929+0000","last_peered":"2024-03-22T12:57:04.185929+0000","last_clean":"2024-03-22T12:57:04.185929+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T12:57:04.185929+0000","last_undegraded":"2024-03-22T12:57:04.185929+0000","last_fullsized":"2024-03-22T12:57:04.185929+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:13:57.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045995+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T12:57:05.045995+0000","last_peered":"2024-03-22T12:57:05.045995+0000","last_clean":"2024-03-22T12:57:05.045995+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T12:57:05.045995+0000","last_undegraded":"2024-03-22T12:57:05.045995+0000","last_fullsized":"2024-03-22T12:57:05.045995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:19.797978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946253+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T12:57:04.946253+0000","last_peered":"2024-03-22T12:57:04.946253+0000","last_clean":"2024-03-22T12:57:04.946253+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T12:57:04.946253+0000","last_undegraded":"2024-03-22T12:57:04.946253+0000","last_fullsized":"2024-03-22T12:57:04.946253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171180+0000","last_change":"2024-03-22T12:56:30.299214+0000","last_active":"2024-03-22T12:57:04.171180+0000","last_peered":"2024-03-22T12:57:04.171180+0000","last_clean":"2024-03-22T12:57:04.171180+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T12:57:04.171180+0000","last_undegraded":"2024-03-22T12:57:04.171180+0000","last_fullsized":"2024-03-22T12:57:04.171180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:35.356723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247391+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T12:57:04.247391+0000","last_peered":"2024-03-22T12:57:04.247391+0000","last_clean":"2024-03-22T12:57:04.247391+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T12:57:04.247391+0000","last_undegraded":"2024-03-22T12:57:04.247391+0000","last_fullsized":"2024-03-22T12:57:04.247391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.189179+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T12:57:04.189179+0000","last_peered":"2024-03-22T12:57:04.189179+0000","last_clean":"2024-03-22T12:57:04.189179+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T12:57:04.189179+0000","last_undegraded":"2024-03-22T12:57:04.189179+0000","last_fullsized":"2024-03-22T12:57:04.189179+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:57:33.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.182435+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T12:57:04.182435+0000","last_peered":"2024-03-22T12:57:04.182435+0000","last_clean":"2024-03-22T12:57:04.182435+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T12:57:04.182435+0000","last_undegraded":"2024-03-22T12:57:04.182435+0000","last_fullsized":"2024-03-22T12:57:04.182435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:38.987151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946819+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T12:57:04.946819+0000","last_peered":"2024-03-22T12:57:04.946819+0000","last_clean":"2024-03-22T12:57:04.946819+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T12:57:04.946819+0000","last_undegraded":"2024-03-22T12:57:04.946819+0000","last_fullsized":"2024-03-22T12:57:04.946819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946220+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T12:57:04.946220+0000","last_peered":"2024-03-22T12:57:04.946220+0000","last_clean":"2024-03-22T12:57:04.946220+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T12:57:04.946220+0000","last_undegraded":"2024-03-22T12:57:04.946220+0000","last_fullsized":"2024-03-22T12:57:04.946220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184477+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T12:57:04.184477+0000","last_peered":"2024-03-22T12:57:04.184477+0000","last_clean":"2024-03-22T12:57:04.184477+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T12:57:04.184477+0000","last_undegraded":"2024-03-22T12:57:04.184477+0000","last_fullsized":"2024-03-22T12:57:04.184477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170174+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T12:57:04.170174+0000","last_peered":"2024-03-22T12:57:04.170174+0000","last_clean":"2024-03-22T12:57:04.170174+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T12:57:04.170174+0000","last_undegraded":"2024-03-22T12:57:04.170174+0000","last_fullsized":"2024-03-22T12:57:04.170174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174348+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T12:57:04.174348+0000","last_peered":"2024-03-22T12:57:04.174348+0000","last_clean":"2024-03-22T12:57:04.174348+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T12:57:04.174348+0000","last_undegraded":"2024-03-22T12:57:04.174348+0000","last_fullsized":"2024-03-22T12:57:04.174348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042563+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T12:57:05.042563+0000","last_peered":"2024-03-22T12:57:05.042563+0000","last_clean":"2024-03-22T12:57:05.042563+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T12:57:05.042563+0000","last_undegraded":"2024-03-22T12:57:05.042563+0000","last_fullsized":"2024-03-22T12:57:05.042563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:23.598956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.251063+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T12:57:04.251063+0000","last_peered":"2024-03-22T12:57:04.251063+0000","last_clean":"2024-03-22T12:57:04.251063+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T12:57:04.251063+0000","last_undegraded":"2024-03-22T12:57:04.251063+0000","last_fullsized":"2024-03-22T12:57:04.251063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247414+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T12:57:04.247414+0000","last_peered":"2024-03-22T12:57:04.247414+0000","last_clean":"2024-03-22T12:57:04.247414+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T12:57:04.247414+0000","last_undegraded":"2024-03-22T12:57:04.247414+0000","last_fullsized":"2024-03-22T12:57:04.247414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355010+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T12:57:04.355010+0000","last_peered":"2024-03-22T12:57:04.355010+0000","last_clean":"2024-03-22T12:57:04.355010+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T12:57:04.355010+0000","last_undegraded":"2024-03-22T12:57:04.355010+0000","last_fullsized":"2024-03-22T12:57:04.355010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172977+0000","last_change":"2024-03-22T12:56:30.309258+0000","last_active":"2024-03-22T12:57:04.172977+0000","last_peered":"2024-03-22T12:57:04.172977+0000","last_clean":"2024-03-22T12:57:04.172977+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T12:57:04.172977+0000","last_undegraded":"2024-03-22T12:57:04.172977+0000","last_fullsized":"2024-03-22T12:57:04.172977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354578+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T12:57:04.354578+0000","last_peered":"2024-03-22T12:57:04.354578+0000","last_clean":"2024-03-22T12:57:04.354578+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T12:57:04.354578+0000","last_undegraded":"2024-03-22T12:57:04.354578+0000","last_fullsized":"2024-03-22T12:57:04.354578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946219+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T12:57:04.946219+0000","last_peered":"2024-03-22T12:57:04.946219+0000","last_clean":"2024-03-22T12:57:04.946219+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T12:57:04.946219+0000","last_undegraded":"2024-03-22T12:57:04.946219+0000","last_fullsized":"2024-03-22T12:57:04.946219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354584+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T12:57:04.354584+0000","last_peered":"2024-03-22T12:57:04.354584+0000","last_clean":"2024-03-22T12:57:04.354584+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T12:57:04.354584+0000","last_undegraded":"2024-03-22T12:57:04.354584+0000","last_fullsized":"2024-03-22T12:57:04.354584+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946376+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T12:57:04.946376+0000","last_peered":"2024-03-22T12:57:04.946376+0000","last_clean":"2024-03-22T12:57:04.946376+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T12:57:04.946376+0000","last_undegraded":"2024-03-22T12:57:04.946376+0000","last_fullsized":"2024-03-22T12:57:04.946376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171812+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T12:57:04.171812+0000","last_peered":"2024-03-22T12:57:04.171812+0000","last_clean":"2024-03-22T12:57:04.171812+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T12:57:04.171812+0000","last_undegraded":"2024-03-22T12:57:04.171812+0000","last_fullsized":"2024-03-22T12:57:04.171812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:14.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185524+0000","last_change":"2024-03-22T12:56:30.303254+0000","last_active":"2024-03-22T12:57:04.185524+0000","last_peered":"2024-03-22T12:57:04.185524+0000","last_clean":"2024-03-22T12:57:04.185524+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T12:57:04.185524+0000","last_undegraded":"2024-03-22T12:57:04.185524+0000","last_fullsized":"2024-03-22T12:57:04.185524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354841+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T12:57:04.354841+0000","last_peered":"2024-03-22T12:57:04.354841+0000","last_clean":"2024-03-22T12:57:04.354841+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T12:57:04.354841+0000","last_undegraded":"2024-03-22T12:57:04.354841+0000","last_fullsized":"2024-03-22T12:57:04.354841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509455,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":132541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509455,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":131905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":136243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":132611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":401408,"data_stored":722807,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":722751,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":135711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542159,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":131779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T12:57:15.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T12:57:16.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.070+0000 7fbc95e00640 1 Processor -- start 2024-03-22T12:57:16.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.070+0000 7fbc95e00640 1 -- start start 2024-03-22T12:57:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.072+0000 7fbc95e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbc900c7ee0 0x7fbc9015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.072+0000 7fbc95e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc900c68c0 0x7fbc9006b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.072+0000 7fbc95e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc900c4740 con 0x7fbc90154240 2024-03-22T12:57:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.072+0000 7fbc95e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9015e130 con 0x7fbc900c68c0 2024-03-22T12:57:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.072+0000 7fbc95e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc90154620 con 0x7fbc900c7ee0 2024-03-22T12:57:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.073+0000 7fbc8f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc900c68c0 0x7fbc9006b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.073+0000 7fbc8f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc900c68c0 0x7fbc9006b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33684/0 (socket says 172.21.15.6:33684) 2024-03-22T12:57:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.073+0000 7fbc8f400640 1 -- 172.21.15.6:0/1963357663 learned_addr learned my addr 172.21.15.6:0/1963357663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.074+0000 7fbc8e000640 1 --2- 172.21.15.6:0/1963357663 >> v2:172.21.15.92:3300/0 conn(0x7fbc900c7ee0 0x7fbc9015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.077+0000 7fbc8f400640 1 -- 172.21.15.6:0/1963357663 >> v2:172.21.15.92:3300/0 conn(0x7fbc900c7ee0 msgr2=0x7fbc9015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:16.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.077+0000 7fbc8f400640 1 --2- 172.21.15.6:0/1963357663 >> v2:172.21.15.92:3300/0 conn(0x7fbc900c7ee0 0x7fbc9015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.077+0000 7fbc8f400640 1 -- 172.21.15.6:0/1963357663 >> v1:172.21.15.92:6789/0 conn(0x7fbc90154240 legacy=0x7fbc900c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:16.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.077+0000 7fbc8f400640 1 -- 172.21.15.6:0/1963357663 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc900c6e00 con 0x7fbc900c68c0 2024-03-22T12:57:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.080+0000 7fbc8f400640 1 --2- 172.21.15.6:0/1963357663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc9006b080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc9006be90 tx=0x7fbc8802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e3954a692ef133 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:16.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.085+0000 7fbc8d600640 1 -- 172.21.15.6:0/1963357663 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8802cb20 con 0x7fbc900c68c0 2024-03-22T12:57:16.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.086+0000 7fbc8d600640 1 -- 172.21.15.6:0/1963357663 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8802ccc0 con 0x7fbc900c68c0 2024-03-22T12:57:16.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.086+0000 7fbc8d600640 1 -- 172.21.15.6:0/1963357663 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88034610 con 0x7fbc900c68c0 2024-03-22T12:57:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.086+0000 7fbc95e00640 1 -- 172.21.15.6:0/1963357663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 msgr2=0x7fbc9006b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.086+0000 7fbc95e00640 1 --2- 172.21.15.6:0/1963357663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc9006b080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc9006be90 tx=0x7fbc8802bf80 comp rx=0 tx=0).stop 2024-03-22T12:57:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.087+0000 7fbc95e00640 1 -- 172.21.15.6:0/1963357663 shutdown_connections 2024-03-22T12:57:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.087+0000 7fbc95e00640 1 --2- 172.21.15.6:0/1963357663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc9006b080 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.087+0000 7fbc95e00640 1 --2- 172.21.15.6:0/1963357663 >> v2:172.21.15.92:3300/0 conn(0x7fbc900c7ee0 0x7fbc9015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.087+0000 7fbc95e00640 1 -- 172.21.15.6:0/1963357663 >> 172.21.15.6:0/1963357663 conn(0x7fbc900cb930 msgr2=0x7fbc900cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.087+0000 7fbc95e00640 1 -- 172.21.15.6:0/1963357663 shutdown_connections 2024-03-22T12:57:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.087+0000 7fbc95e00640 1 -- 172.21.15.6:0/1963357663 wait complete. 2024-03-22T12:57:16.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.089+0000 7fbc95e00640 1 Processor -- start 2024-03-22T12:57:16.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.090+0000 7fbc95e00640 1 -- start start 2024-03-22T12:57:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.091+0000 7fbc95e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc900c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.091+0000 7fbc95e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc900c7ee0 0x7fbc901e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.091+0000 7fbc95e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc90154240 0x7fbc901f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.091+0000 7fbc95e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9015dae0 con 0x7fbc900c68c0 2024-03-22T12:57:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.091+0000 7fbc95e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9015d800 con 0x7fbc90154240 2024-03-22T12:57:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.091+0000 7fbc95e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9015ddc0 con 0x7fbc900c7ee0 2024-03-22T12:57:16.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.092+0000 7fbc8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc900c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.092+0000 7fbc8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc90154240 0x7fbc901f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.092+0000 7fbc8e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc900c7ee0 0x7fbc901e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.092+0000 7fbc8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc900c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33692/0 (socket says 172.21.15.6:33692) 2024-03-22T12:57:16.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.092+0000 7fbc8ea00640 1 -- 172.21.15.6:0/4250190647 learned_addr learned my addr 172.21.15.6:0/4250190647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:16.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.096+0000 7fbc8ea00640 1 -- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc900c7ee0 msgr2=0x7fbc901e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.096+0000 7fbc8ea00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc900c7ee0 0x7fbc901e5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.096+0000 7fbc8ea00640 1 -- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc90154240 msgr2=0x7fbc901f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.096+0000 7fbc8ea00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc90154240 0x7fbc901f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.096+0000 7fbc8ea00640 1 -- 172.21.15.6:0/4250190647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90072770 con 0x7fbc900c68c0 2024-03-22T12:57:16.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.099+0000 7fbc8ea00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc900c9730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000b3e0 tx=0x7fbc8000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:16.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.104+0000 7fbc87400640 1 -- 172.21.15.6:0/4250190647 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80015020 con 0x7fbc900c68c0 2024-03-22T12:57:16.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.104+0000 7fbc87400640 1 -- 172.21.15.6:0/4250190647 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc80011850 con 0x7fbc900c68c0 2024-03-22T12:57:16.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.104+0000 7fbc87400640 1 -- 172.21.15.6:0/4250190647 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc80011c40 con 0x7fbc900c68c0 2024-03-22T12:57:16.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.104+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc90150410 con 0x7fbc900c68c0 2024-03-22T12:57:16.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.105+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc901f3a10 con 0x7fbc900c68c0 2024-03-22T12:57:16.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.106+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc90068150 con 0x7fbc900c68c0 2024-03-22T12:57:16.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.131+0000 7fbc87400640 1 -- 172.21.15.6:0/4250190647 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbc80018020 con 0x7fbc900c68c0 2024-03-22T12:57:16.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.132+0000 7fbc87400640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbc7006bc20 0x7fbc7006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.132+0000 7fbc8e000640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbc7006bc20 0x7fbc7006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.133+0000 7fbc8e000640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbc7006bc20 0x7fbc7006e0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc90072230 tx=0x7fbc7c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.431+0000 7fbc87400640 1 -- 172.21.15.6:0/4250190647 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fbc80011de0 con 0x7fbc900c68c0 2024-03-22T12:57:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.432+0000 7fbc87400640 1 -- 172.21.15.6:0/4250190647 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbc90068150 con 0x7fbc900c68c0 2024-03-22T12:57:16.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.689+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc901f47b0 con 0x7fbc7006bc20 2024-03-22T12:57:16.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.698+0000 7fbc87400640 1 -- 172.21.15.6:0/4250190647 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332709 (secure 0 0 0) 0x7fbc901f47b0 con 0x7fbc7006bc20 2024-03-22T12:57:16.703 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:16.704 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbc7006bc20 msgr2=0x7fbc7006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbc7006bc20 0x7fbc7006e0e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc90072230 tx=0x7fbc7c009190 comp rx=0 tx=0).stop 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 msgr2=0x7fbc900c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc900c9730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000b3e0 tx=0x7fbc8000b8a0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 shutdown_connections 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbc7006bc20 0x7fbc7006e0e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc90154240 0x7fbc901f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc900c7ee0 0x7fbc901e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 --2- 172.21.15.6:0/4250190647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc900c68c0 0x7fbc900c9730 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 >> 172.21.15.6:0/4250190647 conn(0x7fbc900cb930 msgr2=0x7fbc90153360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:16.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 shutdown_connections 2024-03-22T12:57:16.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.703+0000 7fbc95e00640 1 -- 172.21.15.6:0/4250190647 wait complete. 2024-03-22T12:57:16.722 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-03-22T12:57:15.699563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218852,"kb_used_data":2724,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755868,"statfs":{"total":773094113280,"available":772870008832,"internally_reserved":0,"allocated":2789376,"data_stored":2246348,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.531428"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172444+0000","last_change":"2024-03-22T12:56:28.778294+0000","last_active":"2024-03-22T12:57:04.172444+0000","last_peered":"2024-03-22T12:57:04.172444+0000","last_clean":"2024-03-22T12:57:04.172444+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T12:57:04.172444+0000","last_undegraded":"2024-03-22T12:57:04.172444+0000","last_fullsized":"2024-03-22T12:57:04.172444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174773+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T12:57:04.174773+0000","last_peered":"2024-03-22T12:57:04.174773+0000","last_clean":"2024-03-22T12:57:04.174773+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T12:57:04.174773+0000","last_undegraded":"2024-03-22T12:57:04.174773+0000","last_fullsized":"2024-03-22T12:57:04.174773+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173401+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T12:57:04.173401+0000","last_peered":"2024-03-22T12:57:04.173401+0000","last_clean":"2024-03-22T12:57:04.173401+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T12:57:04.173401+0000","last_undegraded":"2024-03-22T12:57:04.173401+0000","last_fullsized":"2024-03-22T12:57:04.173401+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:35:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174963+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T12:57:04.174963+0000","last_peered":"2024-03-22T12:57:04.174963+0000","last_clean":"2024-03-22T12:57:04.174963+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T12:57:04.174963+0000","last_undegraded":"2024-03-22T12:57:04.174963+0000","last_fullsized":"2024-03-22T12:57:04.174963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172065+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T12:57:04.172065+0000","last_peered":"2024-03-22T12:57:04.172065+0000","last_clean":"2024-03-22T12:57:04.172065+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T12:57:04.172065+0000","last_undegraded":"2024-03-22T12:57:04.172065+0000","last_fullsized":"2024-03-22T12:57:04.172065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042070+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T12:57:05.042070+0000","last_peered":"2024-03-22T12:57:05.042070+0000","last_clean":"2024-03-22T12:57:05.042070+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T12:57:05.042070+0000","last_undegraded":"2024-03-22T12:57:05.042070+0000","last_fullsized":"2024-03-22T12:57:05.042070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:40.477732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"24'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355140+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T12:57:04.355140+0000","last_peered":"2024-03-22T12:57:04.355140+0000","last_clean":"2024-03-22T12:57:04.355140+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T12:57:04.355140+0000","last_undegraded":"2024-03-22T12:57:04.355140+0000","last_fullsized":"2024-03-22T12:57:04.355140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946408+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T12:57:04.946408+0000","last_peered":"2024-03-22T12:57:04.946408+0000","last_clean":"2024-03-22T12:57:04.946408+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T12:57:04.946408+0000","last_undegraded":"2024-03-22T12:57:04.946408+0000","last_fullsized":"2024-03-22T12:57:04.946408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042290+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T12:57:05.042290+0000","last_peered":"2024-03-22T12:57:05.042290+0000","last_clean":"2024-03-22T12:57:05.042290+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T12:57:05.042290+0000","last_undegraded":"2024-03-22T12:57:05.042290+0000","last_fullsized":"2024-03-22T12:57:05.042290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:29.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354571+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T12:57:04.354571+0000","last_peered":"2024-03-22T12:57:04.354571+0000","last_clean":"2024-03-22T12:57:04.354571+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T12:57:04.354571+0000","last_undegraded":"2024-03-22T12:57:04.354571+0000","last_fullsized":"2024-03-22T12:57:04.354571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946567+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T12:57:04.946567+0000","last_peered":"2024-03-22T12:57:04.946567+0000","last_clean":"2024-03-22T12:57:04.946567+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T12:57:04.946567+0000","last_undegraded":"2024-03-22T12:57:04.946567+0000","last_fullsized":"2024-03-22T12:57:04.946567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042408+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T12:57:05.042408+0000","last_peered":"2024-03-22T12:57:05.042408+0000","last_clean":"2024-03-22T12:57:05.042408+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T12:57:05.042408+0000","last_undegraded":"2024-03-22T12:57:05.042408+0000","last_fullsized":"2024-03-22T12:57:05.042408+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247053+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T12:57:04.247053+0000","last_peered":"2024-03-22T12:57:04.247053+0000","last_clean":"2024-03-22T12:57:04.247053+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T12:57:04.247053+0000","last_undegraded":"2024-03-22T12:57:04.247053+0000","last_fullsized":"2024-03-22T12:57:04.247053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.305489+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T12:57:12.305489+0000","last_peered":"2024-03-22T12:57:12.305489+0000","last_clean":"2024-03-22T12:57:12.305489+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T12:57:12.305489+0000","last_undegraded":"2024-03-22T12:57:12.305489+0000","last_fullsized":"2024-03-22T12:57:12.305489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042319+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T12:57:05.042319+0000","last_peered":"2024-03-22T12:57:05.042319+0000","last_clean":"2024-03-22T12:57:05.042319+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T12:57:05.042319+0000","last_undegraded":"2024-03-22T12:57:05.042319+0000","last_fullsized":"2024-03-22T12:57:05.042319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:29.130909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174615+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T12:57:04.174615+0000","last_peered":"2024-03-22T12:57:04.174615+0000","last_clean":"2024-03-22T12:57:04.174615+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T12:57:04.174615+0000","last_undegraded":"2024-03-22T12:57:04.174615+0000","last_fullsized":"2024-03-22T12:57:04.174615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185126+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T12:57:04.185126+0000","last_peered":"2024-03-22T12:57:04.185126+0000","last_clean":"2024-03-22T12:57:04.185126+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T12:57:04.185126+0000","last_undegraded":"2024-03-22T12:57:04.185126+0000","last_fullsized":"2024-03-22T12:57:04.185126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042296+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T12:57:05.042296+0000","last_peered":"2024-03-22T12:57:05.042296+0000","last_clean":"2024-03-22T12:57:05.042296+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T12:57:05.042296+0000","last_undegraded":"2024-03-22T12:57:05.042296+0000","last_fullsized":"2024-03-22T12:57:05.042296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:40.776689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173279+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T12:57:04.173279+0000","last_peered":"2024-03-22T12:57:04.173279+0000","last_clean":"2024-03-22T12:57:04.173279+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T12:57:04.173279+0000","last_undegraded":"2024-03-22T12:57:04.173279+0000","last_fullsized":"2024-03-22T12:57:04.173279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.413103+0000","last_change":"2024-03-22T12:56:28.757253+0000","last_active":"2024-03-22T12:57:12.413103+0000","last_peered":"2024-03-22T12:57:12.413103+0000","last_clean":"2024-03-22T12:57:12.413103+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T12:57:12.413103+0000","last_undegraded":"2024-03-22T12:57:12.413103+0000","last_fullsized":"2024-03-22T12:57:12.413103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:48.506816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185414+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T12:57:04.185414+0000","last_peered":"2024-03-22T12:57:04.185414+0000","last_clean":"2024-03-22T12:57:04.185414+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T12:57:04.185414+0000","last_undegraded":"2024-03-22T12:57:04.185414+0000","last_fullsized":"2024-03-22T12:57:04.185414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042646+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T12:57:05.042646+0000","last_peered":"2024-03-22T12:57:05.042646+0000","last_clean":"2024-03-22T12:57:05.042646+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T12:57:05.042646+0000","last_undegraded":"2024-03-22T12:57:05.042646+0000","last_fullsized":"2024-03-22T12:57:05.042646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:34.508555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.306479+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T12:57:12.306479+0000","last_peered":"2024-03-22T12:57:12.306479+0000","last_clean":"2024-03-22T12:57:12.306479+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T12:57:12.306479+0000","last_undegraded":"2024-03-22T12:57:12.306479+0000","last_fullsized":"2024-03-22T12:57:12.306479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042323+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T12:57:05.042323+0000","last_peered":"2024-03-22T12:57:05.042323+0000","last_clean":"2024-03-22T12:57:05.042323+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T12:57:05.042323+0000","last_undegraded":"2024-03-22T12:57:05.042323+0000","last_fullsized":"2024-03-22T12:57:05.042323+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:09.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172429+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T12:57:04.172429+0000","last_peered":"2024-03-22T12:57:04.172429+0000","last_clean":"2024-03-22T12:57:04.172429+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T12:57:04.172429+0000","last_undegraded":"2024-03-22T12:57:04.172429+0000","last_fullsized":"2024-03-22T12:57:04.172429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.306102+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T12:57:12.306102+0000","last_peered":"2024-03-22T12:57:12.306102+0000","last_clean":"2024-03-22T12:57:12.306102+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T12:57:12.306102+0000","last_undegraded":"2024-03-22T12:57:12.306102+0000","last_fullsized":"2024-03-22T12:57:12.306102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042166+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T12:57:05.042166+0000","last_peered":"2024-03-22T12:57:05.042166+0000","last_clean":"2024-03-22T12:57:05.042166+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T12:57:05.042166+0000","last_undegraded":"2024-03-22T12:57:05.042166+0000","last_fullsized":"2024-03-22T12:57:05.042166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:13.307688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176050+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T12:57:04.176050+0000","last_peered":"2024-03-22T12:57:04.176050+0000","last_clean":"2024-03-22T12:57:04.176050+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T12:57:04.176050+0000","last_undegraded":"2024-03-22T12:57:04.176050+0000","last_fullsized":"2024-03-22T12:57:04.176050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:06.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185611+0000","last_change":"2024-03-22T12:56:28.767925+0000","last_active":"2024-03-22T12:57:04.185611+0000","last_peered":"2024-03-22T12:57:04.185611+0000","last_clean":"2024-03-22T12:57:04.185611+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T12:57:04.185611+0000","last_undegraded":"2024-03-22T12:57:04.185611+0000","last_fullsized":"2024-03-22T12:57:04.185611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172825+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T12:57:04.172825+0000","last_peered":"2024-03-22T12:57:04.172825+0000","last_clean":"2024-03-22T12:57:04.172825+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T12:57:04.172825+0000","last_undegraded":"2024-03-22T12:57:04.172825+0000","last_fullsized":"2024-03-22T12:57:04.172825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:40:09.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185111+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T12:57:04.185111+0000","last_peered":"2024-03-22T12:57:04.185111+0000","last_clean":"2024-03-22T12:57:04.185111+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T12:57:04.185111+0000","last_undegraded":"2024-03-22T12:57:04.185111+0000","last_fullsized":"2024-03-22T12:57:04.185111+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355281+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T12:57:04.355281+0000","last_peered":"2024-03-22T12:57:04.355281+0000","last_clean":"2024-03-22T12:57:04.355281+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T12:57:04.355281+0000","last_undegraded":"2024-03-22T12:57:04.355281+0000","last_fullsized":"2024-03-22T12:57:04.355281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170966+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T12:57:04.170966+0000","last_peered":"2024-03-22T12:57:04.170966+0000","last_clean":"2024-03-22T12:57:04.170966+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T12:57:04.170966+0000","last_undegraded":"2024-03-22T12:57:04.170966+0000","last_fullsized":"2024-03-22T12:57:04.170966+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946629+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T12:57:04.946629+0000","last_peered":"2024-03-22T12:57:04.946629+0000","last_clean":"2024-03-22T12:57:04.946629+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T12:57:04.946629+0000","last_undegraded":"2024-03-22T12:57:04.946629+0000","last_fullsized":"2024-03-22T12:57:04.946629+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185202+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T12:57:04.185202+0000","last_peered":"2024-03-22T12:57:04.185202+0000","last_clean":"2024-03-22T12:57:04.185202+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T12:57:04.185202+0000","last_undegraded":"2024-03-22T12:57:04.185202+0000","last_fullsized":"2024-03-22T12:57:04.185202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045655+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T12:57:05.045655+0000","last_peered":"2024-03-22T12:57:05.045655+0000","last_clean":"2024-03-22T12:57:05.045655+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T12:57:05.045655+0000","last_undegraded":"2024-03-22T12:57:05.045655+0000","last_fullsized":"2024-03-22T12:57:05.045655+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:45.197847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042741+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T12:57:05.042741+0000","last_peered":"2024-03-22T12:57:05.042741+0000","last_clean":"2024-03-22T12:57:05.042741+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T12:57:05.042741+0000","last_undegraded":"2024-03-22T12:57:05.042741+0000","last_fullsized":"2024-03-22T12:57:05.042741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:23.839356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946541+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T12:57:04.946541+0000","last_peered":"2024-03-22T12:57:04.946541+0000","last_clean":"2024-03-22T12:57:04.946541+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T12:57:04.946541+0000","last_undegraded":"2024-03-22T12:57:04.946541+0000","last_fullsized":"2024-03-22T12:57:04.946541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184845+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T12:57:04.184845+0000","last_peered":"2024-03-22T12:57:04.184845+0000","last_clean":"2024-03-22T12:57:04.184845+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T12:57:04.184845+0000","last_undegraded":"2024-03-22T12:57:04.184845+0000","last_fullsized":"2024-03-22T12:57:04.184845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184792+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T12:57:04.184792+0000","last_peered":"2024-03-22T12:57:04.184792+0000","last_clean":"2024-03-22T12:57:04.184792+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T12:57:04.184792+0000","last_undegraded":"2024-03-22T12:57:04.184792+0000","last_fullsized":"2024-03-22T12:57:04.184792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.179893+0000","last_change":"2024-03-22T12:56:28.763384+0000","last_active":"2024-03-22T12:57:04.179893+0000","last_peered":"2024-03-22T12:57:04.179893+0000","last_clean":"2024-03-22T12:57:04.179893+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T12:57:04.179893+0000","last_undegraded":"2024-03-22T12:57:04.179893+0000","last_fullsized":"2024-03-22T12:57:04.179893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:15:45.463056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.952755+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T12:57:04.952755+0000","last_peered":"2024-03-22T12:57:04.952755+0000","last_clean":"2024-03-22T12:57:04.952755+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T12:57:04.952755+0000","last_undegraded":"2024-03-22T12:57:04.952755+0000","last_fullsized":"2024-03-22T12:57:04.952755+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:48.651995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.169659+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T12:57:04.169659+0000","last_peered":"2024-03-22T12:57:04.169659+0000","last_clean":"2024-03-22T12:57:04.169659+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T12:57:04.169659+0000","last_undegraded":"2024-03-22T12:57:04.169659+0000","last_fullsized":"2024-03-22T12:57:04.169659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:36:50.048241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354403+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T12:57:04.354403+0000","last_peered":"2024-03-22T12:57:04.354403+0000","last_clean":"2024-03-22T12:57:04.354403+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T12:57:04.354403+0000","last_undegraded":"2024-03-22T12:57:04.354403+0000","last_fullsized":"2024-03-22T12:57:04.354403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172286+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T12:57:04.172286+0000","last_peered":"2024-03-22T12:57:04.172286+0000","last_clean":"2024-03-22T12:57:04.172286+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T12:57:04.172286+0000","last_undegraded":"2024-03-22T12:57:04.172286+0000","last_fullsized":"2024-03-22T12:57:04.172286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:29:31.400078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174048+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T12:57:04.174048+0000","last_peered":"2024-03-22T12:57:04.174048+0000","last_clean":"2024-03-22T12:57:04.174048+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T12:57:04.174048+0000","last_undegraded":"2024-03-22T12:57:04.174048+0000","last_fullsized":"2024-03-22T12:57:04.174048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.459550+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T12:57:12.459550+0000","last_peered":"2024-03-22T12:57:12.459550+0000","last_clean":"2024-03-22T12:57:12.459550+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T12:57:12.459550+0000","last_undegraded":"2024-03-22T12:57:12.459550+0000","last_fullsized":"2024-03-22T12:57:12.459550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354430+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T12:57:04.354430+0000","last_peered":"2024-03-22T12:57:04.354430+0000","last_clean":"2024-03-22T12:57:04.354430+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T12:57:04.354430+0000","last_undegraded":"2024-03-22T12:57:04.354430+0000","last_fullsized":"2024-03-22T12:57:04.354430+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247232+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T12:57:04.247232+0000","last_peered":"2024-03-22T12:57:04.247232+0000","last_clean":"2024-03-22T12:57:04.247232+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T12:57:04.247232+0000","last_undegraded":"2024-03-22T12:57:04.247232+0000","last_fullsized":"2024-03-22T12:57:04.247232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.181294+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T12:57:04.181294+0000","last_peered":"2024-03-22T12:57:04.181294+0000","last_clean":"2024-03-22T12:57:04.181294+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T12:57:04.181294+0000","last_undegraded":"2024-03-22T12:57:04.181294+0000","last_fullsized":"2024-03-22T12:57:04.181294+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247035+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T12:57:04.247035+0000","last_peered":"2024-03-22T12:57:04.247035+0000","last_clean":"2024-03-22T12:57:04.247035+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T12:57:04.247035+0000","last_undegraded":"2024-03-22T12:57:04.247035+0000","last_fullsized":"2024-03-22T12:57:04.247035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946466+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T12:57:04.946466+0000","last_peered":"2024-03-22T12:57:04.946466+0000","last_clean":"2024-03-22T12:57:04.946466+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T12:57:04.946466+0000","last_undegraded":"2024-03-22T12:57:04.946466+0000","last_fullsized":"2024-03-22T12:57:04.946466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.186063+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T12:57:04.186063+0000","last_peered":"2024-03-22T12:57:04.186063+0000","last_clean":"2024-03-22T12:57:04.186063+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T12:57:04.186063+0000","last_undegraded":"2024-03-22T12:57:04.186063+0000","last_fullsized":"2024-03-22T12:57:04.186063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:22.329365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171343+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T12:57:04.171343+0000","last_peered":"2024-03-22T12:57:04.171343+0000","last_clean":"2024-03-22T12:57:04.171343+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T12:57:04.171343+0000","last_undegraded":"2024-03-22T12:57:04.171343+0000","last_fullsized":"2024-03-22T12:57:04.171343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355029+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T12:57:04.355029+0000","last_peered":"2024-03-22T12:57:04.355029+0000","last_clean":"2024-03-22T12:57:04.355029+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T12:57:04.355029+0000","last_undegraded":"2024-03-22T12:57:04.355029+0000","last_fullsized":"2024-03-22T12:57:04.355029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.182677+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T12:57:04.182677+0000","last_peered":"2024-03-22T12:57:04.182677+0000","last_clean":"2024-03-22T12:57:04.182677+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T12:57:04.182677+0000","last_undegraded":"2024-03-22T12:57:04.182677+0000","last_fullsized":"2024-03-22T12:57:04.182677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946109+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T12:57:04.946109+0000","last_peered":"2024-03-22T12:57:04.946109+0000","last_clean":"2024-03-22T12:57:04.946109+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T12:57:04.946109+0000","last_undegraded":"2024-03-22T12:57:04.946109+0000","last_fullsized":"2024-03-22T12:57:04.946109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042539+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T12:57:05.042539+0000","last_peered":"2024-03-22T12:57:05.042539+0000","last_clean":"2024-03-22T12:57:05.042539+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T12:57:05.042539+0000","last_undegraded":"2024-03-22T12:57:05.042539+0000","last_fullsized":"2024-03-22T12:57:05.042539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:24:46.021828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042059+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T12:57:05.042059+0000","last_peered":"2024-03-22T12:57:05.042059+0000","last_clean":"2024-03-22T12:57:05.042059+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T12:57:05.042059+0000","last_undegraded":"2024-03-22T12:57:05.042059+0000","last_fullsized":"2024-03-22T12:57:05.042059+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:38.316319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354615+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T12:57:04.354615+0000","last_peered":"2024-03-22T12:57:04.354615+0000","last_clean":"2024-03-22T12:57:04.354615+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T12:57:04.354615+0000","last_undegraded":"2024-03-22T12:57:04.354615+0000","last_fullsized":"2024-03-22T12:57:04.354615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'10","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.403151+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T12:57:12.403151+0000","last_peered":"2024-03-22T12:57:12.403151+0000","last_clean":"2024-03-22T12:57:12.403151+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T12:57:12.403151+0000","last_undegraded":"2024-03-22T12:57:12.403151+0000","last_fullsized":"2024-03-22T12:57:12.403151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:20:57.722062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172194+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T12:57:04.172194+0000","last_peered":"2024-03-22T12:57:04.172194+0000","last_clean":"2024-03-22T12:57:04.172194+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T12:57:04.172194+0000","last_undegraded":"2024-03-22T12:57:04.172194+0000","last_fullsized":"2024-03-22T12:57:04.172194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247088+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T12:57:04.247088+0000","last_peered":"2024-03-22T12:57:04.247088+0000","last_clean":"2024-03-22T12:57:04.247088+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T12:57:04.247088+0000","last_undegraded":"2024-03-22T12:57:04.247088+0000","last_fullsized":"2024-03-22T12:57:04.247088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247103+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T12:57:04.247103+0000","last_peered":"2024-03-22T12:57:04.247103+0000","last_clean":"2024-03-22T12:57:04.247103+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T12:57:04.247103+0000","last_undegraded":"2024-03-22T12:57:04.247103+0000","last_fullsized":"2024-03-22T12:57:04.247103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176522+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T12:57:04.176522+0000","last_peered":"2024-03-22T12:57:04.176522+0000","last_clean":"2024-03-22T12:57:04.176522+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T12:57:04.176522+0000","last_undegraded":"2024-03-22T12:57:04.176522+0000","last_fullsized":"2024-03-22T12:57:04.176522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:51.788457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354739+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T12:57:04.354739+0000","last_peered":"2024-03-22T12:57:04.354739+0000","last_clean":"2024-03-22T12:57:04.354739+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T12:57:04.354739+0000","last_undegraded":"2024-03-22T12:57:04.354739+0000","last_fullsized":"2024-03-22T12:57:04.354739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.459262+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T12:57:12.459262+0000","last_peered":"2024-03-22T12:57:12.459262+0000","last_clean":"2024-03-22T12:57:12.459262+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T12:57:12.459262+0000","last_undegraded":"2024-03-22T12:57:12.459262+0000","last_fullsized":"2024-03-22T12:57:12.459262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.949816+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T12:57:04.949816+0000","last_peered":"2024-03-22T12:57:04.949816+0000","last_clean":"2024-03-22T12:57:04.949816+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T12:57:04.949816+0000","last_undegraded":"2024-03-22T12:57:04.949816+0000","last_fullsized":"2024-03-22T12:57:04.949816+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:08:12.860933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170220+0000","last_change":"2024-03-22T12:56:30.299969+0000","last_active":"2024-03-22T12:57:04.170220+0000","last_peered":"2024-03-22T12:57:04.170220+0000","last_clean":"2024-03-22T12:57:04.170220+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T12:57:04.170220+0000","last_undegraded":"2024-03-22T12:57:04.170220+0000","last_fullsized":"2024-03-22T12:57:04.170220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.300624+0000","last_change":"2024-03-22T12:56:28.777142+0000","last_active":"2024-03-22T12:57:12.300624+0000","last_peered":"2024-03-22T12:57:12.300624+0000","last_clean":"2024-03-22T12:57:12.300624+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T12:57:12.300624+0000","last_undegraded":"2024-03-22T12:57:12.300624+0000","last_fullsized":"2024-03-22T12:57:12.300624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:49.411808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946528+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T12:57:04.946528+0000","last_peered":"2024-03-22T12:57:04.946528+0000","last_clean":"2024-03-22T12:57:04.946528+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T12:57:04.946528+0000","last_undegraded":"2024-03-22T12:57:04.946528+0000","last_fullsized":"2024-03-22T12:57:04.946528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354901+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T12:57:04.354901+0000","last_peered":"2024-03-22T12:57:04.354901+0000","last_clean":"2024-03-22T12:57:04.354901+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T12:57:04.354901+0000","last_undegraded":"2024-03-22T12:57:04.354901+0000","last_fullsized":"2024-03-22T12:57:04.354901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'12","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.308529+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T12:57:12.308529+0000","last_peered":"2024-03-22T12:57:12.308529+0000","last_clean":"2024-03-22T12:57:12.308529+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T12:57:12.308529+0000","last_undegraded":"2024-03-22T12:57:12.308529+0000","last_fullsized":"2024-03-22T12:57:12.308529+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184655+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T12:57:04.184655+0000","last_peered":"2024-03-22T12:57:04.184655+0000","last_clean":"2024-03-22T12:57:04.184655+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T12:57:04.184655+0000","last_undegraded":"2024-03-22T12:57:04.184655+0000","last_fullsized":"2024-03-22T12:57:04.184655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042567+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T12:57:05.042567+0000","last_peered":"2024-03-22T12:57:05.042567+0000","last_clean":"2024-03-22T12:57:05.042567+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T12:57:05.042567+0000","last_undegraded":"2024-03-22T12:57:05.042567+0000","last_fullsized":"2024-03-22T12:57:05.042567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:49.776407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.186011+0000","last_change":"2024-03-22T12:56:28.769041+0000","last_active":"2024-03-22T12:57:04.186011+0000","last_peered":"2024-03-22T12:57:04.186011+0000","last_clean":"2024-03-22T12:57:04.186011+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T12:57:04.186011+0000","last_undegraded":"2024-03-22T12:57:04.186011+0000","last_fullsized":"2024-03-22T12:57:04.186011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042461+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T12:57:05.042461+0000","last_peered":"2024-03-22T12:57:05.042461+0000","last_clean":"2024-03-22T12:57:05.042461+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T12:57:05.042461+0000","last_undegraded":"2024-03-22T12:57:05.042461+0000","last_fullsized":"2024-03-22T12:57:05.042461+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:56.815720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172275+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T12:57:04.172275+0000","last_peered":"2024-03-22T12:57:04.172275+0000","last_clean":"2024-03-22T12:57:04.172275+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T12:57:04.172275+0000","last_undegraded":"2024-03-22T12:57:04.172275+0000","last_fullsized":"2024-03-22T12:57:04.172275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.457584+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T12:57:12.457584+0000","last_peered":"2024-03-22T12:57:12.457584+0000","last_clean":"2024-03-22T12:57:12.457584+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T12:57:12.457584+0000","last_undegraded":"2024-03-22T12:57:12.457584+0000","last_fullsized":"2024-03-22T12:57:12.457584+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173079+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T12:57:04.173079+0000","last_peered":"2024-03-22T12:57:04.173079+0000","last_clean":"2024-03-22T12:57:04.173079+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T12:57:04.173079+0000","last_undegraded":"2024-03-22T12:57:04.173079+0000","last_fullsized":"2024-03-22T12:57:04.173079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:06:47.716384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946555+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T12:57:04.946555+0000","last_peered":"2024-03-22T12:57:04.946555+0000","last_clean":"2024-03-22T12:57:04.946555+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T12:57:04.946555+0000","last_undegraded":"2024-03-22T12:57:04.946555+0000","last_fullsized":"2024-03-22T12:57:04.946555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.307509+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T12:57:12.307509+0000","last_peered":"2024-03-22T12:57:12.307509+0000","last_clean":"2024-03-22T12:57:12.307509+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T12:57:12.307509+0000","last_undegraded":"2024-03-22T12:57:12.307509+0000","last_fullsized":"2024-03-22T12:57:12.307509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174378+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T12:57:04.174378+0000","last_peered":"2024-03-22T12:57:04.174378+0000","last_clean":"2024-03-22T12:57:04.174378+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T12:57:04.174378+0000","last_undegraded":"2024-03-22T12:57:04.174378+0000","last_fullsized":"2024-03-22T12:57:04.174378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172090+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T12:57:04.172090+0000","last_peered":"2024-03-22T12:57:04.172090+0000","last_clean":"2024-03-22T12:57:04.172090+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T12:57:04.172090+0000","last_undegraded":"2024-03-22T12:57:04.172090+0000","last_fullsized":"2024-03-22T12:57:04.172090+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184400+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T12:57:04.184400+0000","last_peered":"2024-03-22T12:57:04.184400+0000","last_clean":"2024-03-22T12:57:04.184400+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T12:57:04.184400+0000","last_undegraded":"2024-03-22T12:57:04.184400+0000","last_fullsized":"2024-03-22T12:57:04.184400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946306+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T12:57:04.946306+0000","last_peered":"2024-03-22T12:57:04.946306+0000","last_clean":"2024-03-22T12:57:04.946306+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T12:57:04.946306+0000","last_undegraded":"2024-03-22T12:57:04.946306+0000","last_fullsized":"2024-03-22T12:57:04.946306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247413+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T12:57:04.247413+0000","last_peered":"2024-03-22T12:57:04.247413+0000","last_clean":"2024-03-22T12:57:04.247413+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T12:57:04.247413+0000","last_undegraded":"2024-03-22T12:57:04.247413+0000","last_fullsized":"2024-03-22T12:57:04.247413+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173181+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T12:57:04.173181+0000","last_peered":"2024-03-22T12:57:04.173181+0000","last_clean":"2024-03-22T12:57:04.173181+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T12:57:04.173181+0000","last_undegraded":"2024-03-22T12:57:04.173181+0000","last_fullsized":"2024-03-22T12:57:04.173181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174556+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T12:57:04.174556+0000","last_peered":"2024-03-22T12:57:04.174556+0000","last_clean":"2024-03-22T12:57:04.174556+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T12:57:04.174556+0000","last_undegraded":"2024-03-22T12:57:04.174556+0000","last_fullsized":"2024-03-22T12:57:04.174556+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:31:41.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"24'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247241+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T12:57:04.247241+0000","last_peered":"2024-03-22T12:57:04.247241+0000","last_clean":"2024-03-22T12:57:04.247241+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T12:57:04.247241+0000","last_undegraded":"2024-03-22T12:57:04.247241+0000","last_fullsized":"2024-03-22T12:57:04.247241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.047312+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T12:57:05.047312+0000","last_peered":"2024-03-22T12:57:05.047312+0000","last_clean":"2024-03-22T12:57:05.047312+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T12:57:05.047312+0000","last_undegraded":"2024-03-22T12:57:05.047312+0000","last_fullsized":"2024-03-22T12:57:05.047312+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.180347+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T12:57:04.180347+0000","last_peered":"2024-03-22T12:57:04.180347+0000","last_clean":"2024-03-22T12:57:04.180347+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T12:57:04.180347+0000","last_undegraded":"2024-03-22T12:57:04.180347+0000","last_fullsized":"2024-03-22T12:57:04.180347+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:43.256977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.481695+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T12:57:12.481695+0000","last_peered":"2024-03-22T12:57:12.481695+0000","last_clean":"2024-03-22T12:57:12.481695+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T12:57:12.481695+0000","last_undegraded":"2024-03-22T12:57:12.481695+0000","last_fullsized":"2024-03-22T12:57:12.481695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042643+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T12:57:05.042643+0000","last_peered":"2024-03-22T12:57:05.042643+0000","last_clean":"2024-03-22T12:57:05.042643+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T12:57:05.042643+0000","last_undegraded":"2024-03-22T12:57:05.042643+0000","last_fullsized":"2024-03-22T12:57:05.042643+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:48.882251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174116+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T12:57:04.174116+0000","last_peered":"2024-03-22T12:57:04.174116+0000","last_clean":"2024-03-22T12:57:04.174116+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T12:57:04.174116+0000","last_undegraded":"2024-03-22T12:57:04.174116+0000","last_fullsized":"2024-03-22T12:57:04.174116+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247032+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T12:57:04.247032+0000","last_peered":"2024-03-22T12:57:04.247032+0000","last_clean":"2024-03-22T12:57:04.247032+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T12:57:04.247032+0000","last_undegraded":"2024-03-22T12:57:04.247032+0000","last_fullsized":"2024-03-22T12:57:04.247032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.188189+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T12:57:04.188189+0000","last_peered":"2024-03-22T12:57:04.188189+0000","last_clean":"2024-03-22T12:57:04.188189+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T12:57:04.188189+0000","last_undegraded":"2024-03-22T12:57:04.188189+0000","last_fullsized":"2024-03-22T12:57:04.188189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946357+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T12:57:04.946357+0000","last_peered":"2024-03-22T12:57:04.946357+0000","last_clean":"2024-03-22T12:57:04.946357+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T12:57:04.946357+0000","last_undegraded":"2024-03-22T12:57:04.946357+0000","last_fullsized":"2024-03-22T12:57:04.946357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171397+0000","last_change":"2024-03-22T12:56:30.300244+0000","last_active":"2024-03-22T12:57:04.171397+0000","last_peered":"2024-03-22T12:57:04.171397+0000","last_clean":"2024-03-22T12:57:04.171397+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T12:57:04.171397+0000","last_undegraded":"2024-03-22T12:57:04.171397+0000","last_fullsized":"2024-03-22T12:57:04.171397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174764+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T12:57:04.174764+0000","last_peered":"2024-03-22T12:57:04.174764+0000","last_clean":"2024-03-22T12:57:04.174764+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T12:57:04.174764+0000","last_undegraded":"2024-03-22T12:57:04.174764+0000","last_fullsized":"2024-03-22T12:57:04.174764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185804+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T12:57:04.185804+0000","last_peered":"2024-03-22T12:57:04.185804+0000","last_clean":"2024-03-22T12:57:04.185804+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T12:57:04.185804+0000","last_undegraded":"2024-03-22T12:57:04.185804+0000","last_fullsized":"2024-03-22T12:57:04.185804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247526+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T12:57:04.247526+0000","last_peered":"2024-03-22T12:57:04.247526+0000","last_clean":"2024-03-22T12:57:04.247526+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T12:57:04.247526+0000","last_undegraded":"2024-03-22T12:57:04.247526+0000","last_fullsized":"2024-03-22T12:57:04.247526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.175763+0000","last_change":"2024-03-22T12:56:30.308903+0000","last_active":"2024-03-22T12:57:04.175763+0000","last_peered":"2024-03-22T12:57:04.175763+0000","last_clean":"2024-03-22T12:57:04.175763+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T12:57:04.175763+0000","last_undegraded":"2024-03-22T12:57:04.175763+0000","last_fullsized":"2024-03-22T12:57:04.175763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185929+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T12:57:04.185929+0000","last_peered":"2024-03-22T12:57:04.185929+0000","last_clean":"2024-03-22T12:57:04.185929+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T12:57:04.185929+0000","last_undegraded":"2024-03-22T12:57:04.185929+0000","last_fullsized":"2024-03-22T12:57:04.185929+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:13:57.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045995+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T12:57:05.045995+0000","last_peered":"2024-03-22T12:57:05.045995+0000","last_clean":"2024-03-22T12:57:05.045995+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T12:57:05.045995+0000","last_undegraded":"2024-03-22T12:57:05.045995+0000","last_fullsized":"2024-03-22T12:57:05.045995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:19.797978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946253+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T12:57:04.946253+0000","last_peered":"2024-03-22T12:57:04.946253+0000","last_clean":"2024-03-22T12:57:04.946253+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T12:57:04.946253+0000","last_undegraded":"2024-03-22T12:57:04.946253+0000","last_fullsized":"2024-03-22T12:57:04.946253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171180+0000","last_change":"2024-03-22T12:56:30.299214+0000","last_active":"2024-03-22T12:57:04.171180+0000","last_peered":"2024-03-22T12:57:04.171180+0000","last_clean":"2024-03-22T12:57:04.171180+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T12:57:04.171180+0000","last_undegraded":"2024-03-22T12:57:04.171180+0000","last_fullsized":"2024-03-22T12:57:04.171180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:35.356723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247391+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T12:57:04.247391+0000","last_peered":"2024-03-22T12:57:04.247391+0000","last_clean":"2024-03-22T12:57:04.247391+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T12:57:04.247391+0000","last_undegraded":"2024-03-22T12:57:04.247391+0000","last_fullsized":"2024-03-22T12:57:04.247391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.189179+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T12:57:04.189179+0000","last_peered":"2024-03-22T12:57:04.189179+0000","last_clean":"2024-03-22T12:57:04.189179+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T12:57:04.189179+0000","last_undegraded":"2024-03-22T12:57:04.189179+0000","last_fullsized":"2024-03-22T12:57:04.189179+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:57:33.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.182435+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T12:57:04.182435+0000","last_peered":"2024-03-22T12:57:04.182435+0000","last_clean":"2024-03-22T12:57:04.182435+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T12:57:04.182435+0000","last_undegraded":"2024-03-22T12:57:04.182435+0000","last_fullsized":"2024-03-22T12:57:04.182435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:38.987151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946819+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T12:57:04.946819+0000","last_peered":"2024-03-22T12:57:04.946819+0000","last_clean":"2024-03-22T12:57:04.946819+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T12:57:04.946819+0000","last_undegraded":"2024-03-22T12:57:04.946819+0000","last_fullsized":"2024-03-22T12:57:04.946819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946220+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T12:57:04.946220+0000","last_peered":"2024-03-22T12:57:04.946220+0000","last_clean":"2024-03-22T12:57:04.946220+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T12:57:04.946220+0000","last_undegraded":"2024-03-22T12:57:04.946220+0000","last_fullsized":"2024-03-22T12:57:04.946220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184477+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T12:57:04.184477+0000","last_peered":"2024-03-22T12:57:04.184477+0000","last_clean":"2024-03-22T12:57:04.184477+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T12:57:04.184477+0000","last_undegraded":"2024-03-22T12:57:04.184477+0000","last_fullsized":"2024-03-22T12:57:04.184477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170174+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T12:57:04.170174+0000","last_peered":"2024-03-22T12:57:04.170174+0000","last_clean":"2024-03-22T12:57:04.170174+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T12:57:04.170174+0000","last_undegraded":"2024-03-22T12:57:04.170174+0000","last_fullsized":"2024-03-22T12:57:04.170174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174348+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T12:57:04.174348+0000","last_peered":"2024-03-22T12:57:04.174348+0000","last_clean":"2024-03-22T12:57:04.174348+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T12:57:04.174348+0000","last_undegraded":"2024-03-22T12:57:04.174348+0000","last_fullsized":"2024-03-22T12:57:04.174348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042563+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T12:57:05.042563+0000","last_peered":"2024-03-22T12:57:05.042563+0000","last_clean":"2024-03-22T12:57:05.042563+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T12:57:05.042563+0000","last_undegraded":"2024-03-22T12:57:05.042563+0000","last_fullsized":"2024-03-22T12:57:05.042563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:23.598956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.251063+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T12:57:04.251063+0000","last_peered":"2024-03-22T12:57:04.251063+0000","last_clean":"2024-03-22T12:57:04.251063+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T12:57:04.251063+0000","last_undegraded":"2024-03-22T12:57:04.251063+0000","last_fullsized":"2024-03-22T12:57:04.251063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247414+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T12:57:04.247414+0000","last_peered":"2024-03-22T12:57:04.247414+0000","last_clean":"2024-03-22T12:57:04.247414+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T12:57:04.247414+0000","last_undegraded":"2024-03-22T12:57:04.247414+0000","last_fullsized":"2024-03-22T12:57:04.247414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355010+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T12:57:04.355010+0000","last_peered":"2024-03-22T12:57:04.355010+0000","last_clean":"2024-03-22T12:57:04.355010+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T12:57:04.355010+0000","last_undegraded":"2024-03-22T12:57:04.355010+0000","last_fullsized":"2024-03-22T12:57:04.355010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172977+0000","last_change":"2024-03-22T12:56:30.309258+0000","last_active":"2024-03-22T12:57:04.172977+0000","last_peered":"2024-03-22T12:57:04.172977+0000","last_clean":"2024-03-22T12:57:04.172977+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T12:57:04.172977+0000","last_undegraded":"2024-03-22T12:57:04.172977+0000","last_fullsized":"2024-03-22T12:57:04.172977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354578+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T12:57:04.354578+0000","last_peered":"2024-03-22T12:57:04.354578+0000","last_clean":"2024-03-22T12:57:04.354578+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T12:57:04.354578+0000","last_undegraded":"2024-03-22T12:57:04.354578+0000","last_fullsized":"2024-03-22T12:57:04.354578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946219+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T12:57:04.946219+0000","last_peered":"2024-03-22T12:57:04.946219+0000","last_clean":"2024-03-22T12:57:04.946219+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T12:57:04.946219+0000","last_undegraded":"2024-03-22T12:57:04.946219+0000","last_fullsized":"2024-03-22T12:57:04.946219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354584+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T12:57:04.354584+0000","last_peered":"2024-03-22T12:57:04.354584+0000","last_clean":"2024-03-22T12:57:04.354584+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T12:57:04.354584+0000","last_undegraded":"2024-03-22T12:57:04.354584+0000","last_fullsized":"2024-03-22T12:57:04.354584+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946376+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T12:57:04.946376+0000","last_peered":"2024-03-22T12:57:04.946376+0000","last_clean":"2024-03-22T12:57:04.946376+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T12:57:04.946376+0000","last_undegraded":"2024-03-22T12:57:04.946376+0000","last_fullsized":"2024-03-22T12:57:04.946376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171812+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T12:57:04.171812+0000","last_peered":"2024-03-22T12:57:04.171812+0000","last_clean":"2024-03-22T12:57:04.171812+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T12:57:04.171812+0000","last_undegraded":"2024-03-22T12:57:04.171812+0000","last_fullsized":"2024-03-22T12:57:04.171812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:14.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185524+0000","last_change":"2024-03-22T12:56:30.303254+0000","last_active":"2024-03-22T12:57:04.185524+0000","last_peered":"2024-03-22T12:57:04.185524+0000","last_clean":"2024-03-22T12:57:04.185524+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T12:57:04.185524+0000","last_undegraded":"2024-03-22T12:57:04.185524+0000","last_fullsized":"2024-03-22T12:57:04.185524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354841+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T12:57:04.354841+0000","last_peered":"2024-03-22T12:57:04.354841+0000","last_clean":"2024-03-22T12:57:04.354841+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T12:57:04.354841+0000","last_undegraded":"2024-03-22T12:57:04.354841+0000","last_fullsized":"2024-03-22T12:57:04.354841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509455,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":132541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509455,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":131905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":136243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":132611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":401408,"data_stored":722807,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":722751,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":135711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542159,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":131779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T12:57:16.727 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-22T12:57:16.727 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-22T12:57:16.727 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-22T12:57:16.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-22T12:57:16.728 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-22T12:57:16.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.957+0000 7f05ed600640 1 Processor -- start 2024-03-22T12:57:16.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.957+0000 7f05ed600640 1 -- start start 2024-03-22T12:57:16.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.959+0000 7f05ed600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f05e80cc380 0x7f05e806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.959+0000 7f05ed600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05e80ce8a0 0x7f05e815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.959+0000 7f05ed600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e80cee00 con 0x7f05e80cc760 2024-03-22T12:57:16.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.959+0000 7f05ed600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e80cf0c0 con 0x7f05e80cc380 2024-03-22T12:57:16.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.959+0000 7f05ed600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e80cf3a0 con 0x7f05e80ce8a0 2024-03-22T12:57:16.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.960+0000 7f05e7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05e80ce8a0 0x7f05e815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.960+0000 7f05e7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05e80ce8a0 0x7f05e815ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60766/0 (socket says 172.21.15.6:60766) 2024-03-22T12:57:16.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.960+0000 7f05e7400640 1 -- 172.21.15.6:0/3755743819 learned_addr learned my addr 172.21.15.6:0/3755743819 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:16.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.960+0000 7f05e7e00640 1 --2- 172.21.15.6:0/3755743819 >> v2:172.21.15.6:3300/0 conn(0x7f05e80cc380 0x7f05e806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.963+0000 7f05e7400640 1 -- 172.21.15.6:0/3755743819 >> v2:172.21.15.6:3300/0 conn(0x7f05e80cc380 msgr2=0x7f05e806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:16.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.963+0000 7f05e7400640 1 --2- 172.21.15.6:0/3755743819 >> v2:172.21.15.6:3300/0 conn(0x7f05e80cc380 0x7f05e806c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.963+0000 7f05e7400640 1 -- 172.21.15.6:0/3755743819 >> v1:172.21.15.6:6789/0 conn(0x7f05e80cc760 legacy=0x7f05e80cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.963+0000 7f05e7400640 1 -- 172.21.15.6:0/3755743819 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e80cf680 con 0x7f05e80ce8a0 2024-03-22T12:57:16.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.966+0000 7f05e7400640 1 --2- 172.21.15.6:0/3755743819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 0x7f05e815ae30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05e80cb7f0 tx=0x7f05d402b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1baa22e20f08195b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:16.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.969+0000 7f05e6a00640 1 -- 172.21.15.6:0/3755743819 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4034070 con 0x7f05e80ce8a0 2024-03-22T12:57:16.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.971+0000 7f05e6a00640 1 -- 172.21.15.6:0/3755743819 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d4030880 con 0x7f05e80ce8a0 2024-03-22T12:57:16.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.971+0000 7f05e6a00640 1 -- 172.21.15.6:0/3755743819 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4030c50 con 0x7f05e80ce8a0 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 -- 172.21.15.6:0/3755743819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 msgr2=0x7f05e815ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 --2- 172.21.15.6:0/3755743819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 0x7f05e815ae30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05e80cb7f0 tx=0x7f05d402b9c0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 -- 172.21.15.6:0/3755743819 shutdown_connections 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 --2- 172.21.15.6:0/3755743819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 0x7f05e815ae30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 --2- 172.21.15.6:0/3755743819 >> v2:172.21.15.6:3300/0 conn(0x7f05e80cc380 0x7f05e806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 -- 172.21.15.6:0/3755743819 >> 172.21.15.6:0/3755743819 conn(0x7f05e80b1480 msgr2=0x7f05e80b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 -- 172.21.15.6:0/3755743819 shutdown_connections 2024-03-22T12:57:16.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.972+0000 7f05ed600640 1 -- 172.21.15.6:0/3755743819 wait complete. 2024-03-22T12:57:16.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.974+0000 7f05ed600640 1 Processor -- start 2024-03-22T12:57:16.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.974+0000 7f05ed600640 1 -- start start 2024-03-22T12:57:16.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.975+0000 7f05ed600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05e80cc380 0x7f05e8152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.975+0000 7f05ed600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e80cc760 0x7f05e815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.976+0000 7f05ed600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 0x7f05e80b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:16.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.976+0000 7f05ed600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e81549d0 con 0x7f05e80cc760 2024-03-22T12:57:16.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.976+0000 7f05ed600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e815a910 con 0x7f05e80ce8a0 2024-03-22T12:57:16.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.976+0000 7f05ed600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e80ccb70 con 0x7f05e80cc380 2024-03-22T12:57:16.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.976+0000 7f05e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e80cc760 0x7f05e815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.976+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05e80cc380 0x7f05e8152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.976+0000 7f05ecc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 0x7f05e80b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:16.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.977+0000 7f05e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e80cc760 0x7f05e815b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33716/0 (socket says 172.21.15.6:33716) 2024-03-22T12:57:16.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.977+0000 7f05e7400640 1 -- 172.21.15.6:0/4136673952 learned_addr learned my addr 172.21.15.6:0/4136673952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:16.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.981+0000 7f05e7e00640 1 -- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 msgr2=0x7f05e80b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.981+0000 7f05e7e00640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 0x7f05e80b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.981+0000 7f05e7e00640 1 -- 172.21.15.6:0/4136673952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e80cc760 msgr2=0x7f05e815b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:16.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.981+0000 7f05e7e00640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e80cc760 0x7f05e815b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:16.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.981+0000 7f05e7e00640 1 -- 172.21.15.6:0/4136673952 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e815bd40 con 0x7f05e80cc380 2024-03-22T12:57:16.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.983+0000 7f05e7e00640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05e80cc380 0x7f05e8152d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f05d800e880 tx=0x7f05d800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:16.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.987+0000 7f05dfe00640 1 -- 172.21.15.6:0/4136673952 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8017070 con 0x7f05e80cc380 2024-03-22T12:57:16.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.987+0000 7f05dfe00640 1 -- 172.21.15.6:0/4136673952 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d800c940 con 0x7f05e80cc380 2024-03-22T12:57:16.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.987+0000 7f05dfe00640 1 -- 172.21.15.6:0/4136673952 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d800cd30 con 0x7f05e80cc380 2024-03-22T12:57:16.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.987+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e815cc80 con 0x7f05e80cc380 2024-03-22T12:57:16.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.987+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e815d240 con 0x7f05e80cc380 2024-03-22T12:57:16.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:16.989+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e8075650 con 0x7f05e80cc380 2024-03-22T12:57:17.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.008+0000 7f05dfe00640 1 -- 172.21.15.6:0/4136673952 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f05d801a070 con 0x7f05e80cc380 2024-03-22T12:57:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.009+0000 7f05dfe00640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f05c806bc50 0x7f05c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.010+0000 7f05e7400640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f05c806bc50 0x7f05c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.010+0000 7f05dfe00640 1 -- 172.21.15.6:0/4136673952 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f05d8057630 con 0x7f05e80cc380 2024-03-22T12:57:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.011+0000 7f05e7400640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f05c806bc50 0x7f05c806e110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f05e8152ae0 tx=0x7f05d403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:17.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.274+0000 7f05dfe00640 1 -- 172.21.15.6:0/4136673952 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f05d8059590 con 0x7f05e80cc380 2024-03-22T12:57:17.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.542+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f05e8068150 con 0x7f05e80cc380 2024-03-22T12:57:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.777+0000 7f05dfe00640 1 -- 172.21.15.6:0/4136673952 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+172579 (secure 0 0 0) 0x7f05d805d630 con 0x7f05e80cc380 2024-03-22T12:57:17.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f05c806bc50 msgr2=0x7f05c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f05c806bc50 0x7f05c806e110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f05e8152ae0 tx=0x7f05d403b040 comp rx=0 tx=0).stop 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05e80cc380 msgr2=0x7f05e8152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05e80cc380 0x7f05e8152d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f05d800e880 tx=0x7f05d800ed40 comp rx=0 tx=0).stop 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 shutdown_connections 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f05c806bc50 0x7f05c806e110 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e80ce8a0 0x7f05e80b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e80cc760 0x7f05e815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 --2- 172.21.15.6:0/4136673952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05e80cc380 0x7f05e8152d80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 >> 172.21.15.6:0/4136673952 conn(0x7f05e80b1480 msgr2=0x7f05e80cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 shutdown_connections 2024-03-22T12:57:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.782+0000 7f05ed600640 1 -- 172.21.15.6:0/4136673952 wait complete. 2024-03-22T12:57:17.802 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":2032190056},{"type":"v1","addr":"172.21.15.92:6801","nonce":2032190056}]},"active_addr":"172.21.15.92:6801/2032190056","active_change":"2024-03-22T12:56:01.315206+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":3154317806}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":1540272576}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":3278162427}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":2969647948}]}]} 2024-03-22T12:57:17.805 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-22T12:57:17.805 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-22T12:57:17.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:17.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.996+0000 7f1719e00640 1 Processor -- start 2024-03-22T12:57:17.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.996+0000 7f1719e00640 1 -- start start 2024-03-22T12:57:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.997+0000 7f1719e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17140cc640 0x7f171406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.998+0000 7f1719e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:17.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.998+0000 7f1719e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171406dd00 con 0x7f171415b7a0 2024-03-22T12:57:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.998+0000 7f1719e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171406d330 con 0x7f17140cc640 2024-03-22T12:57:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.998+0000 7f1719e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171406c440 con 0x7f17140cca20 2024-03-22T12:57:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.998+0000 7f1713e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.999+0000 7f1713e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57222/0 (socket says 172.21.15.6:57222) 2024-03-22T12:57:17.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:17.999+0000 7f1713e00640 1 -- 172.21.15.6:0/203537301 learned_addr learned my addr 172.21.15.6:0/203537301 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.001+0000 7f1713e00640 1 -- 172.21.15.6:0/203537301 >> v2:172.21.15.6:3300/0 conn(0x7f17140cc640 msgr2=0x7f171406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.001+0000 7f1713e00640 1 --2- 172.21.15.6:0/203537301 >> v2:172.21.15.6:3300/0 conn(0x7f17140cc640 0x7f171406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.001+0000 7f1713e00640 1 -- 172.21.15.6:0/203537301 >> v1:172.21.15.6:6789/0 conn(0x7f171415b7a0 legacy=0x7f171415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:18.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.001+0000 7f1713e00640 1 -- 172.21.15.6:0/203537301 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171406c720 con 0x7f17140cca20 2024-03-22T12:57:18.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.003+0000 7f1713e00640 1 --2- 172.21.15.6:0/203537301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ca340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1700009870 tx=0x7f170002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=447b09484151d3a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.008+0000 7f1713400640 1 -- 172.21.15.6:0/203537301 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700034070 con 0x7f17140cca20 2024-03-22T12:57:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.008+0000 7f1713400640 1 -- 172.21.15.6:0/203537301 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700030840 con 0x7f17140cca20 2024-03-22T12:57:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.008+0000 7f1713400640 1 -- 172.21.15.6:0/203537301 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700030c10 con 0x7f17140cca20 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 -- 172.21.15.6:0/203537301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 msgr2=0x7f17140ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 --2- 172.21.15.6:0/203537301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ca340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1700009870 tx=0x7f170002b980 comp rx=0 tx=0).stop 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 -- 172.21.15.6:0/203537301 shutdown_connections 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 --2- 172.21.15.6:0/203537301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ca340 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 --2- 172.21.15.6:0/203537301 >> v2:172.21.15.6:3300/0 conn(0x7f17140cc640 0x7f171406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 -- 172.21.15.6:0/203537301 >> 172.21.15.6:0/203537301 conn(0x7f17140cf9c0 msgr2=0x7f17140cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 -- 172.21.15.6:0/203537301 shutdown_connections 2024-03-22T12:57:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.009+0000 7f1719e00640 1 -- 172.21.15.6:0/203537301 wait complete. 2024-03-22T12:57:18.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.012+0000 7f1719e00640 1 Processor -- start 2024-03-22T12:57:18.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.012+0000 7f1719e00640 1 -- start start 2024-03-22T12:57:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.013+0000 7f1719e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17140cc640 0x7f17141f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.013+0000 7f1719e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.013+0000 7f1719e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f171415b7a0 0x7f171414e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.013+0000 7f1719e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171406cfc0 con 0x7f17140cc640 2024-03-22T12:57:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.013+0000 7f1719e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171406cce0 con 0x7f171415b7a0 2024-03-22T12:57:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.013+0000 7f1719e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171415b000 con 0x7f17140cca20 2024-03-22T12:57:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.014+0000 7f1713e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.014+0000 7f1713e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ce560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57226/0 (socket says 172.21.15.6:57226) 2024-03-22T12:57:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.015+0000 7f1713e00640 1 -- 172.21.15.6:0/2935997835 learned_addr learned my addr 172.21.15.6:0/2935997835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:18.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.018+0000 7f1713e00640 1 -- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f171415b7a0 msgr2=0x7f171414e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.018+0000 7f1713e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f171415b7a0 0x7f171414e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.018+0000 7f1713e00640 1 -- 172.21.15.6:0/2935997835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17140cc640 msgr2=0x7f17141f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.018+0000 7f1713e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17140cc640 0x7f17141f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.018+0000 7f1713e00640 1 -- 172.21.15.6:0/2935997835 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17141f1a20 con 0x7f17140cca20 2024-03-22T12:57:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.021+0000 7f1713e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ce560 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1700000c00 tx=0x7f1700004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.026+0000 7f1710c00640 1 -- 172.21.15.6:0/2935997835 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700048070 con 0x7f17140cca20 2024-03-22T12:57:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.026+0000 7f1710c00640 1 -- 172.21.15.6:0/2935997835 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700030640 con 0x7f17140cca20 2024-03-22T12:57:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.026+0000 7f1710c00640 1 -- 172.21.15.6:0/2935997835 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700034040 con 0x7f17140cca20 2024-03-22T12:57:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.026+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17140ceed0 con 0x7f17140cca20 2024-03-22T12:57:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.026+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17140b21d0 con 0x7f17140cca20 2024-03-22T12:57:18.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.027+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1714075650 con 0x7f17140cca20 2024-03-22T12:57:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.068+0000 7f1710c00640 1 -- 172.21.15.6:0/2935997835 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f170004b070 con 0x7f17140cca20 2024-03-22T12:57:18.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.069+0000 7f1710c00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f16f406bc20 0x7f16f406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.070+0000 7f1718a00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f16f406bc20 0x7f16f406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:18.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.070+0000 7f1718a00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f16f406bc20 0x7f16f406e0e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1708009710 tx=0x7f1708006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:18.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.072+0000 7f1710c00640 1 -- 172.21.15.6:0/2935997835 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f17000307f0 con 0x7f17140cca20 2024-03-22T12:57:18.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.394+0000 7f1710c00640 1 -- 172.21.15.6:0/2935997835 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1700052020 con 0x7f17140cca20 2024-03-22T12:57:18.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.600+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17140af100 con 0x7f17140cca20 2024-03-22T12:57:18.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.640+0000 7f1710c00640 1 -- 172.21.15.6:0/2935997835 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7f170007bed0 con 0x7f17140cca20 2024-03-22T12:57:18.641 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:18.642 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:18.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f16f406bc20 msgr2=0x7f16f406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f16f406bc20 0x7f16f406e0e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1708009710 tx=0x7f1708006c90 comp rx=0 tx=0).stop 2024-03-22T12:57:18.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 msgr2=0x7f17140ce560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ce560 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1700000c00 tx=0x7f1700004730 comp rx=0 tx=0).stop 2024-03-22T12:57:18.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 shutdown_connections 2024-03-22T12:57:18.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f16f406bc20 0x7f16f406e0e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f171415b7a0 0x7f171414e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17140cca20 0x7f17140ce560 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 --2- 172.21.15.6:0/2935997835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17140cc640 0x7f17141f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 >> 172.21.15.6:0/2935997835 conn(0x7f17140cf9c0 msgr2=0x7f1714159900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:18.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 shutdown_connections 2024-03-22T12:57:18.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.643+0000 7f1719e00640 1 -- 172.21.15.6:0/2935997835 wait complete. 2024-03-22T12:57:18.660 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-22T12:57:18.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:18.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.864+0000 7fa5e6800640 1 Processor -- start 2024-03-22T12:57:18.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.864+0000 7fa5e6800640 1 -- start start 2024-03-22T12:57:18.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.865+0000 7fa5e6800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e00ccad0 0x7fa5e006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.865+0000 7fa5e6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5e015b0e0 0x7fa5e015b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.865+0000 7fa5e6800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e006c7a0 con 0x7fa5e00ca520 2024-03-22T12:57:18.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.865+0000 7fa5e6800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e00cd000 con 0x7fa5e015b0e0 2024-03-22T12:57:18.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.865+0000 7fa5e6800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e006ca80 con 0x7fa5e00ccad0 2024-03-22T12:57:18.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.866+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e00ccad0 0x7fa5e006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:18.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.867+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e00ccad0 0x7fa5e006d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57240/0 (socket says 172.21.15.6:57240) 2024-03-22T12:57:18.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.867+0000 7fa5dea00640 1 -- 172.21.15.6:0/1124491442 learned_addr learned my addr 172.21.15.6:0/1124491442 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:18.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.867+0000 7fa5dfe00640 1 --2- 172.21.15.6:0/1124491442 >> v2:172.21.15.6:3300/0 conn(0x7fa5e015b0e0 0x7fa5e015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:18.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.869+0000 7fa5dea00640 1 -- 172.21.15.6:0/1124491442 >> v2:172.21.15.6:3300/0 conn(0x7fa5e015b0e0 msgr2=0x7fa5e015b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:18.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.869+0000 7fa5dea00640 1 --2- 172.21.15.6:0/1124491442 >> v2:172.21.15.6:3300/0 conn(0x7fa5e015b0e0 0x7fa5e015b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.869+0000 7fa5dea00640 1 -- 172.21.15.6:0/1124491442 >> v1:172.21.15.6:6789/0 conn(0x7fa5e00ca520 legacy=0x7fa5e00cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.869+0000 7fa5dea00640 1 -- 172.21.15.6:0/1124491442 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e006cd60 con 0x7fa5e00ccad0 2024-03-22T12:57:18.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.871+0000 7fa5dea00640 1 --2- 172.21.15.6:0/1124491442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e00ccad0 0x7fa5e006d840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009870 tx=0x7fa5c802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d86a5d04ff1c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.875+0000 7fa5de000640 1 -- 172.21.15.6:0/1124491442 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8034070 con 0x7fa5e00ccad0 2024-03-22T12:57:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.875+0000 7fa5de000640 1 -- 172.21.15.6:0/1124491442 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c8030840 con 0x7fa5e00ccad0 2024-03-22T12:57:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.875+0000 7fa5de000640 1 -- 172.21.15.6:0/1124491442 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8030c10 con 0x7fa5e00ccad0 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 -- 172.21.15.6:0/1124491442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e00ccad0 msgr2=0x7fa5e006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1124491442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e00ccad0 0x7fa5e006d840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8009870 tx=0x7fa5c802b980 comp rx=0 tx=0).stop 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 -- 172.21.15.6:0/1124491442 shutdown_connections 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1124491442 >> v2:172.21.15.6:3300/0 conn(0x7fa5e015b0e0 0x7fa5e015b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1124491442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e00ccad0 0x7fa5e006d840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 -- 172.21.15.6:0/1124491442 >> 172.21.15.6:0/1124491442 conn(0x7fa5e00b16b0 msgr2=0x7fa5e00b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 -- 172.21.15.6:0/1124491442 shutdown_connections 2024-03-22T12:57:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.876+0000 7fa5e6800640 1 -- 172.21.15.6:0/1124491442 wait complete. 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.878+0000 7fa5e6800640 1 Processor -- start 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.878+0000 7fa5e6800640 1 -- start start 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.879+0000 7fa5e6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5e00ca520 0x7fa5e00b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.879+0000 7fa5e6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5e00ccad0 0x7fa5e00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.879+0000 7fa5e6800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e015b0e0 0x7fa5e00ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.879+0000 7fa5e6800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e0154c20 con 0x7fa5e00ca520 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.879+0000 7fa5e6800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e006d320 con 0x7fa5e00ccad0 2024-03-22T12:57:18.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.879+0000 7fa5e6800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00ca900 con 0x7fa5e015b0e0 2024-03-22T12:57:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.880+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e015b0e0 0x7fa5e00ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.880+0000 7fa5df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5e00ca520 0x7fa5e00b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.880+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5e00ccad0 0x7fa5e00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.880+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e015b0e0 0x7fa5e00ce500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57246/0 (socket says 172.21.15.6:57246) 2024-03-22T12:57:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.880+0000 7fa5dfe00640 1 -- 172.21.15.6:0/1202463950 learned_addr learned my addr 172.21.15.6:0/1202463950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:18.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.885+0000 7fa5dfe00640 1 -- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5e00ccad0 msgr2=0x7fa5e00b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.885+0000 7fa5dfe00640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5e00ccad0 0x7fa5e00b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.885+0000 7fa5dfe00640 1 -- 172.21.15.6:0/1202463950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5e00ca520 msgr2=0x7fa5e00b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:18.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.885+0000 7fa5dfe00640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5e00ca520 0x7fa5e00b4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:18.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.885+0000 7fa5dfe00640 1 -- 172.21.15.6:0/1202463950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00b5460 con 0x7fa5e015b0e0 2024-03-22T12:57:18.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.888+0000 7fa5dfe00640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e015b0e0 0x7fa5e00ce500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009d30 tx=0x7fa5d800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:18.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.893+0000 7fa5d7400640 1 -- 172.21.15.6:0/1202463950 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8018020 con 0x7fa5e015b0e0 2024-03-22T12:57:18.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.893+0000 7fa5d7400640 1 -- 172.21.15.6:0/1202463950 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8014480 con 0x7fa5e015b0e0 2024-03-22T12:57:18.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.893+0000 7fa5d7400640 1 -- 172.21.15.6:0/1202463950 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8014870 con 0x7fa5e015b0e0 2024-03-22T12:57:18.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.893+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e00b5740 con 0x7fa5e015b0e0 2024-03-22T12:57:18.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.893+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e01f0950 con 0x7fa5e015b0e0 2024-03-22T12:57:18.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:18.895+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e0068150 con 0x7fa5e015b0e0 2024-03-22T12:57:19.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.164+0000 7fa5d7400640 1 -- 172.21.15.6:0/1202463950 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa5d8005d10 con 0x7fa5e015b0e0 2024-03-22T12:57:19.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.165+0000 7fa5d7400640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa5bc06bc50 0x7fa5bc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.166+0000 7fa5df400640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa5bc06bc50 0x7fa5bc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.167+0000 7fa5df400640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa5bc06bc50 0x7fa5bc06e110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0001780 tx=0x7fa5d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.168+0000 7fa5d7400640 1 -- 172.21.15.6:0/1202463950 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fa5d8058080 con 0x7fa5e015b0e0 2024-03-22T12:57:19.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.188+0000 7fa5d7400640 1 -- 172.21.15.6:0/1202463950 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa5e0068150 con 0x7fa5e015b0e0 2024-03-22T12:57:19.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.405+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5e00afa60 con 0x7fa5e015b0e0 2024-03-22T12:57:19.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.486+0000 7fa5d7400640 1 -- 172.21.15.6:0/1202463950 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7fa5e00afa60 con 0x7fa5e015b0e0 2024-03-22T12:57:19.488 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:19.488 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:19.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa5bc06bc50 msgr2=0x7fa5bc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa5bc06bc50 0x7fa5bc06e110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0001780 tx=0x7fa5d0009040 comp rx=0 tx=0).stop 2024-03-22T12:57:19.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e015b0e0 msgr2=0x7fa5e00ce500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e015b0e0 0x7fa5e00ce500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009d30 tx=0x7fa5d800c490 comp rx=0 tx=0).stop 2024-03-22T12:57:19.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 shutdown_connections 2024-03-22T12:57:19.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5e015b0e0 0x7fa5e00ce500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa5bc06bc50 0x7fa5bc06e110 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5e00ccad0 0x7fa5e00b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 --2- 172.21.15.6:0/1202463950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5e00ca520 0x7fa5e00b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 >> 172.21.15.6:0/1202463950 conn(0x7fa5e00b16b0 msgr2=0x7fa5e00c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:19.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 shutdown_connections 2024-03-22T12:57:19.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.490+0000 7fa5e6800640 1 -- 172.21.15.6:0/1202463950 wait complete. 2024-03-22T12:57:19.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-22T12:57:19.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-22T12:57:19.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-22T12:57:19.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-22T12:57:19.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-22T12:57:19.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-22T12:57:19.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-22T12:57:19.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-22T12:57:19.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.884+0000 7fe272e00640 1 Processor -- start 2024-03-22T12:57:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.887+0000 7ffb3e800640 1 Processor -- start 2024-03-22T12:57:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.887+0000 7ffb3e800640 1 -- start start 2024-03-22T12:57:19.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.884+0000 7fe272e00640 1 -- start start 2024-03-22T12:57:19.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.890+0000 7fe272e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe2640dac60 0x7fe26404b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.890+0000 7fe272e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe264034a40 0x7fe264034eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.891+0000 7fe271a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe26404b500 0x7fe264005ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40468/0 (socket says 172.21.15.6:40468) 2024-03-22T12:57:19.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.891+0000 7fe271a00640 1 -- 172.21.15.6:0/1910026031 learned_addr learned my addr 172.21.15.6:0/1910026031 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:19.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.891+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2640353f0 con 0x7fe26404b500 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.891+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe264007760 con 0x7fe2640dac60 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.892+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe264006900 con 0x7fe264034a40 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.892+0000 7fe272400640 1 --2- 172.21.15.6:0/1910026031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe264034a40 0x7fe264034eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.889+0000 7ffb3e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb380ca520 0x7ffb3806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.889+0000 7ffb3e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb3815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.889+0000 7ffb3e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3806c7a0 con 0x7ffb380ca900 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.889+0000 7ffb3e800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb380cd000 con 0x7ffb380ca520 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.889+0000 7ffb3e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3806ca80 con 0x7ffb3815b0e0 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.891+0000 7ffb2ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb3815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.892+0000 7ffb2ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb3815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57260/0 (socket says 172.21.15.6:57260) 2024-03-22T12:57:19.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.892+0000 7ffb2ea00640 1 -- 172.21.15.6:0/286562532 learned_addr learned my addr 172.21.15.6:0/286562532 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.886+0000 7f7950a00640 1 Processor -- start 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.886+0000 7f7950a00640 1 -- start start 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.888+0000 7f7950a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c1646f0 0x7f794c0c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.888+0000 7f7950a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f794c071620 0x7f794c167670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.888+0000 7f7950a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f794c167bb0 0x7f794c0c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.892+0000 7f7950a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c164ad0 con 0x7f794c167bb0 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.892+0000 7f7950a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c16a8f0 con 0x7f794c071620 2024-03-22T12:57:19.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.892+0000 7f7950a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c16abd0 con 0x7f794c1646f0 2024-03-22T12:57:19.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.895+0000 7fe271000640 1 --2- 172.21.15.6:0/1910026031 >> v2:172.21.15.92:3300/0 conn(0x7fe2640dac60 0x7fe26404b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.895+0000 7f7949e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f794c071620 0x7f794c167670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.895+0000 7ffb37400640 1 --2- 172.21.15.6:0/286562532 >> v2:172.21.15.92:3300/0 conn(0x7ffb380ca520 0x7ffb3806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.896+0000 7f7949e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f794c071620 0x7f794c167670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60810/0 (socket says 172.21.15.6:60810) 2024-03-22T12:57:19.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.896+0000 7f7949e00640 1 -- 172.21.15.6:0/1131246431 learned_addr learned my addr 172.21.15.6:0/1131246431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.898+0000 7f9c1a600640 1 Processor -- start 2024-03-22T12:57:19.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.898+0000 7fe26be00640 1 -- 172.21.15.6:0/1910026031 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279581007 0 0) 0x7fe2640353f0 con 0x7fe26404b500 2024-03-22T12:57:19.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.896+0000 7ffb2ea00640 1 -- 172.21.15.6:0/286562532 >> v2:172.21.15.92:3300/0 conn(0x7ffb380ca520 msgr2=0x7ffb3806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.896+0000 7ffb2ea00640 1 --2- 172.21.15.6:0/286562532 >> v2:172.21.15.92:3300/0 conn(0x7ffb380ca520 0x7ffb3806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.896+0000 7ffb2ea00640 1 -- 172.21.15.6:0/286562532 >> v1:172.21.15.6:6789/0 conn(0x7ffb380ca900 legacy=0x7ffb380cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.896+0000 7ffb2ea00640 1 -- 172.21.15.6:0/286562532 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3806cd60 con 0x7ffb3815b0e0 2024-03-22T12:57:19.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.898+0000 7f794a800640 1 --2- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c1646f0 0x7f794c0c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.899+0000 7fe26be00640 1 -- 172.21.15.6:0/1910026031 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe254003520 con 0x7fe26404b500 2024-03-22T12:57:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.898+0000 7f9c1a600640 1 -- start start 2024-03-22T12:57:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f9c1a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c140c7a80 0x7f9c140c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f9c1a600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9c140c7230 0x7f9c141566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f9c1a600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140c4520 con 0x7f9c141539f0 2024-03-22T12:57:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f9c1a600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140c7ea0 con 0x7f9c140c7a80 2024-03-22T12:57:19.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f9c1a600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c14153dd0 con 0x7f9c140c7230 2024-03-22T12:57:19.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f7949e00640 1 -- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c1646f0 msgr2=0x7f794c0c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f7949e00640 1 --2- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c1646f0 0x7f794c0c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f7949e00640 1 -- 172.21.15.6:0/1131246431 >> v2:172.21.15.6:3300/0 conn(0x7f794c167bb0 msgr2=0x7f794c0c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f7949e00640 1 --2- 172.21.15.6:0/1131246431 >> v2:172.21.15.6:3300/0 conn(0x7f794c167bb0 0x7f794c0c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.900+0000 7f7949e00640 1 -- 172.21.15.6:0/1131246431 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c16aeb0 con 0x7f794c071620 2024-03-22T12:57:19.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.902+0000 7fe272400640 1 -- 172.21.15.6:0/1910026031 >> v2:172.21.15.92:3300/0 conn(0x7fe2640dac60 msgr2=0x7fe26404b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.902+0000 7fe272400640 1 --2- 172.21.15.6:0/1910026031 >> v2:172.21.15.92:3300/0 conn(0x7fe2640dac60 0x7fe26404b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.902+0000 7fe272400640 1 -- 172.21.15.6:0/1910026031 >> v1:172.21.15.92:6789/0 conn(0x7fe26404b500 legacy=0x7fe264005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.902+0000 7fe272400640 1 -- 172.21.15.6:0/1910026031 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264006be0 con 0x7fe264034a40 2024-03-22T12:57:19.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.907+0000 7fe272400640 1 --2- 172.21.15.6:0/1910026031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe264034a40 0x7fe264034eb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe2580097a0 tx=0x7fe25802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c71f49319182d98c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.908+0000 7f9c13400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9c141539f0 0x7f9c140c5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55134/0 (socket says 172.21.15.6:55134) 2024-03-22T12:57:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.908+0000 7f9c13400640 1 -- 172.21.15.6:0/2854830023 learned_addr learned my addr 172.21.15.6:0/2854830023 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.908+0000 7f9c12a00640 1 --2- 172.21.15.6:0/2854830023 >> v2:172.21.15.6:3300/0 conn(0x7f9c140c7a80 0x7f9c140c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.907+0000 7ffb2ea00640 1 --2- 172.21.15.6:0/286562532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb3815b980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009e50 tx=0x7ffb2802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c67ddff049fde777 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.911+0000 7ffb36a00640 1 -- 172.21.15.6:0/286562532 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2802cb40 con 0x7ffb3815b0e0 2024-03-22T12:57:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.912+0000 7fc55d000640 1 Processor -- start 2024-03-22T12:57:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.912+0000 7fc55d000640 1 -- start start 2024-03-22T12:57:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.912+0000 7fe26be00640 1 -- 172.21.15.6:0/1910026031 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258034070 con 0x7fe264034a40 2024-03-22T12:57:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.912+0000 7f9c13e00640 1 --2- 172.21.15.6:0/2854830023 >> v2:172.21.15.92:3300/0 conn(0x7f9c140c7230 0x7f9c141566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.913+0000 7f7949e00640 1 --2- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c071620 0x7f794c167670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f794c0c94d0 tx=0x7f793c02bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b150891777a911 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.914+0000 7ffb36a00640 1 -- 172.21.15.6:0/286562532 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2802cce0 con 0x7ffb3815b0e0 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.914+0000 7fc55d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc5580ce9f0 0x7fc5580c2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.914+0000 7fc55d000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580cadd0 con 0x7fc5580c9b50 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.914+0000 7fc55d000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55815f5f0 con 0x7fc5580cc5a0 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.914+0000 7fc55d000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5580cd090 con 0x7fc5580ce9f0 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.915+0000 7f9c12a00640 1 -- 172.21.15.6:0/2854830023 >> v2:172.21.15.92:3300/0 conn(0x7f9c140c7230 msgr2=0x7f9c141566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.915+0000 7f9c12a00640 1 --2- 172.21.15.6:0/2854830023 >> v2:172.21.15.92:3300/0 conn(0x7f9c140c7230 0x7f9c141566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.915+0000 7f9c12a00640 1 -- 172.21.15.6:0/2854830023 >> v1:172.21.15.6:6789/0 conn(0x7f9c141539f0 legacy=0x7f9c140c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.915+0000 7f9c12a00640 1 -- 172.21.15.6:0/2854830023 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140c47c0 con 0x7f9c140c7a80 2024-03-22T12:57:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.915+0000 7ffb3e800640 1 -- 172.21.15.6:0/286562532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 msgr2=0x7ffb3815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.915+0000 7ffb36a00640 1 -- 172.21.15.6:0/286562532 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28034610 con 0x7ffb3815b0e0 2024-03-22T12:57:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.915+0000 7ffb3e800640 1 --2- 172.21.15.6:0/286562532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb3815b980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009e50 tx=0x7ffb2802bfa0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.916+0000 7ffb3e800640 1 -- 172.21.15.6:0/286562532 shutdown_connections 2024-03-22T12:57:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.916+0000 7ffb3e800640 1 --2- 172.21.15.6:0/286562532 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb3815b980 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.916+0000 7ffb3e800640 1 --2- 172.21.15.6:0/286562532 >> v2:172.21.15.92:3300/0 conn(0x7ffb380ca520 0x7ffb3806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.916+0000 7ffb3e800640 1 -- 172.21.15.6:0/286562532 >> 172.21.15.6:0/286562532 conn(0x7ffb380b16b0 msgr2=0x7ffb380b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:19.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.917+0000 7ffb3e800640 1 -- 172.21.15.6:0/286562532 shutdown_connections 2024-03-22T12:57:19.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.917+0000 7ffb3e800640 1 -- 172.21.15.6:0/286562532 wait complete. 2024-03-22T12:57:19.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.918+0000 7fe26be00640 1 -- 172.21.15.6:0/1910026031 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258030800 con 0x7fe264034a40 2024-03-22T12:57:19.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.918+0000 7fe26be00640 1 -- 172.21.15.6:0/1910026031 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258030bb0 con 0x7fe264034a40 2024-03-22T12:57:19.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.918+0000 7fc556800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc5580ce9f0 0x7fc5580c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.919+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe264034a40 msgr2=0x7fe264034eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.919+0000 7fe272e00640 1 --2- 172.21.15.6:0/1910026031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe264034a40 0x7fe264034eb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe2580097a0 tx=0x7fe25802b940 comp rx=0 tx=0).stop 2024-03-22T12:57:19.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.919+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 shutdown_connections 2024-03-22T12:57:19.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.919+0000 7fe272e00640 1 --2- 172.21.15.6:0/1910026031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe264034a40 0x7fe264034eb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.919+0000 7fe272e00640 1 --2- 172.21.15.6:0/1910026031 >> v2:172.21.15.92:3300/0 conn(0x7fe2640dac60 0x7fe26404b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.919+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 >> 172.21.15.6:0/1910026031 conn(0x7fe264050180 msgr2=0x7fe264050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:19.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.920+0000 7fc556800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc5580ce9f0 0x7fc5580c2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33782/0 (socket says 172.21.15.6:33782) 2024-03-22T12:57:19.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.920+0000 7fc556800640 1 -- 172.21.15.6:0/2852892662 learned_addr learned my addr 172.21.15.6:0/2852892662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.920+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 shutdown_connections 2024-03-22T12:57:19.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.921+0000 7ffb3e800640 1 Processor -- start 2024-03-22T12:57:19.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.921+0000 7ffb3e800640 1 -- start start 2024-03-22T12:57:19.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.920+0000 7fe272e00640 1 -- 172.21.15.6:0/1910026031 wait complete. 2024-03-22T12:57:19.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.923+0000 7fe272e00640 1 Processor -- start 2024-03-22T12:57:19.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.924+0000 7f7949400640 1 -- 172.21.15.6:0/1131246431 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c02cb20 con 0x7f794c071620 2024-03-22T12:57:19.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.924+0000 7f7949400640 1 -- 172.21.15.6:0/1131246431 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c02ccc0 con 0x7f794c071620 2024-03-22T12:57:19.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.924+0000 7f7949400640 1 -- 172.21.15.6:0/1131246431 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c034610 con 0x7f794c071620 2024-03-22T12:57:19.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.924+0000 7fe272e00640 1 -- start start 2024-03-22T12:57:19.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.924+0000 7ffb3e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb380ca520 0x7ffb380c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.924+0000 7ffb3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb380ca900 0x7ffb38153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.925+0000 7ffb3e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb381543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.925+0000 7ffb3e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380cad60 con 0x7ffb380ca900 2024-03-22T12:57:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.925+0000 7ffb3e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3806d320 con 0x7ffb380ca520 2024-03-22T12:57:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.925+0000 7ffb3e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38154c20 con 0x7ffb3815b0e0 2024-03-22T12:57:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.925+0000 7fe272e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe264034a40 0x7fe2641705c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7fe272e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe26404b500 0x7fe2640cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7fe272e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2640dac60 0x7fe2640cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7fe272e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe264007480 con 0x7fe2640dac60 2024-03-22T12:57:19.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7fe272e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640071a0 con 0x7fe264034a40 2024-03-22T12:57:19.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7fe272e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640065f0 con 0x7fe26404b500 2024-03-22T12:57:19.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7ffb37e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb381543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7f7950a00640 1 -- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c071620 msgr2=0x7f794c167670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7f7950a00640 1 --2- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c071620 0x7f794c167670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f794c0c94d0 tx=0x7f793c02bf80 comp rx=0 tx=0).stop 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7f7950a00640 1 -- 172.21.15.6:0/1131246431 shutdown_connections 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7f7950a00640 1 --2- 172.21.15.6:0/1131246431 >> v2:172.21.15.6:3300/0 conn(0x7f794c167bb0 0x7f794c0c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7f7950a00640 1 --2- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c071620 0x7f794c167670 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7f7950a00640 1 --2- 172.21.15.6:0/1131246431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c1646f0 0x7f794c0c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7f7950a00640 1 -- 172.21.15.6:0/1131246431 >> 172.21.15.6:0/1131246431 conn(0x7f794c06f0e0 msgr2=0x7f794c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7fc554a00640 1 -- 172.21.15.6:0/2852892662 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576176262 0 0) 0x7fc55815f5f0 con 0x7fc5580cc5a0 2024-03-22T12:57:19.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.926+0000 7fc554a00640 1 -- 172.21.15.6:0/2852892662 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc53c003520 con 0x7fc5580cc5a0 2024-03-22T12:57:19.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7ffb37e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb381543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57282/0 (socket says 172.21.15.6:57282) 2024-03-22T12:57:19.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7ffb37e00640 1 -- 172.21.15.6:0/1165778487 learned_addr learned my addr 172.21.15.6:0/1165778487 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7ffb37400640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb380ca520 0x7ffb380c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7f7950a00640 1 -- 172.21.15.6:0/1131246431 shutdown_connections 2024-03-22T12:57:19.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7fe271000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe26404b500 0x7fe2640cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7f7950a00640 1 -- 172.21.15.6:0/1131246431 wait complete. 2024-03-22T12:57:19.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.927+0000 7fe271a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe264034a40 0x7fe2641705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.928+0000 7fe271000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe26404b500 0x7fe2640cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57288/0 (socket says 172.21.15.6:57288) 2024-03-22T12:57:19.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.928+0000 7fe271000640 1 -- 172.21.15.6:0/2096332775 learned_addr learned my addr 172.21.15.6:0/2096332775 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.929+0000 7fc556800640 1 -- 172.21.15.6:0/2852892662 >> v1:172.21.15.92:6789/0 conn(0x7fc5580cc5a0 legacy=0x7fc5580cc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.929+0000 7fc556800640 1 -- 172.21.15.6:0/2852892662 >> v1:172.21.15.6:6789/0 conn(0x7fc5580c9b50 legacy=0x7fc5580ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.929+0000 7fc556800640 1 -- 172.21.15.6:0/2852892662 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55815f8d0 con 0x7fc5580ce9f0 2024-03-22T12:57:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.931+0000 7f7950a00640 1 Processor -- start 2024-03-22T12:57:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.931+0000 7f7950a00640 1 -- start start 2024-03-22T12:57:19.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.931+0000 7f9c12a00640 1 --2- 172.21.15.6:0/2854830023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c140c7a80 0x7f9c140c6cf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c1415a930 tx=0x7f9c0402cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a624a9d4cbf70f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.932+0000 7ffb2ea00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb380ca900 0x7ffb38153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7f7950a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794c071620 0x7f794c157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7f7950a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c1646f0 0x7f794c1fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7f7950a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c167bb0 0x7f794c1ff170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7ffb37e00640 1 -- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb380ca520 msgr2=0x7ffb380c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7ffb37e00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb380ca520 0x7ffb380c7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7ffb37e00640 1 -- 172.21.15.6:0/1165778487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb380ca900 msgr2=0x7ffb38153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7ffb37e00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb380ca900 0x7ffb38153e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.933+0000 7ffb37e00640 1 -- 172.21.15.6:0/1165778487 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380c8630 con 0x7ffb3815b0e0 2024-03-22T12:57:19.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.934+0000 7f3d3ea00640 1 Processor -- start 2024-03-22T12:57:19.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.934+0000 7f7950a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c0c7e20 con 0x7f794c071620 2024-03-22T12:57:19.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.934+0000 7f7950a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c167150 con 0x7f794c1646f0 2024-03-22T12:57:19.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.934+0000 7f7950a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c0c7fa0 con 0x7f794c167bb0 2024-03-22T12:57:19.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.934+0000 7f3d3ea00640 1 -- start start 2024-03-22T12:57:19.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.936+0000 7f3d3ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 0x7f3d3815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.936+0000 7f3d3ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d380ca390 0x7f3d380cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.936+0000 7f3d3ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d381549b0 con 0x7f3d380c9b10 2024-03-22T12:57:19.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.936+0000 7f3d3ea00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3815b1d0 con 0x7f3d380ca390 2024-03-22T12:57:19.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.936+0000 7f3d3ea00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380ca840 con 0x7f3d380c9690 2024-03-22T12:57:19.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.938+0000 7fe271000640 1 -- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe264034a40 msgr2=0x7fe2641705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.938+0000 7fe271000640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe264034a40 0x7fe2641705c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.938+0000 7fe271000640 1 -- 172.21.15.6:0/2096332775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2640dac60 msgr2=0x7fe2640cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.938+0000 7fe271000640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2640dac60 0x7fe2640cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.938+0000 7fe271000640 1 -- 172.21.15.6:0/2096332775 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264176610 con 0x7fe26404b500 2024-03-22T12:57:19.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.939+0000 7fc556800640 1 --2- 172.21.15.6:0/2852892662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580ce9f0 0x7fc5580c2840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc550009e80 tx=0x7fc55002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b0cd455427f898 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.942+0000 7ffb37e00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb381543a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000c3c0 tx=0x7ffb3000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.942+0000 7fe271000640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe26404b500 0x7fe2640cecc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe26000e830 tx=0x7fe26000ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.943+0000 7f7949e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c1646f0 0x7f794c1fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.944+0000 7f3d2fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d380ca390 0x7f3d380cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.944+0000 7f7949e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c1646f0 0x7f794c1fec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60838/0 (socket says 172.21.15.6:60838) 2024-03-22T12:57:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.944+0000 7f7949e00640 1 -- 172.21.15.6:0/477022923 learned_addr learned my addr 172.21.15.6:0/477022923 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.945+0000 7f3d2fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d380ca390 0x7f3d380cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60852/0 (socket says 172.21.15.6:60852) 2024-03-22T12:57:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.945+0000 7f3d2fe00640 1 -- 172.21.15.6:0/211316103 learned_addr learned my addr 172.21.15.6:0/211316103 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.947+0000 7f9c12000640 1 -- 172.21.15.6:0/2854830023 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0403a070 con 0x7f9c140c7a80 2024-03-22T12:57:19.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.947+0000 7f9c12000640 1 -- 172.21.15.6:0/2854830023 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0402d9f0 con 0x7f9c140c7a80 2024-03-22T12:57:19.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.947+0000 7f9c12000640 1 -- 172.21.15.6:0/2854830023 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0402de00 con 0x7f9c140c7a80 2024-03-22T12:57:19.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.948+0000 7f9c1a600640 1 -- 172.21.15.6:0/2854830023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c140c7a80 msgr2=0x7f9c140c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f7949e00640 1 -- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c167bb0 msgr2=0x7f794c1ff170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f7949e00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f794c167bb0 0x7f794c1ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f7949e00640 1 -- 172.21.15.6:0/477022923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794c071620 msgr2=0x7f794c157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.948+0000 7f9c1a600640 1 --2- 172.21.15.6:0/2854830023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c140c7a80 0x7f9c140c6cf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c1415a930 tx=0x7f9c0402cc40 comp rx=0 tx=0).stop 2024-03-22T12:57:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f7949e00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794c071620 0x7f794c157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f9c1a600640 1 -- 172.21.15.6:0/2854830023 shutdown_connections 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f9c1a600640 1 --2- 172.21.15.6:0/2854830023 >> v2:172.21.15.92:3300/0 conn(0x7f9c140c7230 0x7f9c141566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f9c1a600640 1 --2- 172.21.15.6:0/2854830023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c140c7a80 0x7f9c140c6cf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f9c1a600640 1 -- 172.21.15.6:0/2854830023 >> 172.21.15.6:0/2854830023 conn(0x7f9c140cb930 msgr2=0x7f9c140cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f9c1a600640 1 -- 172.21.15.6:0/2854830023 shutdown_connections 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f7949e00640 1 -- 172.21.15.6:0/477022923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c2055d0 con 0x7f794c1646f0 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.949+0000 7f9c1a600640 1 -- 172.21.15.6:0/2854830023 wait complete. 2024-03-22T12:57:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.951+0000 7f9c1a600640 1 Processor -- start 2024-03-22T12:57:19.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.951+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb300052d0 con 0x7ffb3815b0e0 2024-03-22T12:57:19.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.951+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30005470 con 0x7ffb3815b0e0 2024-03-22T12:57:19.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.951+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30007240 con 0x7ffb3815b0e0 2024-03-22T12:57:19.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.951+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb381f84f0 con 0x7ffb3815b0e0 2024-03-22T12:57:19.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.952+0000 7fc554a00640 1 -- 172.21.15.6:0/2852892662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55002cbb0 con 0x7fc5580ce9f0 2024-03-22T12:57:19.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.952+0000 7fc554a00640 1 -- 172.21.15.6:0/2852892662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55002cd50 con 0x7fc5580ce9f0 2024-03-22T12:57:19.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.952+0000 7fc554a00640 1 -- 172.21.15.6:0/2852892662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550034610 con 0x7fc5580ce9f0 2024-03-22T12:57:19.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.951+0000 7f9c1a600640 1 -- start start 2024-03-22T12:57:19.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.951+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb380c88d0 con 0x7ffb3815b0e0 2024-03-22T12:57:19.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.953+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffb380756c0 con 0x7ffb3815b0e0 2024-03-22T12:57:19.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.953+0000 7fc55d000640 1 -- 172.21.15.6:0/2852892662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580ce9f0 msgr2=0x7fc5580c2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.953+0000 7fc55d000640 1 --2- 172.21.15.6:0/2852892662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580ce9f0 0x7fc5580c2840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc550009e80 tx=0x7fc55002c010 comp rx=0 tx=0).stop 2024-03-22T12:57:19.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7f3d37e00640 1 --2- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 0x7f3d3815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26000caa0 con 0x7fe26404b500 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe260019070 con 0x7fe26404b500 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260014410 con 0x7fe26404b500 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2641768b0 con 0x7fe26404b500 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe264176e70 con 0x7fe26404b500 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7f3d2fe00640 1 -- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 msgr2=0x7f3d3815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7f3d2fe00640 1 --2- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 0x7f3d3815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7f3d2fe00640 1 -- 172.21.15.6:0/211316103 >> v1:172.21.15.6:6789/0 conn(0x7f3d380c9b10 legacy=0x7f3d380ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7f3d2fe00640 1 -- 172.21.15.6:0/211316103 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380cab20 con 0x7f3d380ca390 2024-03-22T12:57:19.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7f9c1a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c140c7230 0x7f9c14154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.954+0000 7f9c1a600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c140c7a80 0x7f9c141eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe264008130 con 0x7fe26404b500 2024-03-22T12:57:19.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7f9c1a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c141539f0 0x7f9c141ed030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7f9c1a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c80c0 con 0x7f9c141539f0 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7f9c1a600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c4d80 con 0x7f9c140c7230 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7f9c1a600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c3ac0 con 0x7f9c140c7a80 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7fc55d000640 1 -- 172.21.15.6:0/2852892662 shutdown_connections 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7fc55d000640 1 --2- 172.21.15.6:0/2852892662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580ce9f0 0x7fc5580c2840 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7fc55d000640 1 -- 172.21.15.6:0/2852892662 >> 172.21.15.6:0/2852892662 conn(0x7fc5580b16c0 msgr2=0x7fc5580b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7fc55d000640 1 -- 172.21.15.6:0/2852892662 shutdown_connections 2024-03-22T12:57:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7fc55d000640 1 -- 172.21.15.6:0/2852892662 wait complete. 2024-03-22T12:57:19.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.956+0000 7f7949e00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c1646f0 0x7f794c1fec30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f793c02c590 tx=0x7f793c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.958+0000 7fc55d000640 1 Processor -- start 2024-03-22T12:57:19.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.959+0000 7fc55d000640 1 -- start start 2024-03-22T12:57:19.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.960+0000 7fc55d000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5580c9b50 0x7fc5580b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.960+0000 7fc55d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580cc5a0 0x7fc558158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.960+0000 7fc55d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5580ce9f0 0x7fc5580c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.960+0000 7fc55d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580ca810 con 0x7fc5580cc5a0 2024-03-22T12:57:19.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.960+0000 7fc55d000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580caaf0 con 0x7fc5580ce9f0 2024-03-22T12:57:19.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.960+0000 7fc55d000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55815dea0 con 0x7fc5580c9b50 2024-03-22T12:57:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.961+0000 7f75b0a00640 1 Processor -- start 2024-03-22T12:57:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.961+0000 7f75b0a00640 1 -- start start 2024-03-22T12:57:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.961+0000 7fc555400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580cc5a0 0x7fc558158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.962+0000 7f9c13e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c141539f0 0x7f9c141ed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.962+0000 7fc555400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580cc5a0 0x7fc558158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33848/0 (socket says 172.21.15.6:33848) 2024-03-22T12:57:19.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.962+0000 7fc555400640 1 -- 172.21.15.6:0/1603520635 learned_addr learned my addr 172.21.15.6:0/1603520635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.962+0000 7f75b0a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac0ca520 0x7f75ac06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.962+0000 7f75b0a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75ac15b0e0 0x7f75ac15b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.962+0000 7f75b0a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac06c7a0 con 0x7f75ac0ca900 2024-03-22T12:57:19.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.963+0000 7f75b0a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac0cd000 con 0x7f75ac15b0e0 2024-03-22T12:57:19.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.963+0000 7f75b0a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac06ca80 con 0x7f75ac0ca520 2024-03-22T12:57:19.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.964+0000 7f3d2fe00640 1 --2- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380ca390 0x7f3d380cc190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009870 tx=0x7f3d2002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51be2c03f1801f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.964+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c043020 con 0x7f794c1646f0 2024-03-22T12:57:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.965+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c03e450 con 0x7f794c1646f0 2024-03-22T12:57:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.965+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c03e840 con 0x7f794c1646f0 2024-03-22T12:57:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.965+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c1fde70 con 0x7f794c1646f0 2024-03-22T12:57:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.965+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c1fe430 con 0x7f794c1646f0 2024-03-22T12:57:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.966+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f794c1fdc20 con 0x7f794c1646f0 2024-03-22T12:57:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.966+0000 7f9c13400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c140c7230 0x7f9c14154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.968+0000 7fc556800640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5580ce9f0 0x7fc5580c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.968+0000 7fc555400640 1 -- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5580c9b50 msgr2=0x7fc5580b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.968+0000 7fc555400640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5580c9b50 0x7fc5580b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.968+0000 7fc555400640 1 -- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5580ce9f0 msgr2=0x7fc5580c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.968+0000 7fc555400640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5580ce9f0 0x7fc5580c3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.968+0000 7fc555400640 1 -- 172.21.15.6:0/1603520635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580c3510 con 0x7fc5580cc5a0 2024-03-22T12:57:19.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.969+0000 7f9c13e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c141539f0 0x7f9c141ed030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33834/0 (socket says 172.21.15.6:33834) 2024-03-22T12:57:19.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.969+0000 7f9c13e00640 1 -- 172.21.15.6:0/1685594067 learned_addr learned my addr 172.21.15.6:0/1685594067 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:19.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.971+0000 7fc555400640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580cc5a0 0x7fc558158280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc54400c430 tx=0x7fc54400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.972+0000 7f9c12a00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c140c7a80 0x7f9c141eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:19.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.975+0000 7f9c13e00640 1 -- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c140c7a80 msgr2=0x7f9c141eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.975+0000 7f9c13e00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c140c7a80 0x7f9c141eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.975+0000 7f9c13e00640 1 -- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c140c7230 msgr2=0x7f9c14154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:19.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.975+0000 7f9c13e00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c140c7230 0x7f9c14154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:19.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.975+0000 7f9c13e00640 1 -- 172.21.15.6:0/1685594067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c141fdde0 con 0x7f9c141539f0 2024-03-22T12:57:19.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.977+0000 7ffb53200640 1 Processor -- start 2024-03-22T12:57:19.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.977+0000 7ffb53200640 1 -- start start 2024-03-22T12:57:19.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.979+0000 7ffb53200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 0x7ffb4c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.979+0000 7ffb53200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb4c0cca20 0x7ffb4c0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:19.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.979+0000 7ffb53200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c06dd00 con 0x7ffb4c15b7a0 2024-03-22T12:57:19.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.979+0000 7ffb53200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c06d330 con 0x7ffb4c0cca20 2024-03-22T12:57:19.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.979+0000 7ffb53200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c06c440 con 0x7ffb4c0cc640 2024-03-22T12:57:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.981+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544011790 con 0x7fc5580cc5a0 2024-03-22T12:57:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.981+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc544017070 con 0x7fc5580cc5a0 2024-03-22T12:57:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.981+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55815be50 con 0x7fc5580cc5a0 2024-03-22T12:57:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.982+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544011b70 con 0x7fc5580cc5a0 2024-03-22T12:57:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.982+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55815c410 con 0x7fc5580cc5a0 2024-03-22T12:57:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.982+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc5580756c0 con 0x7fc5580cc5a0 2024-03-22T12:57:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.983+0000 7f9c13e00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c141539f0 0x7f9c141ed030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000d800 tx=0x7f9c0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:19.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:19.993+0000 7ffb51400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb4c0cca20 0x7ffb4c0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.001+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffb300073e0 con 0x7ffb3815b0e0 2024-03-22T12:57:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.002+0000 7ffb2fe00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc20 0x7ffb1406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.002+0000 7ffb37400640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc20 0x7ffb1406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.003+0000 7ffb37400640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc20 0x7ffb1406e0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffb200079a0 tx=0x7ffb20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.007+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7ffb30056f60 con 0x7ffb3815b0e0 2024-03-22T12:57:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.007+0000 7ffb2fe00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7ffb140756b0 0x7ffb14077b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.007+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 --> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb30018e50 con 0x7ffb140756b0 2024-03-22T12:57:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.007+0000 7ffb2ea00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7ffb140756b0 0x7ffb14077b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.008+0000 7ffb2ea00640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7ffb140756b0 0x7ffb14077b50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.008+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb30055160 con 0x7ffb3815b0e0 2024-03-22T12:57:20.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.009+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== osd.5 v2:172.21.15.92:6810/1588636001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffb30018e50 con 0x7ffb140756b0 2024-03-22T12:57:20.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.019+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f793c046020 con 0x7f794c1646f0 2024-03-22T12:57:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.020+0000 7f7942a00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f792806bbd0 0x7f792806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.020+0000 7f794a800640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f792806bbd0 0x7f792806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.021+0000 7f794a800640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f792806bbd0 0x7f792806e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7934006fd0 tx=0x7f7934008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.024+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f793c050e30 con 0x7f794c1646f0 2024-03-22T12:57:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.024+0000 7f7942a00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f79280755e0 0x7f7928077a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.024+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 --> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f793c035e90 con 0x7f79280755e0 2024-03-22T12:57:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.024+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f793c078160 con 0x7f794c1646f0 2024-03-22T12:57:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.024+0000 7f794b200640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f79280755e0 0x7f7928077a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.025+0000 7f794b200640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f79280755e0 0x7f7928077a80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.025+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 --> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffb380cb080 con 0x7ffb140756b0 2024-03-22T12:57:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.025+0000 7ffb51400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb4c0cca20 0x7ffb4c0ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60886/0 (socket says 172.21.15.6:60886) 2024-03-22T12:57:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.025+0000 7ffb51400640 1 -- 172.21.15.6:0/1580522515 learned_addr learned my addr 172.21.15.6:0/1580522515 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.026+0000 7ffb2fe00640 1 -- 172.21.15.6:0/1165778487 <== osd.5 v2:172.21.15.92:6810/1588636001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffb380cb080 con 0x7ffb140756b0 2024-03-22T12:57:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.026+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== osd.7 v2:172.21.15.92:6818/1078860650 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f793c035e90 con 0x7f79280755e0 2024-03-22T12:57:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.026+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7ffb140756b0 msgr2=0x7ffb14077b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.026+0000 7ffb3e800640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7ffb140756b0 0x7ffb14077b50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc20 msgr2=0x7ffb1406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7ffb3e800640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc20 0x7ffb1406e0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffb200079a0 tx=0x7ffb20008040 comp rx=0 tx=0).stop 2024-03-22T12:57:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7f3d37400640 1 -- 172.21.15.6:0/211316103 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20035020 con 0x7f3d380ca390 2024-03-22T12:57:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7f3d37400640 1 -- 172.21.15.6:0/211316103 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d20031870 con 0x7f3d380ca390 2024-03-22T12:57:20.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7f3d37400640 1 -- 172.21.15.6:0/211316103 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20031c80 con 0x7f3d380ca390 2024-03-22T12:57:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 msgr2=0x7ffb381543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7ffb3e800640 1 --2- 172.21.15.6:0/1165778487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815b0e0 0x7ffb381543a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000c3c0 tx=0x7ffb3000c880 comp rx=0 tx=0).stop 2024-03-22T12:57:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7ffb37e00640 1 -- 172.21.15.6:0/1165778487 reap_dead start 2024-03-22T12:57:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 shutdown_connections 2024-03-22T12:57:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 -- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380ca390 msgr2=0x7f3d380cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 --2- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380ca390 0x7f3d380cc190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009870 tx=0x7f3d2002b910 comp rx=0 tx=0).stop 2024-03-22T12:57:20.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 >> 172.21.15.6:0/1165778487 conn(0x7ffb380b16b0 msgr2=0x7ffb380ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 -- 172.21.15.6:0/211316103 shutdown_connections 2024-03-22T12:57:20.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 --2- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380ca390 0x7f3d380cc190 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 --2- 172.21.15.6:0/211316103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 0x7f3d3815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 -- 172.21.15.6:0/211316103 >> 172.21.15.6:0/211316103 conn(0x7f3d380cfa10 msgr2=0x7f3d380cfe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7ffb51400640 1 -- 172.21.15.6:0/1580522515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 msgr2=0x7ffb4c06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7ffb51400640 1 --2- 172.21.15.6:0/1580522515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 0x7ffb4c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7ffb51400640 1 -- 172.21.15.6:0/1580522515 >> v1:172.21.15.6:6789/0 conn(0x7ffb4c15b7a0 legacy=0x7ffb4c15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 -- 172.21.15.6:0/211316103 shutdown_connections 2024-03-22T12:57:20.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7ffb51400640 1 -- 172.21.15.6:0/1580522515 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c06c720 con 0x7ffb4c0cca20 2024-03-22T12:57:20.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.028+0000 7f3d3ea00640 1 -- 172.21.15.6:0/211316103 wait complete. 2024-03-22T12:57:20.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.030+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 shutdown_connections 2024-03-22T12:57:20.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.030+0000 7ffb3e800640 1 -- 172.21.15.6:0/1165778487 wait complete. 2024-03-22T12:57:20.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.027+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc544011d10 con 0x7fc5580cc5a0 2024-03-22T12:57:20.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.030+0000 7fc54e000640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc53806bc20 0x7fc53806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.030+0000 7f75a9400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75ac15b0e0 0x7f75ac15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.030+0000 7f75a9400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75ac15b0e0 0x7f75ac15b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60876/0 (socket says 172.21.15.6:60876) 2024-03-22T12:57:20.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.030+0000 7f75a9400640 1 -- 172.21.15.6:0/2735676195 learned_addr learned my addr 172.21.15.6:0/2735676195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.030+0000 7fc555e00640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc53806bc20 0x7fc53806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.031+0000 7f3d3ea00640 1 Processor -- start 2024-03-22T12:57:20.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.031+0000 7f3d3ea00640 1 -- start start 2024-03-22T12:57:20.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.031+0000 7fc555e00640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc53806bc20 0x7fc53806e0e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc5480010e0 tx=0x7fc548007950 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.032+0000 7f3d3ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 0x7f3d38155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f3d3ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380c9b10 0x7f3d381e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f3d3ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d380ca390 0x7f3d381e5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f3d3ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380ce4a0 con 0x7f3d380ca390 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f3d3ea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380ce1c0 con 0x7f3d380c9b10 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f3d3ea00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380ce780 con 0x7f3d380c9690 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f75a9400640 1 -- 172.21.15.6:0/2735676195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac0ca520 msgr2=0x7f75ac06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f75a9400640 1 --2- 172.21.15.6:0/2735676195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac0ca520 0x7f75ac06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f75a9400640 1 -- 172.21.15.6:0/2735676195 >> v1:172.21.15.6:6789/0 conn(0x7f75ac0ca900 legacy=0x7f75ac0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.033+0000 7f75a9400640 1 -- 172.21.15.6:0/2735676195 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac06cd60 con 0x7f75ac15b0e0 2024-03-22T12:57:20.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.034+0000 7ffb51400640 1 --2- 172.21.15.6:0/1580522515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c0ca340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c009870 tx=0x7ffb3c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e7fe87c84f2b37d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.035+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe2600145b0 con 0x7fe26404b500 2024-03-22T12:57:20.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.037+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fc544057790 con 0x7fc5580cc5a0 2024-03-22T12:57:20.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.037+0000 7fe269600640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe24c06bc50 0x7fe24c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.037+0000 7fc54e000640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7fc5380756c0 0x7fc538077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.037+0000 7fe271a00640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe24c06bc50 0x7fe24c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.037+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 --> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc544014e70 con 0x7fc5380756c0 2024-03-22T12:57:20.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.037+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc544055990 con 0x7fc5580cc5a0 2024-03-22T12:57:20.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.038+0000 7fc556800640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7fc5380756c0 0x7fc538077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.038+0000 7fe271a00640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe24c06bc50 0x7fe24c06e110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0501b0 tx=0x7fe26c06a9a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.039+0000 7fc556800640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7fc5380756c0 0x7fc538077b60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.040+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0000b660 con 0x7f9c141539f0 2024-03-22T12:57:20.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.040+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c00019070 con 0x7f9c141539f0 2024-03-22T12:57:20.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.040+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0000ba60 con 0x7f9c141539f0 2024-03-22T12:57:20.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.040+0000 7f9c1a600640 1 -- 172.21.15.6:0/1685594067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c141fe0a0 con 0x7f9c141539f0 2024-03-22T12:57:20.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.040+0000 7f9c1a600640 1 -- 172.21.15.6:0/1685594067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14151460 con 0x7f9c141539f0 2024-03-22T12:57:20.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.040+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== osd.6 v2:172.21.15.92:6802/3646124871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc544014e70 con 0x7fc5380756c0 2024-03-22T12:57:20.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.040+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fe260057bd0 con 0x7fe26404b500 2024-03-22T12:57:20.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.041+0000 7fe269600640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7fe24c0756d0 0x7fe24c077b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.041+0000 7f75a9400640 1 --2- 172.21.15.6:0/2735676195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac15b0e0 0x7f75ac15b980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7598009870 tx=0x7f759802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d135bcebf938c74e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.042+0000 7ffb50a00640 1 -- 172.21.15.6:0/1580522515 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c034070 con 0x7ffb4c0cca20 2024-03-22T12:57:20.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.042+0000 7ffb50a00640 1 -- 172.21.15.6:0/1580522515 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c030840 con 0x7ffb4c0cca20 2024-03-22T12:57:20.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.042+0000 7ffb50a00640 1 -- 172.21.15.6:0/1580522515 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c034070 con 0x7ffb4c0cca20 2024-03-22T12:57:20.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.042+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 --> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe260012e50 con 0x7fe24c0756d0 2024-03-22T12:57:20.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.042+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe260055dd0 con 0x7fe26404b500 2024-03-22T12:57:20.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.042+0000 7fe272400640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7fe24c0756d0 0x7fe24c077b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.042+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380c9b10 0x7f3d381e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.043+0000 7ffb53200640 1 -- 172.21.15.6:0/1580522515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 msgr2=0x7ffb4c0ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.043+0000 7ffb53200640 1 --2- 172.21.15.6:0/1580522515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c0ca340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c009870 tx=0x7ffb3c02b980 comp rx=0 tx=0).stop 2024-03-22T12:57:20.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.043+0000 7ffb53200640 1 -- 172.21.15.6:0/1580522515 shutdown_connections 2024-03-22T12:57:20.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.043+0000 7ffb53200640 1 --2- 172.21.15.6:0/1580522515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c0ca340 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.044+0000 7ffb53200640 1 --2- 172.21.15.6:0/1580522515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 0x7ffb4c06d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.044+0000 7ffb53200640 1 -- 172.21.15.6:0/1580522515 >> 172.21.15.6:0/1580522515 conn(0x7ffb4c0cf9c0 msgr2=0x7ffb4c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.043+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9be8001030 con 0x7f9c141539f0 2024-03-22T12:57:20.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.044+0000 7fe272400640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7fe24c0756d0 0x7fe24c077b70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.045+0000 7ffb53200640 1 -- 172.21.15.6:0/1580522515 shutdown_connections 2024-03-22T12:57:20.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.043+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380c9b10 0x7f3d381e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60892/0 (socket says 172.21.15.6:60892) 2024-03-22T12:57:20.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.043+0000 7f3d2fe00640 1 -- 172.21.15.6:0/3299094536 learned_addr learned my addr 172.21.15.6:0/3299094536 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.046+0000 7ffb53200640 1 -- 172.21.15.6:0/1580522515 wait complete. 2024-03-22T12:57:20.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.049+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== osd.2 v2:172.21.15.6:6813/2211174308 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe260012e50 con 0x7fe24c0756d0 2024-03-22T12:57:20.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.049+0000 7ffb53200640 1 Processor -- start 2024-03-22T12:57:20.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.049+0000 7ffb53200640 1 -- start start 2024-03-22T12:57:20.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.050+0000 7f3d37e00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 0x7f3d38155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7ffb53200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 0x7ffb4c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7f3d2fe00640 1 -- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 msgr2=0x7f3d38155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7f3d2fe00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d380c9690 0x7f3d38155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7f3d2fe00640 1 -- 172.21.15.6:0/3299094536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d380ca390 msgr2=0x7f3d381e5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7f3d2fe00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d380ca390 0x7f3d381e5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7f3d2fe00640 1 -- 172.21.15.6:0/3299094536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d38151dc0 con 0x7f3d380c9b10 2024-03-22T12:57:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7ffb53200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.052+0000 7ffb53200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb4c15b7a0 0x7ffb4c1f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.053+0000 7ffb53200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c06cfc0 con 0x7ffb4c15b7a0 2024-03-22T12:57:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.053+0000 7ffb53200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c06cce0 con 0x7ffb4c0cca20 2024-03-22T12:57:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.053+0000 7ffb53200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c15b000 con 0x7ffb4c0cc640 2024-03-22T12:57:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.056+0000 7f3d2fe00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380c9b10 0x7f3d381e4d30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3d200090b0 tx=0x7f3d200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.055+0000 7ffb51400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.057+0000 7ffb51400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60900/0 (socket says 172.21.15.6:60900) 2024-03-22T12:57:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.057+0000 7ffb51400640 1 -- 172.21.15.6:0/1298378776 learned_addr learned my addr 172.21.15.6:0/1298378776 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.059 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.058+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 --> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f794c075650 con 0x7f79280755e0 2024-03-22T12:57:20.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-22T12:57:20.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.059+0000 7ffb51e00640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 0x7ffb4c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.059+0000 7f7942a00640 1 -- 172.21.15.6:0/477022923 <== osd.7 v2:172.21.15.92:6818/1078860650 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f794c075650 con 0x7f79280755e0 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.059+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f79280755e0 msgr2=0x7f7928077a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.059+0000 7f7950a00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f79280755e0 0x7f7928077a80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.060+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f792806bbd0 msgr2=0x7f792806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.060+0000 7f7950a00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f792806bbd0 0x7f792806e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7934006fd0 tx=0x7f7934008040 comp rx=0 tx=0).stop 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.060+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c1646f0 msgr2=0x7f794c1fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.060+0000 7f7950a00640 1 --2- 172.21.15.6:0/477022923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f794c1646f0 0x7f794c1fec30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f793c02c590 tx=0x7f793c004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.060+0000 7f794b200640 1 -- 172.21.15.6:0/477022923 reap_dead start 2024-03-22T12:57:20.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.060+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 shutdown_connections 2024-03-22T12:57:20.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.060+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 >> 172.21.15.6:0/477022923 conn(0x7f794c06f0e0 msgr2=0x7f794c0cc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.061+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 shutdown_connections 2024-03-22T12:57:20.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.061+0000 7f7950a00640 1 -- 172.21.15.6:0/477022923 wait complete. 2024-03-22T12:57:20.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.061+0000 7f75a8a00640 1 -- 172.21.15.6:0/2735676195 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598035020 con 0x7f75ac15b0e0 2024-03-22T12:57:20.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.061+0000 7f75a8a00640 1 -- 172.21.15.6:0/2735676195 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598031640 con 0x7f75ac15b0e0 2024-03-22T12:57:20.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.061+0000 7f75a8a00640 1 -- 172.21.15.6:0/2735676195 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759802d040 con 0x7f75ac15b0e0 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7ffb51400640 1 -- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 msgr2=0x7ffb4c1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7ffb51400640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb4c0cc640 0x7ffb4c1564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7ffb51400640 1 -- 172.21.15.6:0/1298378776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb4c15b7a0 msgr2=0x7ffb4c1f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7ffb51400640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb4c15b7a0 0x7ffb4c1f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.064+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20040070 con 0x7f3d380c9b10 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.064+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d20030d00 con 0x7f3d380c9b10 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.064+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20035040 con 0x7f3d380c9b10 2024-03-22T12:57:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.062+0000 7f75b0a00640 1 -- 172.21.15.6:0/2735676195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac15b0e0 msgr2=0x7f75ac15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.062+0000 7f75b0a00640 1 --2- 172.21.15.6:0/2735676195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac15b0e0 0x7f75ac15b980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7598009870 tx=0x7f759802b8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.063+0000 7f75b0a00640 1 -- 172.21.15.6:0/2735676195 shutdown_connections 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.063+0000 7f75b0a00640 1 --2- 172.21.15.6:0/2735676195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac15b0e0 0x7f75ac15b980 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.063+0000 7f75b0a00640 1 --2- 172.21.15.6:0/2735676195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac0ca520 0x7f75ac06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.063+0000 7f75b0a00640 1 -- 172.21.15.6:0/2735676195 >> 172.21.15.6:0/2735676195 conn(0x7f75ac0b16b0 msgr2=0x7f75ac0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.064+0000 7f75b0a00640 1 -- 172.21.15.6:0/2735676195 shutdown_connections 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.064+0000 7f75b0a00640 1 -- 172.21.15.6:0/2735676195 wait complete. 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.066+0000 7f75b0a00640 1 Processor -- start 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.067+0000 7f75b0a00640 1 -- start start 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f75b0a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac0ca520 0x7f75ac0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f75b0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75ac0ca900 0x7f75ac0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f75b0a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac15b0e0 0x7f75ac0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f75b0a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0cad60 con 0x7f75ac0ca900 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f75b0a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac06d320 con 0x7f75ac0ca520 2024-03-22T12:57:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f75b0a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac154c20 con 0x7f75ac15b0e0 2024-03-22T12:57:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d380c3050 con 0x7f3d380c9b10 2024-03-22T12:57:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.068+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d38154520 con 0x7f3d380c9b10 2024-03-22T12:57:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.069+0000 7f75aa800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac15b0e0 0x7f75ac0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.070+0000 7f75aa800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac15b0e0 0x7f75ac0b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57358/0 (socket says 172.21.15.6:57358) 2024-03-22T12:57:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.070+0000 7f75aa800640 1 -- 172.21.15.6:0/880849943 learned_addr learned my addr 172.21.15.6:0/880849943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.069+0000 7ffb51400640 1 -- 172.21.15.6:0/1298378776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0b1070 con 0x7ffb4c0cca20 2024-03-22T12:57:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.072+0000 7f75a9e00640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac0ca520 0x7f75ac0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.071+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3d381555f0 con 0x7f3d380c9b10 2024-03-22T12:57:20.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.074+0000 7ffb51400640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c152190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c02b9c0 tx=0x7ffb3c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.077+0000 7f75aa800640 1 -- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac0ca520 msgr2=0x7f75ac0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.077+0000 7f75aa800640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75ac0ca520 0x7f75ac0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.077+0000 7f75aa800640 1 -- 172.21.15.6:0/880849943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75ac0ca900 msgr2=0x7f75ac0b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.077+0000 7f75aa800640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75ac0ca900 0x7f75ac0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.077+0000 7f75aa800640 1 -- 172.21.15.6:0/880849943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac1f5cf0 con 0x7f75ac15b0e0 2024-03-22T12:57:20.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.080+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 --> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc5580b5b30 con 0x7fc5380756c0 2024-03-22T12:57:20.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.080+0000 7fc54e000640 1 -- 172.21.15.6:0/1603520635 <== osd.6 v2:172.21.15.92:6802/3646124871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc5580b5b30 con 0x7fc5380756c0 2024-03-22T12:57:20.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.081+0000 7f75aa800640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac15b0e0 0x7f75ac0b2c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75a400b990 tx=0x7f75a400be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.081+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7fc5380756c0 msgr2=0x7fc538077b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.081+0000 7fc55d000640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7fc5380756c0 0x7fc538077b60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.081+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc53806bc20 msgr2=0x7fc53806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.081+0000 7fc55d000640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc53806bc20 0x7fc53806e0e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc5480010e0 tx=0x7fc548007950 comp rx=0 tx=0).stop 2024-03-22T12:57:20.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.081+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580cc5a0 msgr2=0x7fc558158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.081+0000 7fc55d000640 1 --2- 172.21.15.6:0/1603520635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5580cc5a0 0x7fc558158280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc54400c430 tx=0x7fc54400c8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.082+0000 7fc556800640 1 -- 172.21.15.6:0/1603520635 reap_dead start 2024-03-22T12:57:20.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.082+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 shutdown_connections 2024-03-22T12:57:20.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.082+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 >> 172.21.15.6:0/1603520635 conn(0x7fc5580b16c0 msgr2=0x7fc5580cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.083+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 shutdown_connections 2024-03-22T12:57:20.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.083+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 --> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2640052c0 con 0x7fe24c0756d0 2024-03-22T12:57:20.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.083+0000 7fc55d000640 1 -- 172.21.15.6:0/1603520635 wait complete. 2024-03-22T12:57:20.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.084+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9c0000bc00 con 0x7f9c141539f0 2024-03-22T12:57:20.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.085+0000 7fe269600640 1 -- 172.21.15.6:0/2096332775 <== osd.2 v2:172.21.15.6:6813/2211174308 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2640052c0 con 0x7fe24c0756d0 2024-03-22T12:57:20.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.085+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7fe24c0756d0 msgr2=0x7fe24c077b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.085+0000 7fe272e00640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7fe24c0756d0 0x7fe24c077b70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.085+0000 7f9c0b400640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bf406bc20 0x7f9bf406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.087 INFO:teuthology.orchestra.run.smithi006.stdout:60129542162 2024-03-22T12:57:20.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-22T12:57:20.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe24c06bc50 msgr2=0x7fe24c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7fe272e00640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe24c06bc50 0x7fe24c06e110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0501b0 tx=0x7fe26c06a9a0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe26404b500 msgr2=0x7fe2640cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7fe272e00640 1 --2- 172.21.15.6:0/2096332775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe26404b500 0x7fe2640cecc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe26000e830 tx=0x7fe26000ecf0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7f9c13400640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bf406bc20 0x7f9bf406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4019070 con 0x7f75ac15b0e0 2024-03-22T12:57:20.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400cba0 con 0x7f75ac15b0e0 2024-03-22T12:57:20.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a4014440 con 0x7f75ac15b0e0 2024-03-22T12:57:20.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7fe272400640 1 -- 172.21.15.6:0/2096332775 reap_dead start 2024-03-22T12:57:20.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac0c6a10 con 0x7f75ac15b0e0 2024-03-22T12:57:20.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 shutdown_connections 2024-03-22T12:57:20.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 >> 172.21.15.6:0/2096332775 conn(0x7fe264050180 msgr2=0x7fe26404cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.087+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 shutdown_connections 2024-03-22T12:57:20.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.087+0000 7fe272e00640 1 -- 172.21.15.6:0/2096332775 wait complete. 2024-03-22T12:57:20.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.086+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac1f5170 con 0x7f75ac15b0e0 2024-03-22T12:57:20.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.087+0000 7f9c13400640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bf406bc20 0x7f9bf406e0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005c00 tx=0x7f9c0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.088+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f75ac0756c0 con 0x7f75ac15b0e0 2024-03-22T12:57:20.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.095+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f9c00057d00 con 0x7f9c141539f0 2024-03-22T12:57:20.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.095+0000 7f9c0b400640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f9bf4075730 0x7f9bf4077bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.095+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 --> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c00001d30 con 0x7f9bf4075730 2024-03-22T12:57:20.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.095+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c00055f00 con 0x7f9c141539f0 2024-03-22T12:57:20.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.096+0000 7f9c12a00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f9bf4075730 0x7f9bf4077bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.098+0000 7f9c12a00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f9bf4075730 0x7f9bf4077bd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.100+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== osd.3 v2:172.21.15.6:6824/484261108 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9c00001d30 con 0x7f9bf4075730 2024-03-22T12:57:20.111 INFO:teuthology.orchestra.run.smithi006.stdout:60129542162 2024-03-22T12:57:20.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-22T12:57:20.115 INFO:teuthology.orchestra.run.smithi006.stdout:60129542161 2024-03-22T12:57:20.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-22T12:57:20.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.116+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d2004b060 con 0x7f3d380c9b10 2024-03-22T12:57:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.118+0000 7f3d34c00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d1406bc50 0x7f3d1406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.118+0000 7f3d37e00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d1406bc50 0x7f3d1406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.119+0000 7f3d37e00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d1406bc50 0x7f3d1406e110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3d28006fd0 tx=0x7f3d28008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.125+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f3d20079120 con 0x7f3d380c9b10 2024-03-22T12:57:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.125+0000 7f3d34c00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7f3d140756d0 0x7f3d14077b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.125+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 --> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d20033ec0 con 0x7f3d140756d0 2024-03-22T12:57:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.126+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d20077320 con 0x7f3d380c9b10 2024-03-22T12:57:20.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.126+0000 7f3d3ca00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7f3d140756d0 0x7f3d14077b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.128+0000 7f3d3ca00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7f3d140756d0 0x7f3d14077b70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.126+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 --> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9be8002dd0 con 0x7f9bf4075730 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.127+0000 7f9c0b400640 1 -- 172.21.15.6:0/1685594067 <== osd.3 v2:172.21.15.6:6824/484261108 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9be8002dd0 con 0x7f9bf4075730 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.128+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f9bf4075730 msgr2=0x7f9bf4077bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.128+0000 7f9c08c00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f9bf4075730 0x7f9bf4077bd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bf406bc20 msgr2=0x7f9bf406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f9c08c00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bf406bc20 0x7f9bf406e0e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005c00 tx=0x7f9c0c009040 comp rx=0 tx=0).stop 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c141539f0 msgr2=0x7f9c141ed030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f9c08c00640 1 --2- 172.21.15.6:0/1685594067 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c141539f0 0x7f9c141ed030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000d800 tx=0x7f9c0000dcc0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f9c13e00640 1 -- 172.21.15.6:0/1685594067 reap_dead start 2024-03-22T12:57:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.126+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f75a401c070 con 0x7f75ac15b0e0 2024-03-22T12:57:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.127+0000 7f75a2000640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f758c06bc20 0x7f758c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.128+0000 7f75a9e00640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f758c06bc20 0x7f758c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f75a9e00640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f758c06bc20 0x7f758c06e0e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f759c0098e0 tx=0x7f759c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== osd.1 v2:172.21.15.6:6807/3168070181 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3d20033ec0 con 0x7f3d140756d0 2024-03-22T12:57:20.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 shutdown_connections 2024-03-22T12:57:20.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.130+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 >> 172.21.15.6:0/1685594067 conn(0x7f9c140cb930 msgr2=0x7f9c140744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.131+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 shutdown_connections 2024-03-22T12:57:20.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.131+0000 7f9c08c00640 1 -- 172.21.15.6:0/1685594067 wait complete. 2024-03-22T12:57:20.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.134+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c034040 con 0x7ffb4c0cca20 2024-03-22T12:57:20.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.134+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c003a20 con 0x7ffb4c0cca20 2024-03-22T12:57:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.134+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c003e10 con 0x7ffb4c0cca20 2024-03-22T12:57:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.136+0000 7ffb53200640 1 -- 172.21.15.6:0/1298378776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c1526d0 con 0x7ffb4c0cca20 2024-03-22T12:57:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.135+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f75a4057460 con 0x7f75ac15b0e0 2024-03-22T12:57:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.136+0000 7ffb53200640 1 -- 172.21.15.6:0/1298378776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c0af900 con 0x7ffb4c0cca20 2024-03-22T12:57:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.135+0000 7f75a2000640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f758c075740 0x7f758c077be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.136+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 --> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75a4010e50 con 0x7f758c075740 2024-03-22T12:57:20.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.136+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75a4055660 con 0x7f75ac15b0e0 2024-03-22T12:57:20.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.137+0000 7f75a9400640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f758c075740 0x7f758c077be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.137+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffb4c0af3d0 con 0x7ffb4c0cca20 2024-03-22T12:57:20.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.142+0000 7f75a9400640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f758c075740 0x7f758c077be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.144+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== osd.4 v2:172.21.15.92:6819/2540113998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f75a4010e50 con 0x7f758c075740 2024-03-22T12:57:20.165 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:20.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-22T12:57:20.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.169+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 --> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3d38075650 con 0x7f3d140756d0 2024-03-22T12:57:20.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.175+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffb3c04b060 con 0x7ffb4c0cca20 2024-03-22T12:57:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.172+0000 7f3d34c00640 1 -- 172.21.15.6:0/3299094536 <== osd.1 v2:172.21.15.6:6807/3168070181 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3d38075650 con 0x7f3d140756d0 2024-03-22T12:57:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.175+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7f3d140756d0 msgr2=0x7f3d14077b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.175+0000 7f3d3ea00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7f3d140756d0 0x7f3d14077b70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d1406bc50 msgr2=0x7f3d1406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d1406bc50 0x7f3d1406e110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3d28006fd0 tx=0x7f3d28008040 comp rx=0 tx=0).stop 2024-03-22T12:57:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380c9b10 msgr2=0x7f3d381e4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 --2- 172.21.15.6:0/3299094536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d380c9b10 0x7f3d381e4d30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3d200090b0 tx=0x7f3d200046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ca00640 1 -- 172.21.15.6:0/3299094536 reap_dead start 2024-03-22T12:57:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 shutdown_connections 2024-03-22T12:57:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 >> 172.21.15.6:0/3299094536 conn(0x7f3d380cfa10 msgr2=0x7f3d38159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 shutdown_connections 2024-03-22T12:57:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.176+0000 7f3d3ea00640 1 -- 172.21.15.6:0/3299094536 wait complete. 2024-03-22T12:57:20.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.180+0000 7ffb46000640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb3006bc50 0x7ffb3006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.180+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7ffb3c0ad5e0 con 0x7ffb4c0cca20 2024-03-22T12:57:20.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.180+0000 7ffb51e00640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb3006bc50 0x7ffb3006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.181+0000 7ffb46000640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7ffb300757f0 0x7ffb30077c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.181+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 --> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb30078360 con 0x7ffb300757f0 2024-03-22T12:57:20.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.181+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb3c0ada70 con 0x7ffb4c0cca20 2024-03-22T12:57:20.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.181+0000 7ffb52800640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7ffb300757f0 0x7ffb30077c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.182+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 --> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f75ac0cb080 con 0x7f758c075740 2024-03-22T12:57:20.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.185+0000 7ffb52800640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7ffb300757f0 0x7ffb30077c90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.185+0000 7ffb51e00640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb3006bc50 0x7ffb3006e110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009990 tx=0x7ffb40009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.186+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== osd.0 v2:172.21.15.6:6800/2278862740 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ffb30078360 con 0x7ffb300757f0 2024-03-22T12:57:20.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.193+0000 7f75a2000640 1 -- 172.21.15.6:0/880849943 <== osd.4 v2:172.21.15.92:6819/2540113998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f75ac0cb080 con 0x7f758c075740 2024-03-22T12:57:20.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.196+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f758c075740 msgr2=0x7f758c077be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.196+0000 7f75b0a00640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f758c075740 0x7f758c077be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.196+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f758c06bc20 msgr2=0x7f758c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.196+0000 7f75b0a00640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f758c06bc20 0x7f758c06e0e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f759c0098e0 tx=0x7f759c009060 comp rx=0 tx=0).stop 2024-03-22T12:57:20.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.196+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac15b0e0 msgr2=0x7f75ac0b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.196+0000 7f75b0a00640 1 --2- 172.21.15.6:0/880849943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75ac15b0e0 0x7f75ac0b2c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75a400b990 tx=0x7f75a400be50 comp rx=0 tx=0).stop 2024-03-22T12:57:20.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.198+0000 7f75aa800640 1 -- 172.21.15.6:0/880849943 reap_dead start 2024-03-22T12:57:20.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.198+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 shutdown_connections 2024-03-22T12:57:20.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.198+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 >> 172.21.15.6:0/880849943 conn(0x7f75ac0b16b0 msgr2=0x7f75ac0cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.198+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 shutdown_connections 2024-03-22T12:57:20.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.198+0000 7f75b0a00640 1 -- 172.21.15.6:0/880849943 wait complete. 2024-03-22T12:57:20.209 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:20.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-22T12:57:20.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.224+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 --> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffb4c075650 con 0x7ffb300757f0 2024-03-22T12:57:20.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.227+0000 7ffb46000640 1 -- 172.21.15.6:0/1298378776 <== osd.0 v2:172.21.15.6:6800/2278862740 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffb4c075650 con 0x7ffb300757f0 2024-03-22T12:57:20.233 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:20.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.232+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7ffb300757f0 msgr2=0x7ffb30077c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.232+0000 7ffb3b400640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7ffb300757f0 0x7ffb30077c90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.233+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb3006bc50 msgr2=0x7ffb3006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.233+0000 7ffb3b400640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb3006bc50 0x7ffb3006e110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009990 tx=0x7ffb40009110 comp rx=0 tx=0).stop 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.233+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 msgr2=0x7ffb4c152190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.233+0000 7ffb3b400640 1 --2- 172.21.15.6:0/1298378776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb4c0cca20 0x7ffb4c152190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c02b9c0 tx=0x7ffb3c0046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.233+0000 7ffb52800640 1 -- 172.21.15.6:0/1298378776 reap_dead start 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.234+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 shutdown_connections 2024-03-22T12:57:20.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.234+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 >> 172.21.15.6:0/1298378776 conn(0x7ffb4c0cf9c0 msgr2=0x7ffb4c15a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.234+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 shutdown_connections 2024-03-22T12:57:20.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.235+0000 7ffb3b400640 1 -- 172.21.15.6:0/1298378776 wait complete. 2024-03-22T12:57:20.274 INFO:teuthology.orchestra.run.smithi006.stdout:64424509456 2024-03-22T12:57:20.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-22T12:57:20.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.412+0000 7f55ac000640 1 Processor -- start 2024-03-22T12:57:20.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.412+0000 7f55ac000640 1 -- start start 2024-03-22T12:57:20.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.416+0000 7f55ac000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f55a4071da0 0x7f55a40d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.417+0000 7f55ac000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a4162470 con 0x7f55a40cf5d0 2024-03-22T12:57:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.417+0000 7f55ac000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a4162750 con 0x7f55a40d0e80 2024-03-22T12:57:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.417+0000 7f55ac000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a415eca0 con 0x7f55a4071da0 2024-03-22T12:57:20.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.434+0000 7f55aa000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f55a40cf5d0 0x7f55a41690a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55188/0 (socket says 172.21.15.6:55188) 2024-03-22T12:57:20.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.435+0000 7f55aa000640 1 -- 172.21.15.6:0/2785650364 learned_addr learned my addr 172.21.15.6:0/2785650364 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:20.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.435+0000 7f55a8c00640 1 --2- 172.21.15.6:0/2785650364 >> v2:172.21.15.6:3300/0 conn(0x7f55a4071da0 0x7f55a40d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.457+0000 7fec8be00640 1 Processor -- start 2024-03-22T12:57:20.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.457+0000 7fec8be00640 1 -- start start 2024-03-22T12:57:20.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.459+0000 7fec8be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec8c156080 0x7fec8c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.459+0000 7fec8be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c156490 con 0x7fec8c150270 2024-03-22T12:57:20.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.459+0000 7fec8be00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c156710 con 0x7fec8c0c7e10 2024-03-22T12:57:20.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.459+0000 7fec8be00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8c0c8770 con 0x7fec8c156080 2024-03-22T12:57:20.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.468+0000 7fec8a000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec8c156080 0x7fec8c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.469+0000 7fec8b400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fec8c0c7e10 0x7fec8c0cb8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40488/0 (socket says 172.21.15.6:40488) 2024-03-22T12:57:20.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.469+0000 7fec8b400640 1 -- 172.21.15.6:0/1896270713 learned_addr learned my addr 172.21.15.6:0/1896270713 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:20.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.475+0000 7fec8a000640 1 -- 172.21.15.6:0/1896270713 >> v1:172.21.15.92:6789/0 conn(0x7fec8c0c7e10 legacy=0x7fec8c0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.475+0000 7fec8a000640 1 -- 172.21.15.6:0/1896270713 >> v1:172.21.15.6:6789/0 conn(0x7fec8c150270 legacy=0x7fec8c0c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.475+0000 7fec8a000640 1 -- 172.21.15.6:0/1896270713 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c1569b0 con 0x7fec8c156080 2024-03-22T12:57:20.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.483+0000 7fec8a000640 1 --2- 172.21.15.6:0/1896270713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c156080 0x7fec8c0c78d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0c88f0 tx=0x7fec8402e540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da73c67c640c860e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.485+0000 7fec89600640 1 -- 172.21.15.6:0/1896270713 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8403a070 con 0x7fec8c156080 2024-03-22T12:57:20.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.486+0000 7f55a8c00640 1 -- 172.21.15.6:0/2785650364 >> v1:172.21.15.92:6789/0 conn(0x7f55a40d0e80 legacy=0x7f55a415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.486+0000 7f55a8c00640 1 -- 172.21.15.6:0/2785650364 >> v1:172.21.15.6:6789/0 conn(0x7f55a40cf5d0 legacy=0x7f55a41690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.486+0000 7f55a8c00640 1 -- 172.21.15.6:0/2785650364 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a4162a30 con 0x7f55a4071da0 2024-03-22T12:57:20.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.490+0000 7f051aa00640 1 Processor -- start 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.490+0000 7fec89600640 1 -- 172.21.15.6:0/1896270713 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8400a4e0 con 0x7fec8c156080 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec8be00640 1 -- 172.21.15.6:0/1896270713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c156080 msgr2=0x7fec8c0c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec89600640 1 -- 172.21.15.6:0/1896270713 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8400a8f0 con 0x7fec8c156080 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec8be00640 1 --2- 172.21.15.6:0/1896270713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c156080 0x7fec8c0c78d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0c88f0 tx=0x7fec8402e540 comp rx=0 tx=0).stop 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec8be00640 1 -- 172.21.15.6:0/1896270713 shutdown_connections 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec8be00640 1 --2- 172.21.15.6:0/1896270713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c156080 0x7fec8c0c78d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec8be00640 1 -- 172.21.15.6:0/1896270713 >> 172.21.15.6:0/1896270713 conn(0x7fec8c06f0e0 msgr2=0x7fec8c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec8be00640 1 -- 172.21.15.6:0/1896270713 shutdown_connections 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.491+0000 7fec8be00640 1 -- 172.21.15.6:0/1896270713 wait complete. 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.490+0000 7f051aa00640 1 -- start start 2024-03-22T12:57:20.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.494+0000 7f051aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0514150730 0x7f0514156e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.494+0000 7f051aa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05140c7fa0 0x7f05140c8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.494+0000 7fec8be00640 1 Processor -- start 2024-03-22T12:57:20.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.495+0000 7fec8be00640 1 -- start start 2024-03-22T12:57:20.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.495+0000 7f051aa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140c8970 con 0x7f05140c6db0 2024-03-22T12:57:20.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.495+0000 7f051aa00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05140c7b10 con 0x7f05140c7fa0 2024-03-22T12:57:20.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.495+0000 7f051aa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05141599b0 con 0x7f0514150730 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.497+0000 7f0518a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05140c7fa0 0x7f05140c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7f0513e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0514150730 0x7f0514156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.499+0000 7f0518a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05140c7fa0 0x7f05140c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60934/0 (socket says 172.21.15.6:60934) 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.499+0000 7f0518a00640 1 -- 172.21.15.6:0/2938006348 learned_addr learned my addr 172.21.15.6:0/2938006348 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.497+0000 7fec8be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c0c7e10 0x7fec8c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.497+0000 7fec8be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec8c150270 0x7fec8c072800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.497+0000 7fec8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec8c156080 0x7fec8c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7fec8be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c0c8400 con 0x7fec8c156080 2024-03-22T12:57:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7fec8aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c0c7e10 0x7fec8c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7fec8a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec8c150270 0x7fec8c072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7fec8be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c156c50 con 0x7fec8c0c7e10 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7fec8be00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c0c42f0 con 0x7fec8c150270 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7fec8a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec8c150270 0x7fec8c072800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57378/0 (socket says 172.21.15.6:57378) 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.498+0000 7fec8aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c0c7e10 0x7fec8c159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60936/0 (socket says 172.21.15.6:60936) 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.499+0000 7fec8aa00640 1 -- 172.21.15.6:0/3193084578 learned_addr learned my addr 172.21.15.6:0/3193084578 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.500+0000 7f11e4a00640 1 Processor -- start 2024-03-22T12:57:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.500+0000 7f11e4a00640 1 -- start start 2024-03-22T12:57:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.502+0000 7f11e4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e00cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.502+0000 7f11e4a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11e00cfe60 0x7f11e00cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.502+0000 7f11e4a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11e00d4050 0x7f11e00d44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.502+0000 7f11e4a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e0162ae0 con 0x7f11e00cfe60 2024-03-22T12:57:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.502+0000 7f11e4a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e00d49e0 con 0x7f11e00d4050 2024-03-22T12:57:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.502+0000 7f11e4a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0161bf0 con 0x7f11e01587c0 2024-03-22T12:57:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.503+0000 7f0513e00640 1 -- 172.21.15.6:0/2938006348 >> v2:172.21.15.92:3300/0 conn(0x7f05140c7fa0 msgr2=0x7f05140c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.503+0000 7f0513e00640 1 --2- 172.21.15.6:0/2938006348 >> v2:172.21.15.92:3300/0 conn(0x7f05140c7fa0 0x7f05140c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.503+0000 7f0513e00640 1 -- 172.21.15.6:0/2938006348 >> v1:172.21.15.6:6789/0 conn(0x7f05140c6db0 legacy=0x7f0514072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.503+0000 7f0513e00640 1 -- 172.21.15.6:0/2938006348 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514159c00 con 0x7f0514150730 2024-03-22T12:57:20.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.506+0000 7f0513e00640 1 --2- 172.21.15.6:0/2938006348 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0514150730 0x7f0514156e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f05040097a0 tx=0x7f050402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f868b90564f71d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.506+0000 7f11de800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e00cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.507+0000 7f11df200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11e00d4050 0x7f11e00d44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.507+0000 7f11de800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e00cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57392/0 (socket says 172.21.15.6:57392) 2024-03-22T12:57:20.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.507+0000 7f11de800640 1 -- 172.21.15.6:0/372743681 learned_addr learned my addr 172.21.15.6:0/372743681 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.511+0000 7fec8aa00640 1 -- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec8c150270 msgr2=0x7fec8c072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.511+0000 7fec8aa00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec8c150270 0x7fec8c072800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.511+0000 7fec8aa00640 1 -- 172.21.15.6:0/3193084578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec8c156080 msgr2=0x7fec8c157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.511+0000 7fec8aa00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec8c156080 0x7fec8c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.511+0000 7fec8aa00640 1 -- 172.21.15.6:0/3193084578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c1587f0 con 0x7fec8c0c7e10 2024-03-22T12:57:20.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.513+0000 7f55a8c00640 1 --2- 172.21.15.6:0/2785650364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a40d1260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f55a415ee20 tx=0x7f559802ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7685acbbaa09e42a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.515+0000 7fec8aa00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c0c7e10 0x7fec8c159e80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00c440 tx=0x7fec7c00c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.517+0000 7f11df200640 1 -- 172.21.15.6:0/372743681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 msgr2=0x7f11e00cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.517+0000 7f11df200640 1 --2- 172.21.15.6:0/372743681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e00cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.517+0000 7f11df200640 1 -- 172.21.15.6:0/372743681 >> v2:172.21.15.6:3300/0 conn(0x7f11e00cfe60 msgr2=0x7f11e00cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.517+0000 7f11df200640 1 --2- 172.21.15.6:0/372743681 >> v2:172.21.15.6:3300/0 conn(0x7f11e00cfe60 0x7f11e00cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.517+0000 7f11df200640 1 -- 172.21.15.6:0/372743681 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0161ed0 con 0x7f11e00d4050 2024-03-22T12:57:20.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.520+0000 7f0512a00640 1 -- 172.21.15.6:0/2938006348 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504035020 con 0x7f0514150730 2024-03-22T12:57:20.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.520+0000 7f0512a00640 1 -- 172.21.15.6:0/2938006348 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504031830 con 0x7f0514150730 2024-03-22T12:57:20.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.521+0000 7f0512a00640 1 -- 172.21.15.6:0/2938006348 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504035020 con 0x7f0514150730 2024-03-22T12:57:20.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.525+0000 7f11df200640 1 --2- 172.21.15.6:0/372743681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00d4050 0x7f11e00d44a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f11e006ea40 tx=0x7f11d002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f46d55a69a8b60c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.522+0000 7f051aa00640 1 -- 172.21.15.6:0/2938006348 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0514150730 msgr2=0x7f0514156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.522+0000 7f051aa00640 1 --2- 172.21.15.6:0/2938006348 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0514150730 0x7f0514156e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f05040097a0 tx=0x7f050402b8d0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.522+0000 7f051aa00640 1 -- 172.21.15.6:0/2938006348 shutdown_connections 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.522+0000 7f051aa00640 1 --2- 172.21.15.6:0/2938006348 >> v2:172.21.15.92:3300/0 conn(0x7f05140c7fa0 0x7f05140c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.522+0000 7f051aa00640 1 --2- 172.21.15.6:0/2938006348 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0514150730 0x7f0514156e80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.522+0000 7f051aa00640 1 -- 172.21.15.6:0/2938006348 >> 172.21.15.6:0/2938006348 conn(0x7f051406e310 msgr2=0x7f051406f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.522+0000 7f051aa00640 1 -- 172.21.15.6:0/2938006348 shutdown_connections 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.523+0000 7f051aa00640 1 -- 172.21.15.6:0/2938006348 wait complete. 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.525+0000 7f051aa00640 1 Processor -- start 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.526+0000 7f051aa00640 1 -- start start 2024-03-22T12:57:20.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.527+0000 7fec82a00640 1 -- 172.21.15.6:0/3193084578 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c017070 con 0x7fec8c0c7e10 2024-03-22T12:57:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.527+0000 7fec82a00640 1 -- 172.21.15.6:0/3193084578 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c0035f0 con 0x7fec8c0c7e10 2024-03-22T12:57:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.527+0000 7fec82a00640 1 -- 172.21.15.6:0/3193084578 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0039e0 con 0x7fec8c0c7e10 2024-03-22T12:57:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.528+0000 7f051aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05140c6db0 0x7f0514150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.528+0000 7f051aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05140c7fa0 0x7f05140cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.527+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8c1573c0 con 0x7fec8c0c7e10 2024-03-22T12:57:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.528+0000 7f051aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0514150730 0x7f05140cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.527+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c1f4ce0 con 0x7fec8c0c7e10 2024-03-22T12:57:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.528+0000 7f051aa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051415a4a0 con 0x7f0514150730 2024-03-22T12:57:20.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.528+0000 7f051aa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051415a1c0 con 0x7f05140c6db0 2024-03-22T12:57:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.528+0000 7f051aa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051415a780 con 0x7f05140c7fa0 2024-03-22T12:57:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.529+0000 7fda57800640 1 Processor -- start 2024-03-22T12:57:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.529+0000 7f0513400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05140c7fa0 0x7f05140cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.529+0000 7f0513400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05140c7fa0 0x7f05140cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57404/0 (socket says 172.21.15.6:57404) 2024-03-22T12:57:20.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.529+0000 7f0513400640 1 -- 172.21.15.6:0/1705114046 learned_addr learned my addr 172.21.15.6:0/1705114046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.530+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec64005290 con 0x7fec8c0c7e10 2024-03-22T12:57:20.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.530+0000 7fda57800640 1 -- start start 2024-03-22T12:57:20.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.532+0000 7f11dd400640 1 -- 172.21.15.6:0/372743681 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0035020 con 0x7f11e00d4050 2024-03-22T12:57:20.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.532+0000 7fda57800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda500cf310 0x7fda500ca550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.532+0000 7fda57800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda500cacd0 con 0x7fda500cef30 2024-03-22T12:57:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.533+0000 7fda57800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda500cc8c0 con 0x7fda500cdbf0 2024-03-22T12:57:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.533+0000 7fda57800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500caa90 con 0x7fda500cf310 2024-03-22T12:57:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.534+0000 7f0513e00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05140c6db0 0x7f0514150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.535+0000 7f0513400640 1 -- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05140c6db0 msgr2=0x7f0514150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.535+0000 7f0513400640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05140c6db0 0x7f0514150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.535+0000 7f0513400640 1 -- 172.21.15.6:0/1705114046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0514150730 msgr2=0x7f05140cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.535+0000 7f0513400640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0514150730 0x7f05140cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.535+0000 7f0513400640 1 -- 172.21.15.6:0/1705114046 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140c3d20 con 0x7f05140c7fa0 2024-03-22T12:57:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.535+0000 7fda55800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fda500cdbf0 0x7fda500ce080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40500/0 (socket says 172.21.15.6:40500) 2024-03-22T12:57:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.535+0000 7fda55800640 1 -- 172.21.15.6:0/2944970 learned_addr learned my addr 172.21.15.6:0/2944970 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:20.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.538+0000 7f55a3e00640 1 -- 172.21.15.6:0/2785650364 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559802f820 con 0x7f55a4071da0 2024-03-22T12:57:20.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.538+0000 7f55a3e00640 1 -- 172.21.15.6:0/2785650364 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559802f9c0 con 0x7f55a4071da0 2024-03-22T12:57:20.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.538+0000 7f55a3e00640 1 -- 172.21.15.6:0/2785650364 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559802fdd0 con 0x7f55a4071da0 2024-03-22T12:57:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.539+0000 7f55ac000640 1 -- 172.21.15.6:0/2785650364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 msgr2=0x7f55a40d1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.539+0000 7f55ac000640 1 --2- 172.21.15.6:0/2785650364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a40d1260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f55a415ee20 tx=0x7f559802ba00 comp rx=0 tx=0).stop 2024-03-22T12:57:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.539+0000 7f55ac000640 1 -- 172.21.15.6:0/2785650364 shutdown_connections 2024-03-22T12:57:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.539+0000 7f55ac000640 1 --2- 172.21.15.6:0/2785650364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a40d1260 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.539+0000 7f55ac000640 1 -- 172.21.15.6:0/2785650364 >> 172.21.15.6:0/2785650364 conn(0x7f55a406f130 msgr2=0x7f55a406f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.540+0000 7f55ac000640 1 -- 172.21.15.6:0/2785650364 shutdown_connections 2024-03-22T12:57:20.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.540+0000 7f0513400640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05140c7fa0 0x7f05140cb650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f050000c440 tx=0x7f050000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.540+0000 7f55ac000640 1 -- 172.21.15.6:0/2785650364 wait complete. 2024-03-22T12:57:20.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.542+0000 7f55ac000640 1 Processor -- start 2024-03-22T12:57:20.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.543+0000 7f55ac000640 1 -- start start 2024-03-22T12:57:20.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.544+0000 7fda4fe00640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda500cf310 0x7fda500ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.545+0000 7f55ac000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a41e9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.546+0000 7f55ac000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55a40cf5d0 0x7f55a40d3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.546+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436079931 0 0) 0x7fda500cacd0 con 0x7fda500cef30 2024-03-22T12:57:20.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.546+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda34003520 con 0x7fda500cef30 2024-03-22T12:57:20.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.546+0000 7f55ac000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55a40d0e80 0x7f55a40d41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.546+0000 7f55ac000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a4158e20 con 0x7f55a4071da0 2024-03-22T12:57:20.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.546+0000 7f55ac000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a4158ca0 con 0x7f55a40cf5d0 2024-03-22T12:57:20.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.546+0000 7f55ac000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a40d08d0 con 0x7f55a40d0e80 2024-03-22T12:57:20.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.547+0000 7f55a9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a41e9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.548+0000 7f55a9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a41e9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33936/0 (socket says 172.21.15.6:33936) 2024-03-22T12:57:20.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.548+0000 7f55a9600640 1 -- 172.21.15.6:0/1649458592 learned_addr learned my addr 172.21.15.6:0/1649458592 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.548+0000 7f050be00640 1 -- 172.21.15.6:0/1705114046 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500017070 con 0x7f05140c7fa0 2024-03-22T12:57:20.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.548+0000 7f050be00640 1 -- 172.21.15.6:0/1705114046 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05000035f0 con 0x7f05140c7fa0 2024-03-22T12:57:20.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.548+0000 7f050be00640 1 -- 172.21.15.6:0/1705114046 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05000039e0 con 0x7f05140c7fa0 2024-03-22T12:57:20.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.548+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0514152f00 con 0x7f05140c7fa0 2024-03-22T12:57:20.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.548+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05141534c0 con 0x7f05140c7fa0 2024-03-22T12:57:20.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.550+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05140757d0 con 0x7f05140c7fa0 2024-03-22T12:57:20.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.557+0000 7f55a9600640 1 -- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55a40d0e80 msgr2=0x7f55a40d41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.557+0000 7f55a9600640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55a40d0e80 0x7f55a40d41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.557+0000 7f55a9600640 1 -- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55a40cf5d0 msgr2=0x7f55a40d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.557+0000 7f55a9600640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55a40cf5d0 0x7f55a40d3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7f55a9600640 1 -- 172.21.15.6:0/1649458592 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a41ea760 con 0x7f55a4071da0 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1270576409 0 0) 0x7fda34003520 con 0x7fda500cef30 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda500cf310 msgr2=0x7fda500ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda500cf310 0x7fda500ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 >> v1:172.21.15.92:6789/0 conn(0x7fda500cdbf0 legacy=0x7fda500ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500ccba0 con 0x7fda500cef30 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7fda3c001670 con 0x7fda500cef30 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 >> v1:172.21.15.6:6789/0 conn(0x7fda500cef30 legacy=0x7fda5015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda340039e0 0x7fda34003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda340043b0 0x7fda34004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda34005080 0x7fda3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500caa90 con 0x7fda34005080 2024-03-22T12:57:20.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3400bf90 con 0x7fda340043b0 2024-03-22T12:57:20.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.558+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3400c190 con 0x7fda340039e0 2024-03-22T12:57:20.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.561+0000 7f55a9600640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a41e9ca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c730 tx=0x7f559c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.566+0000 7fda55800640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda34005080 0x7fda3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.566+0000 7f55a1600640 1 -- 172.21.15.6:0/1649458592 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c016020 con 0x7f55a4071da0 2024-03-22T12:57:20.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.567+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a41e7ac0 con 0x7f55a4071da0 2024-03-22T12:57:20.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.567+0000 7f55a1600640 1 -- 172.21.15.6:0/1649458592 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c006d50 con 0x7f55a4071da0 2024-03-22T12:57:20.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.567+0000 7f55a1600640 1 -- 172.21.15.6:0/1649458592 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c016020 con 0x7f55a4071da0 2024-03-22T12:57:20.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.567+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a40ce140 con 0x7f55a4071da0 2024-03-22T12:57:20.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.568+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a40757d0 con 0x7f55a4071da0 2024-03-22T12:57:20.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.579+0000 7fe03cc00640 1 Processor -- start 2024-03-22T12:57:20.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.579+0000 7fe03cc00640 1 -- start start 2024-03-22T12:57:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.581+0000 7fe03cc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe038151140 0x7fe038073180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.581+0000 7f050be00640 1 -- 172.21.15.6:0/1705114046 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f050001a070 con 0x7f05140c7fa0 2024-03-22T12:57:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.581+0000 7fe03cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0380c7f90 con 0x7fe038157c00 2024-03-22T12:57:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.581+0000 7fe03cc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0380c76f0 con 0x7fe0381577c0 2024-03-22T12:57:20.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.581+0000 7fe03cc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0380736c0 con 0x7fe038151140 2024-03-22T12:57:20.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.582+0000 7f050be00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04f406bc20 0x7f04f406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.583+0000 7f0513e00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04f406bc20 0x7f04f406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.584+0000 7f0513e00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04f406bc20 0x7f04f406e0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0504036000 tx=0x7f050403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.595+0000 7f55a1600640 1 -- 172.21.15.6:0/1649458592 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f559c003970 con 0x7f55a4071da0 2024-03-22T12:57:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.595+0000 7f55a1600640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f558406be00 0x7f558406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.596+0000 7f55a8c00640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f558406be00 0x7f558406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.597+0000 7f55a8c00640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f558406be00 0x7f558406e2c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5598036000 tx=0x7f5598039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.601+0000 7f55a1600640 1 -- 172.21.15.6:0/1649458592 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f559c057630 con 0x7f55a4071da0 2024-03-22T12:57:20.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.607+0000 7f46d9200640 1 Processor -- start 2024-03-22T12:57:20.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.608+0000 7f46d9200640 1 -- start start 2024-03-22T12:57:20.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.610+0000 7f46d9200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f46d40c4b90 0x7f46d40cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.610+0000 7f46d9200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40c5270 con 0x7f46d41547f0 2024-03-22T12:57:20.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.610+0000 7f46d9200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d415e380 con 0x7f46d40c5570 2024-03-22T12:57:20.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.610+0000 7f46d9200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d40c5020 con 0x7f46d40c4b90 2024-03-22T12:57:20.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.612+0000 7fda55800640 1 -- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda340039e0 msgr2=0x7fda34003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.612+0000 7fda55800640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda340039e0 0x7fda34003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.612+0000 7fda55800640 1 -- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda340043b0 msgr2=0x7fda34004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.612+0000 7fda55800640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda340043b0 0x7fda34004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.612+0000 7fda55800640 1 -- 172.21.15.6:0/2944970 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500ccba0 con 0x7fda34005080 2024-03-22T12:57:20.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.616+0000 7fda55800640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda34005080 0x7fda3400b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fda40005130 tx=0x7fda34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f77745d191491fe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.653+0000 7fd68a400640 1 Processor -- start 2024-03-22T12:57:20.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.653+0000 7fd68a400640 1 -- start start 2024-03-22T12:57:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.655+0000 7fd68a400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd68414ebd0 0x7fd6841588b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.655+0000 7fd68a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6840cd640 0x7fd684162920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.655+0000 7fd68a400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd684073570 con 0x7fd684072770 2024-03-22T12:57:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.655+0000 7fd68a400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd684072cd0 con 0x7fd68414ebd0 2024-03-22T12:57:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.655+0000 7fd68a400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd684072fb0 con 0x7fd6840cd640 2024-03-22T12:57:20.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.820+0000 7f11dd400640 1 -- 172.21.15.6:0/372743681 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d0031830 con 0x7f11e00d4050 2024-03-22T12:57:20.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.820+0000 7f11dd400640 1 -- 172.21.15.6:0/372743681 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0031c00 con 0x7f11e00d4050 2024-03-22T12:57:20.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 -- 172.21.15.6:0/372743681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00d4050 msgr2=0x7f11e00d44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 --2- 172.21.15.6:0/372743681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00d4050 0x7f11e00d44a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f11e006ea40 tx=0x7f11d002b8d0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 -- 172.21.15.6:0/372743681 shutdown_connections 2024-03-22T12:57:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 --2- 172.21.15.6:0/372743681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00d4050 0x7f11e00d44a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 --2- 172.21.15.6:0/372743681 >> v2:172.21.15.6:3300/0 conn(0x7f11e00cfe60 0x7f11e00cf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 --2- 172.21.15.6:0/372743681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e00cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 -- 172.21.15.6:0/372743681 >> 172.21.15.6:0/372743681 conn(0x7f11e006f090 msgr2=0x7f11e006f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.821+0000 7f11e4a00640 1 -- 172.21.15.6:0/372743681 shutdown_connections 2024-03-22T12:57:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.822+0000 7f11e4a00640 1 -- 172.21.15.6:0/372743681 wait complete. 2024-03-22T12:57:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.824+0000 7f11e4a00640 1 Processor -- start 2024-03-22T12:57:20.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.824+0000 7f11e4a00640 1 -- start start 2024-03-22T12:57:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.825+0000 7f11e4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00cfe60 0x7f11e015e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.825+0000 7f11e4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11e00d4050 0x7f11e015ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.825+0000 7f11e4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e01f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.825+0000 7f11e4a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0162770 con 0x7f11e00d4050 2024-03-22T12:57:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.825+0000 7f11e4a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0162490 con 0x7f11e00cfe60 2024-03-22T12:57:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.825+0000 7f11e4a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0158ba0 con 0x7f11e01587c0 2024-03-22T12:57:20.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.850+0000 7fd683400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd68414ebd0 0x7fd6841588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.874+0000 7fd683400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd68414ebd0 0x7fd6841588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:32770/0 (socket says 172.21.15.6:32770) 2024-03-22T12:57:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.874+0000 7fd683400640 1 -- 172.21.15.6:0/4017730581 learned_addr learned my addr 172.21.15.6:0/4017730581 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.876+0000 7fec82a00640 1 -- 172.21.15.6:0/3193084578 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fec7c01a070 con 0x7fec8c0c7e10 2024-03-22T12:57:20.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.878+0000 7fec82a00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6c06bc20 0x7fec6c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.878+0000 7fec8a000640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6c06bc20 0x7fec6c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.879+0000 7fec8a000640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6c06bc20 0x7fec6c06e0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec84037000 tx=0x7fec8400e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.881+0000 7fec82a00640 1 -- 172.21.15.6:0/3193084578 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fec7c0566f0 con 0x7fec8c0c7e10 2024-03-22T12:57:20.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.899+0000 7fe036800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe038157c00 0x7fe038161830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55244/0 (socket says 172.21.15.6:55244) 2024-03-22T12:57:20.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.899+0000 7fe036800640 1 -- 172.21.15.6:0/1065306082 learned_addr learned my addr 172.21.15.6:0/1065306082 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:20.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.900+0000 7f46d1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f46d41547f0 0x7f46d40c6450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55260/0 (socket says 172.21.15.6:55260) 2024-03-22T12:57:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.900+0000 7f46d1e00640 1 -- 172.21.15.6:0/878121963 learned_addr learned my addr 172.21.15.6:0/878121963 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.901+0000 7fec82a00640 1 -- 172.21.15.6:0/3193084578 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fec7c021020 con 0x7fec8c0c7e10 2024-03-22T12:57:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.904+0000 7fd683400640 1 -- 172.21.15.6:0/4017730581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6840cd640 msgr2=0x7fd684162920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.904+0000 7fd683400640 1 --2- 172.21.15.6:0/4017730581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6840cd640 0x7fd684162920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.904+0000 7fd683400640 1 -- 172.21.15.6:0/4017730581 >> v1:172.21.15.6:6789/0 conn(0x7fd684072770 legacy=0x7fd6840c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.904+0000 7fd683400640 1 -- 172.21.15.6:0/4017730581 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd684073290 con 0x7fd68414ebd0 2024-03-22T12:57:20.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.908+0000 7f46d1400640 1 --2- 172.21.15.6:0/878121963 >> v2:172.21.15.92:3300/0 conn(0x7f46d40c4b90 0x7f46d40cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.911+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40004d00 con 0x7fda34005080 2024-03-22T12:57:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.911+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda40005ad0 con 0x7fda34005080 2024-03-22T12:57:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.911+0000 7fda4f400640 1 -- 172.21.15.6:0/2944970 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40007070 con 0x7fda34005080 2024-03-22T12:57:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fd683400640 1 --2- 172.21.15.6:0/4017730581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841588b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd674009870 tx=0x7fd67402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35f9867649f03c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fda57800640 1 -- 172.21.15.6:0/2944970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda34005080 msgr2=0x7fda3400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fda57800640 1 --2- 172.21.15.6:0/2944970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda34005080 0x7fda3400b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fda40005130 tx=0x7fda34005830 comp rx=0 tx=0).stop 2024-03-22T12:57:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fda55800640 1 -- 172.21.15.6:0/2944970 reap_dead start 2024-03-22T12:57:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fda57800640 1 -- 172.21.15.6:0/2944970 shutdown_connections 2024-03-22T12:57:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fda57800640 1 -- 172.21.15.6:0/2944970 >> 172.21.15.6:0/2944970 conn(0x7fda500b1700 msgr2=0x7fda500b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fda57800640 1 -- 172.21.15.6:0/2944970 shutdown_connections 2024-03-22T12:57:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.912+0000 7fda57800640 1 -- 172.21.15.6:0/2944970 wait complete. 2024-03-22T12:57:20.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.914+0000 7fda57800640 1 Processor -- start 2024-03-22T12:57:20.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.914+0000 7fda57800640 1 -- start start 2024-03-22T12:57:20.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7f46d1400640 1 -- 172.21.15.6:0/878121963 >> v1:172.21.15.92:6789/0 conn(0x7f46d40c5570 legacy=0x7f46d40ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7f46d1400640 1 -- 172.21.15.6:0/878121963 >> v1:172.21.15.6:6789/0 conn(0x7f46d41547f0 legacy=0x7f46d40c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7f46d1400640 1 -- 172.21.15.6:0/878121963 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d415e660 con 0x7f46d40c4b90 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131768124 0 0) 0x7fe0380c7f90 con 0x7fe038157c00 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c003520 con 0x7fe038157c00 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fda57800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda34005080 0x7fda5015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fda57800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda501e5040 0x7fda5015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fda57800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda50155e00 0x7fda501562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fda57800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500cd440 con 0x7fda50155e00 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fda57800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500cd160 con 0x7fda501e5040 2024-03-22T12:57:20.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.916+0000 7fda57800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500cf750 con 0x7fda34005080 2024-03-22T12:57:20.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.919+0000 7f11de800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00cfe60 0x7f11e015e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.923+0000 7f11de800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00cfe60 0x7f11e015e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:32772/0 (socket says 172.21.15.6:32772) 2024-03-22T12:57:20.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.923+0000 7f11de800640 1 -- 172.21.15.6:0/1384129997 learned_addr learned my addr 172.21.15.6:0/1384129997 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.926+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 520053199 0 0) 0x7fe01c003520 con 0x7fe038157c00 2024-03-22T12:57:20.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.926+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe038151140 msgr2=0x7fe038073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.926+0000 7fe034a00640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe038151140 0x7fe038073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.926+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 >> v1:172.21.15.92:6789/0 conn(0x7fe0381577c0 legacy=0x7fe0380c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0380c79d0 con 0x7fe038157c00 2024-03-22T12:57:20.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7fe0300019f0 con 0x7fe038157c00 2024-03-22T12:57:20.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 >> v1:172.21.15.6:6789/0 conn(0x7fe038157c00 legacy=0x7fe038161830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7f11dde00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11e00d4050 0x7f11e015ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe038151140 0x7fe01c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe01c0040f0 0x7fe01c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe01c004e60 0x7fe01c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0055a0 con 0x7fe01c004e60 2024-03-22T12:57:20.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c00bf00 con 0x7fe01c0040f0 2024-03-22T12:57:20.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c00c080 con 0x7fe038151140 2024-03-22T12:57:20.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.927+0000 7fda4fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda501e5040 0x7fda5015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.932+0000 7f11de800640 1 -- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 msgr2=0x7f11e01f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.932+0000 7f11de800640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e01f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.932+0000 7f11de800640 1 -- 172.21.15.6:0/1384129997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11e00d4050 msgr2=0x7f11e015ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.932+0000 7f11de800640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11e00d4050 0x7f11e015ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.932+0000 7f11de800640 1 -- 172.21.15.6:0/1384129997 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e01e9030 con 0x7f11e00cfe60 2024-03-22T12:57:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.933+0000 7fda4fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda501e5040 0x7fda5015e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:32782/0 (socket says 172.21.15.6:32782) 2024-03-22T12:57:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.933+0000 7fda4fe00640 1 -- 172.21.15.6:0/2588269295 learned_addr learned my addr 172.21.15.6:0/2588269295 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.942+0000 7fda4fe00640 1 -- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda34005080 msgr2=0x7fda5015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.942+0000 7fda4fe00640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda34005080 0x7fda5015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.942+0000 7fda4fe00640 1 -- 172.21.15.6:0/2588269295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda50155e00 msgr2=0x7fda501562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.942+0000 7fda4fe00640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda50155e00 0x7fda501562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.942+0000 7fda4fe00640 1 -- 172.21.15.6:0/2588269295 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda501f0d80 con 0x7fda501e5040 2024-03-22T12:57:20.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.944+0000 7f46d1400640 1 --2- 172.21.15.6:0/878121963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc620 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f46c00097a0 tx=0x7f46c002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d492a20a7b5d2b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.945+0000 7f55a1600640 1 -- 172.21.15.6:0/1649458592 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f559c020020 con 0x7f55a4071da0 2024-03-22T12:57:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.947+0000 7f11de800640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00cfe60 0x7f11e015e320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00d800 tx=0x7f11cc00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.953+0000 7fda4fe00640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda501e5040 0x7fda5015e2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda48009c70 tx=0x7fda4800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.954+0000 7f46d0a00640 1 -- 172.21.15.6:0/878121963 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c00315a0 con 0x7f46d40c4b90 2024-03-22T12:57:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.954+0000 7f46d0a00640 1 -- 172.21.15.6:0/878121963 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c0037070 con 0x7f46d40c4b90 2024-03-22T12:57:20.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.954+0000 7f46d0a00640 1 -- 172.21.15.6:0/878121963 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0031960 con 0x7f46d40c4b90 2024-03-22T12:57:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.955+0000 7f46d9200640 1 -- 172.21.15.6:0/878121963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c4b90 msgr2=0x7f46d40cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.955+0000 7f46d9200640 1 --2- 172.21.15.6:0/878121963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc620 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f46c00097a0 tx=0x7f46c002b840 comp rx=0 tx=0).stop 2024-03-22T12:57:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.955+0000 7f46d9200640 1 -- 172.21.15.6:0/878121963 shutdown_connections 2024-03-22T12:57:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.955+0000 7f46d9200640 1 --2- 172.21.15.6:0/878121963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc620 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.955+0000 7f46d9200640 1 -- 172.21.15.6:0/878121963 >> 172.21.15.6:0/878121963 conn(0x7f46d4066e40 msgr2=0x7f46d4067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.955+0000 7f46d9200640 1 -- 172.21.15.6:0/878121963 shutdown_connections 2024-03-22T12:57:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.956+0000 7f050be00640 1 -- 172.21.15.6:0/1705114046 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f0500056790 con 0x7f05140c7fa0 2024-03-22T12:57:20.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.958+0000 7fe036800640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe01c004e60 0x7fe01c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.955+0000 7f46d9200640 1 -- 172.21.15.6:0/878121963 wait complete. 2024-03-22T12:57:20.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.958+0000 7fd682a00640 1 -- 172.21.15.6:0/4017730581 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd674035020 con 0x7fd68414ebd0 2024-03-22T12:57:20.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.958+0000 7fd682a00640 1 -- 172.21.15.6:0/4017730581 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd674031640 con 0x7fd68414ebd0 2024-03-22T12:57:20.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.958+0000 7fd682a00640 1 -- 172.21.15.6:0/4017730581 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67402d040 con 0x7fd68414ebd0 2024-03-22T12:57:20.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.958+0000 7f46d9200640 1 Processor -- start 2024-03-22T12:57:20.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.958+0000 7f46d9200640 1 -- start start 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.959+0000 7fd68a400640 1 -- 172.21.15.6:0/4017730581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 msgr2=0x7fd6841588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.959+0000 7fd68a400640 1 --2- 172.21.15.6:0/4017730581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841588b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd674009870 tx=0x7fd67402b8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.959+0000 7fd68a400640 1 -- 172.21.15.6:0/4017730581 shutdown_connections 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.959+0000 7fd68a400640 1 --2- 172.21.15.6:0/4017730581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6840cd640 0x7fd684162920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.959+0000 7fd68a400640 1 --2- 172.21.15.6:0/4017730581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841588b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.959+0000 7fd68a400640 1 -- 172.21.15.6:0/4017730581 >> 172.21.15.6:0/4017730581 conn(0x7fd6840ccb50 msgr2=0x7fd6840ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.959+0000 7fd68a400640 1 -- 172.21.15.6:0/4017730581 shutdown_connections 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.960+0000 7fd68a400640 1 -- 172.21.15.6:0/4017730581 wait complete. 2024-03-22T12:57:20.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.960+0000 7f46d9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.960+0000 7f46d9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c5570 0x7f46d42006d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.960+0000 7f46d9200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f46d41547f0 0x7f46d4200c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.960+0000 7f46d9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40cd270 con 0x7f46d40c4b90 2024-03-22T12:57:20.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.960+0000 7f46d9200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40c5f30 con 0x7f46d40c5570 2024-03-22T12:57:20.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.960+0000 7f46d9200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40ca5d0 con 0x7f46d41547f0 2024-03-22T12:57:20.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.962+0000 7fd68a400640 1 Processor -- start 2024-03-22T12:57:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.962+0000 7fd68a400640 1 -- start start 2024-03-22T12:57:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.963+0000 7fd68a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd684072770 0x7fd6840ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.963+0000 7fd68a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6840cd640 0x7fd684142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.964+0000 7fd68a400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:20.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.964+0000 7fd68a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68406d030 con 0x7fd6840cd640 2024-03-22T12:57:20.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.964+0000 7fd68a400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684158390 con 0x7fd68414ebd0 2024-03-22T12:57:20.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.964+0000 7fd68a400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68415b590 con 0x7fd684072770 2024-03-22T12:57:20.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.966+0000 7fe036800640 1 -- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe038151140 msgr2=0x7fe01c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.966+0000 7fe036800640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe038151140 0x7fe01c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.966+0000 7fe036800640 1 -- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe01c0040f0 msgr2=0x7fe01c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.966+0000 7fe036800640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe01c0040f0 0x7fe01c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.966+0000 7fe036800640 1 -- 172.21.15.6:0/1065306082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0380c79d0 con 0x7fe01c004e60 2024-03-22T12:57:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.970+0000 7f050be00640 1 -- 172.21.15.6:0/1705114046 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0500021020 con 0x7f05140c7fa0 2024-03-22T12:57:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.972+0000 7f11d6a00640 1 -- 172.21.15.6:0/1384129997 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc018020 con 0x7f11e00cfe60 2024-03-22T12:57:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.972+0000 7f11d6a00640 1 -- 172.21.15.6:0/1384129997 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11cc0096a0 con 0x7f11e00cfe60 2024-03-22T12:57:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.972+0000 7f11d6a00640 1 -- 172.21.15.6:0/1384129997 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc009a90 con 0x7f11e00cfe60 2024-03-22T12:57:20.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.972+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e01f50f0 con 0x7f11e00cfe60 2024-03-22T12:57:20.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.973+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e01e95b0 con 0x7f11e00cfe60 2024-03-22T12:57:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.974+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e00678d0 con 0x7f11e00cfe60 2024-03-22T12:57:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.979+0000 7fda4cc00640 1 -- 172.21.15.6:0/2588269295 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48003990 con 0x7fda501e5040 2024-03-22T12:57:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.979+0000 7fda4cc00640 1 -- 172.21.15.6:0/2588269295 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48019070 con 0x7fda501e5040 2024-03-22T12:57:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.979+0000 7fda4cc00640 1 -- 172.21.15.6:0/2588269295 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48003d70 con 0x7fda501e5040 2024-03-22T12:57:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.979+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5015b640 con 0x7fda501e5040 2024-03-22T12:57:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.979+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5015bc00 con 0x7fda501e5040 2024-03-22T12:57:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.980+0000 7f46d1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c5570 0x7f46d42006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.980+0000 7fd683e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.981+0000 7f46d1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c5570 0x7f46d42006d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:32798/0 (socket says 172.21.15.6:32798) 2024-03-22T12:57:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.981+0000 7f46d1400640 1 -- 172.21.15.6:0/527246134 learned_addr learned my addr 172.21.15.6:0/527246134 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.981+0000 7fd683e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841476a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:32804/0 (socket says 172.21.15.6:32804) 2024-03-22T12:57:20.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.981+0000 7fd683e00640 1 -- 172.21.15.6:0/4042696128 learned_addr learned my addr 172.21.15.6:0/4042696128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:20.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.981+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda500ca220 con 0x7fda501e5040 2024-03-22T12:57:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.992+0000 7fd683400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd684072770 0x7fd6840ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:20.993+0000 7f46d2800640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f46d41547f0 0x7f46d4200c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.025+0000 7f46d1e00640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.026+0000 7fd67be00640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6840cd640 0x7fd684142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.027+0000 7fe036800640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe01c004e60 0x7fe01c00b9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe03000a880 tx=0x7fe01c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ff181e5d170607 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.031+0000 7f46d1e00640 1 -- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f46d41547f0 msgr2=0x7f46d4200c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.031+0000 7f46d1e00640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f46d41547f0 0x7f46d4200c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.031+0000 7f46d1e00640 1 -- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c5570 msgr2=0x7f46d42006d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.031+0000 7f46d1e00640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c5570 0x7f46d42006d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.031+0000 7f46d1e00640 1 -- 172.21.15.6:0/527246134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d414bca0 con 0x7f46d40c4b90 2024-03-22T12:57:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.045+0000 7fd67be00640 1 -- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd684072770 msgr2=0x7fd6840ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.045+0000 7fd67be00640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd684072770 0x7fd6840ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.045+0000 7fd67be00640 1 -- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 msgr2=0x7fd6841476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.045+0000 7fd67be00640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.045+0000 7fd67be00640 1 -- 172.21.15.6:0/4042696128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6840cc390 con 0x7fd6840cd640 2024-03-22T12:57:21.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.049+0000 7f46d1e00640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46c4000f60 tx=0x7f46c400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.051+0000 7fd67be00640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6840cd640 0x7fd684142cf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c440 tx=0x7fd66c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.051+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03000a480 con 0x7fe01c004e60 2024-03-22T12:57:21.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.051+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe030034650 con 0x7fe01c004e60 2024-03-22T12:57:21.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.051+0000 7fe034a00640 1 -- 172.21.15.6:0/1065306082 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030034ac0 con 0x7fe01c004e60 2024-03-22T12:57:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.052+0000 7fe03cc00640 1 -- 172.21.15.6:0/1065306082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe01c004e60 msgr2=0x7fe01c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.052+0000 7fe03cc00640 1 --2- 172.21.15.6:0/1065306082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe01c004e60 0x7fe01c00b9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe03000a880 tx=0x7fe01c005830 comp rx=0 tx=0).stop 2024-03-22T12:57:21.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.052+0000 7fe036800640 1 -- 172.21.15.6:0/1065306082 reap_dead start 2024-03-22T12:57:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.053+0000 7fe03cc00640 1 -- 172.21.15.6:0/1065306082 shutdown_connections 2024-03-22T12:57:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.053+0000 7fe03cc00640 1 -- 172.21.15.6:0/1065306082 >> 172.21.15.6:0/1065306082 conn(0x7fe0380ccb50 msgr2=0x7fe0380ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.053+0000 7fe03cc00640 1 -- 172.21.15.6:0/1065306082 shutdown_connections 2024-03-22T12:57:21.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.053+0000 7fe03cc00640 1 -- 172.21.15.6:0/1065306082 wait complete. 2024-03-22T12:57:21.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.056+0000 7fe03cc00640 1 Processor -- start 2024-03-22T12:57:21.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.056+0000 7fe03cc00640 1 -- start start 2024-03-22T12:57:21.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.058+0000 7fe03cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe01c004e60 0x7fe03813e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.058+0000 7fe03cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe038159f40 0x7fe03814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.058+0000 7fe03cc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe03814ee30 0x7fe03813c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.058+0000 7fe03cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0381572a0 con 0x7fe038159f40 2024-03-22T12:57:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.058+0000 7fe03cc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe038156fc0 con 0x7fe01c004e60 2024-03-22T12:57:21.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.058+0000 7fe03cc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe038151520 con 0x7fe03814ee30 2024-03-22T12:57:21.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.060+0000 7fe035400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe038159f40 0x7fe03814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.061+0000 7f46ca000640 1 -- 172.21.15.6:0/527246134 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4017070 con 0x7f46d40c4b90 2024-03-22T12:57:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.061+0000 7f46ca000640 1 -- 172.21.15.6:0/527246134 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4005ce0 con 0x7f46d40c4b90 2024-03-22T12:57:21.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.061+0000 7f46ca000640 1 -- 172.21.15.6:0/527246134 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4007310 con 0x7f46d40c4b90 2024-03-22T12:57:21.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.063+0000 7fd67b400640 1 -- 172.21.15.6:0/4042696128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c017070 con 0x7fd6840cd640 2024-03-22T12:57:21.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.063+0000 7fd67b400640 1 -- 172.21.15.6:0/4042696128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c0035f0 con 0x7fd6840cd640 2024-03-22T12:57:21.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.063+0000 7fd67b400640 1 -- 172.21.15.6:0/4042696128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c0039e0 con 0x7fd6840cd640 2024-03-22T12:57:21.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.063+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6841e3e10 con 0x7fd6840cd640 2024-03-22T12:57:21.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.062+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d414c9b0 con 0x7f46d40c4b90 2024-03-22T12:57:21.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.062+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d414cf70 con 0x7f46d40c4b90 2024-03-22T12:57:21.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.063+0000 7fe035400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe038159f40 0x7fe03814e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33992/0 (socket says 172.21.15.6:33992) 2024-03-22T12:57:21.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.063+0000 7fe035400640 1 -- 172.21.15.6:0/3372264796 learned_addr learned my addr 172.21.15.6:0/3372264796 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:21.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.063+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd684148070 con 0x7fd6840cd640 2024-03-22T12:57:21.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.065+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d40ca2a0 con 0x7f46d40c4b90 2024-03-22T12:57:21.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.065+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65c005290 con 0x7fd6840cd640 2024-03-22T12:57:21.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.080+0000 7fe036800640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe03814ee30 0x7fe03813c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.084+0000 7fe036800640 1 -- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe01c004e60 msgr2=0x7fe03813e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.084+0000 7fe036800640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe01c004e60 0x7fe03813e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.084+0000 7fe036800640 1 -- 172.21.15.6:0/3372264796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe038159f40 msgr2=0x7fe03814e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.084+0000 7fe036800640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe038159f40 0x7fe03814e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.084+0000 7fe036800640 1 -- 172.21.15.6:0/3372264796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03814d380 con 0x7fe03814ee30 2024-03-22T12:57:21.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.087+0000 7fe036800640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe03814ee30 0x7fe03813c600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe01c005720 tx=0x7fe030001360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.089+0000 7fe02e000640 1 -- 172.21.15.6:0/3372264796 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03000a460 con 0x7fe03814ee30 2024-03-22T12:57:21.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.089+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0381581d0 con 0x7fe03814ee30 2024-03-22T12:57:21.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.089+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe038158790 con 0x7fe03814ee30 2024-03-22T12:57:21.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.091+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0380757d0 con 0x7fe03814ee30 2024-03-22T12:57:21.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.091+0000 7fe02e000640 1 -- 172.21.15.6:0/3372264796 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe030034650 con 0x7fe03814ee30 2024-03-22T12:57:21.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.092+0000 7fe02e000640 1 -- 172.21.15.6:0/3372264796 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030034800 con 0x7fe03814ee30 2024-03-22T12:57:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.102+0000 7fd67b400640 1 -- 172.21.15.6:0/4042696128 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd66c01a070 con 0x7fd6840cd640 2024-03-22T12:57:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.103+0000 7fd67b400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd66006bc50 0x7fd66006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.103+0000 7fd683400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd66006bc50 0x7fd66006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.104+0000 7fd683400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd66006bc50 0x7fd66006e110 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd674036000 tx=0x7fd67403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.301+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fec64005850 con 0x7fec8c0c7e10 2024-03-22T12:57:21.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.338+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f55a40cf340 con 0x7f55a4071da0 2024-03-22T12:57:21.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.379+0000 7fe02e000640 1 -- 172.21.15.6:0/3372264796 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe030046020 con 0x7fe03814ee30 2024-03-22T12:57:21.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.380+0000 7fe02e000640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe01806bc00 0x7fe01806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.381+0000 7fe035e00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe01806bc00 0x7fe01806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.382+0000 7fe035e00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe01806bc00 0x7fe01806e0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe0280096b0 tx=0x7fe028006ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.383+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f05140c2d20 con 0x7f05140c7fa0 2024-03-22T12:57:21.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.383+0000 7fe02e000640 1 -- 172.21.15.6:0/3372264796 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fe030079830 con 0x7fe03814ee30 2024-03-22T12:57:21.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.395+0000 7fe02e000640 1 -- 172.21.15.6:0/3372264796 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe03007be90 con 0x7fe03814ee30 2024-03-22T12:57:21.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.407+0000 7f050be00640 1 -- 172.21.15.6:0/1705114046 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0500058df0 con 0x7f05140c7fa0 2024-03-22T12:57:21.409 INFO:teuthology.orchestra.run.smithi006.stdout:60129542161 2024-03-22T12:57:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04f406bc20 msgr2=0x7f04f406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04f406bc20 0x7f04f406e0e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0504036000 tx=0x7f050403c040 comp rx=0 tx=0).stop 2024-03-22T12:57:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05140c7fa0 msgr2=0x7f05140cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05140c7fa0 0x7f05140cb650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f050000c440 tx=0x7f050000c900 comp rx=0 tx=0).stop 2024-03-22T12:57:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 shutdown_connections 2024-03-22T12:57:21.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0514150730 0x7f05140cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05140c7fa0 0x7f05140cb650 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f04f406bc20 0x7f04f406e0e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 --2- 172.21.15.6:0/1705114046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05140c6db0 0x7f0514150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 >> 172.21.15.6:0/1705114046 conn(0x7f051406e310 msgr2=0x7f05140c23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 shutdown_connections 2024-03-22T12:57:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.410+0000 7f051aa00640 1 -- 172.21.15.6:0/1705114046 wait complete. 2024-03-22T12:57:21.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542162 got 60129542161 for osd.6 2024-03-22T12:57:21.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.623+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe0380b2b30 con 0x7fe03814ee30 2024-03-22T12:57:21.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.630+0000 7fe02e000640 1 -- 172.21.15.6:0/3372264796 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe03007b790 con 0x7fe03814ee30 2024-03-22T12:57:21.630 INFO:teuthology.orchestra.run.smithi006.stdout:64424509456 2024-03-22T12:57:21.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe01806bc00 msgr2=0x7fe01806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe01806bc00 0x7fe01806e0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe0280096b0 tx=0x7fe028006ca0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe03814ee30 msgr2=0x7fe03813c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe03814ee30 0x7fe03813c600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe01c005720 tx=0x7fe030001360 comp rx=0 tx=0).stop 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 shutdown_connections 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe03814ee30 0x7fe03813c600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe038159f40 0x7fe03814e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe01806bc00 0x7fe01806e0c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 --2- 172.21.15.6:0/3372264796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe01c004e60 0x7fe03813e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 >> 172.21.15.6:0/3372264796 conn(0x7fe0380ccb50 msgr2=0x7fe03814d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 shutdown_connections 2024-03-22T12:57:21.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.632+0000 7fe03cc00640 1 -- 172.21.15.6:0/3372264796 wait complete. 2024-03-22T12:57:21.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509456 for osd.1 2024-03-22T12:57:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.663+0000 7f46ca000640 1 -- 172.21.15.6:0/527246134 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f46c401a070 con 0x7f46d40c4b90 2024-03-22T12:57:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.664+0000 7f46ca000640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f46b406bc50 0x7f46b406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.664+0000 7f46d1400640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f46b406bc50 0x7f46b406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.666+0000 7f46d1400640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f46b406bc50 0x7f46b406e110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f46d4153ce0 tx=0x7f46c003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.678+0000 7fd67b400640 1 -- 172.21.15.6:0/4042696128 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fd66c056810 con 0x7fd6840cd640 2024-03-22T12:57:21.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.678+0000 7fd67b400640 1 -- 172.21.15.6:0/4042696128 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd66c021020 con 0x7fd6840cd640 2024-03-22T12:57:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.714+0000 7fda4cc00640 1 -- 172.21.15.6:0/2588269295 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fda48005d10 con 0x7fda501e5040 2024-03-22T12:57:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.716+0000 7fda4cc00640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda3006bc20 0x7fda3006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.716+0000 7fda54e00640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda3006bc20 0x7fda3006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.717+0000 7fda54e00640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda3006bc20 0x7fda3006e0e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fda3c001eb0 tx=0x7fda3c0016d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.719+0000 7fda4cc00640 1 -- 172.21.15.6:0/2588269295 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fda480585e0 con 0x7fda501e5040 2024-03-22T12:57:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.727+0000 7f46ca000640 1 -- 172.21.15.6:0/527246134 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f46c40566f0 con 0x7f46d40c4b90 2024-03-22T12:57:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.728+0000 7f46ca000640 1 -- 172.21.15.6:0/527246134 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f46d40ca2a0 con 0x7f46d40c4b90 2024-03-22T12:57:21.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.730+0000 7fda4cc00640 1 -- 172.21.15.6:0/2588269295 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fda500ca220 con 0x7fda501e5040 2024-03-22T12:57:21.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.744+0000 7f55a1600640 1 -- 172.21.15.6:0/1649458592 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f559c059c90 con 0x7f55a4071da0 2024-03-22T12:57:21.745 INFO:teuthology.orchestra.run.smithi006.stdout:64424509456 2024-03-22T12:57:21.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f558406be00 msgr2=0x7f558406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f558406be00 0x7f558406e2c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5598036000 tx=0x7f5598039040 comp rx=0 tx=0).stop 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 msgr2=0x7f55a41e9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a41e9ca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c730 tx=0x7f559c00cbf0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 shutdown_connections 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55a40d0e80 0x7f55a40d41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f558406be00 0x7f558406e2c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55a40cf5d0 0x7f55a40d3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 --2- 172.21.15.6:0/1649458592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55a4071da0 0x7f55a41e9ca0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c730 tx=0x7f559c00cbf0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.747+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 >> 172.21.15.6:0/1649458592 conn(0x7f55a406f130 msgr2=0x7f55a4157a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.748+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 shutdown_connections 2024-03-22T12:57:21.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.748+0000 7f55ac000640 1 -- 172.21.15.6:0/1649458592 wait complete. 2024-03-22T12:57:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.756+0000 7f11d6a00640 1 -- 172.21.15.6:0/1384129997 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11cc01b020 con 0x7f11e00cfe60 2024-03-22T12:57:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.758+0000 7f11d6a00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11c006bc20 0x7f11c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:21.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.759+0000 7f11dde00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11c006bc20 0x7f11c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:21.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.760+0000 7f11dde00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11c006bc20 0x7f11c006e0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f11e00d0260 tx=0x7f11d80094f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:21.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.761+0000 7f11d6a00640 1 -- 172.21.15.6:0/1384129997 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f11cc057b80 con 0x7f11e00cfe60 2024-03-22T12:57:21.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509456 for osd.5 2024-03-22T12:57:21.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.777+0000 7f11d6a00640 1 -- 172.21.15.6:0/1384129997 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11e00678d0 con 0x7f11e00cfe60 2024-03-22T12:57:21.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.837+0000 7fec82a00640 1 -- 172.21.15.6:0/3193084578 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec7c0010c0 con 0x7fec8c0c7e10 2024-03-22T12:57:21.841 INFO:teuthology.orchestra.run.smithi006.stdout:60129542161 2024-03-22T12:57:21.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.841+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6c06bc20 msgr2=0x7fec6c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.841+0000 7fec8be00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6c06bc20 0x7fec6c06e0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec84037000 tx=0x7fec8400e040 comp rx=0 tx=0).stop 2024-03-22T12:57:21.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.841+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c0c7e10 msgr2=0x7fec8c159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.841+0000 7fec8be00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c0c7e10 0x7fec8c159e80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00c440 tx=0x7fec7c00c900 comp rx=0 tx=0).stop 2024-03-22T12:57:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 shutdown_connections 2024-03-22T12:57:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec8c156080 0x7fec8c157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec8c150270 0x7fec8c072800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6c06bc20 0x7fec6c06e0e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 --2- 172.21.15.6:0/3193084578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec8c0c7e10 0x7fec8c159e80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 >> 172.21.15.6:0/3193084578 conn(0x7fec8c06f0e0 msgr2=0x7fec8c0c45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 shutdown_connections 2024-03-22T12:57:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.845+0000 7fec8be00640 1 -- 172.21.15.6:0/3193084578 wait complete. 2024-03-22T12:57:21.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542162 got 60129542161 for osd.7 2024-03-22T12:57:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.933+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd65c005850 con 0x7fd6840cd640 2024-03-22T12:57:21.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.956+0000 7fd67b400640 1 -- 172.21.15.6:0/4042696128 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd66c058770 con 0x7fd6840cd640 2024-03-22T12:57:21.957 INFO:teuthology.orchestra.run.smithi006.stdout:64424509455 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd66006bc50 msgr2=0x7fd66006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd66006bc50 0x7fd66006e110 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd674036000 tx=0x7fd67403c040 comp rx=0 tx=0).stop 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6840cd640 msgr2=0x7fd684142cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6840cd640 0x7fd684142cf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c440 tx=0x7fd66c00c840 comp rx=0 tx=0).stop 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 shutdown_connections 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd66006bc50 0x7fd66006e110 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd68414ebd0 0x7fd6841476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6840cd640 0x7fd684142cf0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00c440 tx=0x7fd66c00c840 comp rx=0 tx=0).stop 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 --2- 172.21.15.6:0/4042696128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd684072770 0x7fd6840ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 >> 172.21.15.6:0/4042696128 conn(0x7fd6840ccb50 msgr2=0x7fd6840c8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 shutdown_connections 2024-03-22T12:57:21.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:21.961+0000 7fd68a400640 1 -- 172.21.15.6:0/4042696128 wait complete. 2024-03-22T12:57:21.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509456 got 64424509455 for osd.0 2024-03-22T12:57:22.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.105+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f46d406ddb0 con 0x7f46d40c4b90 2024-03-22T12:57:22.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.157+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fda500757d0 con 0x7fda501e5040 2024-03-22T12:57:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.163+0000 7fda4cc00640 1 -- 172.21.15.6:0/2588269295 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fda48023020 con 0x7fda501e5040 2024-03-22T12:57:22.164 INFO:teuthology.orchestra.run.smithi006.stdout:64424509456 2024-03-22T12:57:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda3006bc20 msgr2=0x7fda3006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda3006bc20 0x7fda3006e0e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fda3c001eb0 tx=0x7fda3c0016d0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda501e5040 msgr2=0x7fda5015e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda501e5040 0x7fda5015e2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda48009c70 tx=0x7fda4800c490 comp rx=0 tx=0).stop 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 shutdown_connections 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda50155e00 0x7fda501562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda3006bc20 0x7fda3006e0e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda501e5040 0x7fda5015e2f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 --2- 172.21.15.6:0/2588269295 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda34005080 0x7fda5015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.166+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 >> 172.21.15.6:0/2588269295 conn(0x7fda500b1700 msgr2=0x7fda500744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.167+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 shutdown_connections 2024-03-22T12:57:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.167+0000 7fda57800640 1 -- 172.21.15.6:0/2588269295 wait complete. 2024-03-22T12:57:22.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.171+0000 7f46ca000640 1 -- 172.21.15.6:0/527246134 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f46d406ddb0 con 0x7f46d40c4b90 2024-03-22T12:57:22.172 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f46b406bc50 msgr2=0x7f46b406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f46b406bc50 0x7f46b406e110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f46d4153ce0 tx=0x7f46c003d040 comp rx=0 tx=0).stop 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46d40c4b90 msgr2=0x7f46d40cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46c4000f60 tx=0x7f46c400c800 comp rx=0 tx=0).stop 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 shutdown_connections 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f46d41547f0 0x7f46d4200c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f46b406bc50 0x7f46b406e110 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46d40c5570 0x7f46d42006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 --2- 172.21.15.6:0/527246134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46d40c4b90 0x7f46d40cc3c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.175+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 >> 172.21.15.6:0/527246134 conn(0x7f46d4066e40 msgr2=0x7f46d40c7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.176+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 shutdown_connections 2024-03-22T12:57:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.176+0000 7f46d9200640 1 -- 172.21.15.6:0/527246134 wait complete. 2024-03-22T12:57:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.188+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f11e00cbca0 con 0x7f11e00cfe60 2024-03-22T12:57:22.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509456 for osd.3 2024-03-22T12:57:22.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509457 for osd.4 2024-03-22T12:57:22.200 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:22.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.224+0000 7f11d6a00640 1 -- 172.21.15.6:0/1384129997 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11cc022020 con 0x7f11e00cfe60 2024-03-22T12:57:22.225 INFO:teuthology.orchestra.run.smithi006.stdout:60129542161 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11c006bc20 msgr2=0x7f11c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11c006bc20 0x7f11c006e0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f11e00d0260 tx=0x7f11d80094f0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00cfe60 msgr2=0x7f11e015e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00cfe60 0x7f11e015e320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00d800 tx=0x7f11cc00dcc0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 shutdown_connections 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11e01587c0 0x7f11e01f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11e00d4050 0x7f11e015ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11c006bc20 0x7f11c006e0e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 --2- 172.21.15.6:0/1384129997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11e00cfe60 0x7f11e015e320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 >> 172.21.15.6:0/1384129997 conn(0x7f11e006f090 msgr2=0x7f11e00cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:22.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 shutdown_connections 2024-03-22T12:57:22.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.228+0000 7f11e4a00640 1 -- 172.21.15.6:0/1384129997 wait complete. 2024-03-22T12:57:22.247 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542161 got 60129542161 for osd.2 2024-03-22T12:57:22.247 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:22.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-22T12:57:22.463 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T12:57:22.453+0000 f5f9640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-22T12:57:22.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.630+0000 7fb467200640 1 Processor -- start 2024-03-22T12:57:22.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.630+0000 7fb467200640 1 -- start start 2024-03-22T12:57:22.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.632+0000 7fb467200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb468154520 0x7fb46815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.632+0000 7fb467200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb46815e320 0x7fb4680c7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.632+0000 7fb467200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb468071db0 con 0x7fb46815a130 2024-03-22T12:57:22.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.632+0000 7fb467200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46815b140 con 0x7fb468154520 2024-03-22T12:57:22.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.632+0000 7fb467200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb468154900 con 0x7fb46815e320 2024-03-22T12:57:22.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.633+0000 7fb466800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb46815e320 0x7fb4680c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.633+0000 7fb465e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb468154520 0x7fb46815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.633+0000 7fb465400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb46815a130 0x7fb4680c66c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55276/0 (socket says 172.21.15.6:55276) 2024-03-22T12:57:22.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.633+0000 7fb465400640 1 -- 172.21.15.6:0/3697384189 learned_addr learned my addr 172.21.15.6:0/3697384189 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:22.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.636+0000 7fb466800640 1 -- 172.21.15.6:0/3697384189 >> v2:172.21.15.6:3300/0 conn(0x7fb468154520 msgr2=0x7fb46815dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:22.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.636+0000 7fb466800640 1 --2- 172.21.15.6:0/3697384189 >> v2:172.21.15.6:3300/0 conn(0x7fb468154520 0x7fb46815dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.636+0000 7fb466800640 1 -- 172.21.15.6:0/3697384189 >> v1:172.21.15.6:6789/0 conn(0x7fb46815a130 legacy=0x7fb4680c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.636+0000 7fb466800640 1 -- 172.21.15.6:0/3697384189 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46815a5c0 con 0x7fb46815e320 2024-03-22T12:57:22.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.638+0000 7fb466800640 1 --2- 172.21.15.6:0/3697384189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb46815e320 0x7fb4680c7420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb460009d90 tx=0x7fb46002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4517366a598f461b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:22.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.642+0000 7fb464a00640 1 -- 172.21.15.6:0/3697384189 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460039070 con 0x7fb46815e320 2024-03-22T12:57:22.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.642+0000 7fb464a00640 1 -- 172.21.15.6:0/3697384189 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46002cc50 con 0x7fb46815e320 2024-03-22T12:57:22.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.642+0000 7fb464a00640 1 -- 172.21.15.6:0/3697384189 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460034440 con 0x7fb46815e320 2024-03-22T12:57:22.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.642+0000 7fb467200640 1 -- 172.21.15.6:0/3697384189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb46815e320 msgr2=0x7fb4680c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.642+0000 7fb467200640 1 --2- 172.21.15.6:0/3697384189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb46815e320 0x7fb4680c7420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb460009d90 tx=0x7fb46002bea0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.643+0000 7fb467200640 1 -- 172.21.15.6:0/3697384189 shutdown_connections 2024-03-22T12:57:22.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.643+0000 7fb467200640 1 --2- 172.21.15.6:0/3697384189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb46815e320 0x7fb4680c7420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.643+0000 7fb467200640 1 --2- 172.21.15.6:0/3697384189 >> v2:172.21.15.6:3300/0 conn(0x7fb468154520 0x7fb46815dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.643+0000 7fb467200640 1 -- 172.21.15.6:0/3697384189 >> 172.21.15.6:0/3697384189 conn(0x7fb46806f0e0 msgr2=0x7fb46806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:22.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.643+0000 7fb467200640 1 -- 172.21.15.6:0/3697384189 shutdown_connections 2024-03-22T12:57:22.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.643+0000 7fb467200640 1 -- 172.21.15.6:0/3697384189 wait complete. 2024-03-22T12:57:22.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.645+0000 7fb467200640 1 Processor -- start 2024-03-22T12:57:22.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.645+0000 7fb467200640 1 -- start start 2024-03-22T12:57:22.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.646+0000 7fb467200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb468154520 0x7fb4680c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.646+0000 7fb467200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb46815e320 0x7fb4681e37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.646+0000 7fb467200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4680c5bb0 0x7fb4681f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.646+0000 7fb467200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46815ae60 con 0x7fb46815e320 2024-03-22T12:57:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.646+0000 7fb467200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46815ab80 con 0x7fb468154520 2024-03-22T12:57:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.646+0000 7fb467200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46815d8c0 con 0x7fb4680c5bb0 2024-03-22T12:57:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.646+0000 7fb465400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb46815e320 0x7fb4681e37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.647+0000 7fb466800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4680c5bb0 0x7fb4681f08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.647+0000 7fb465e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb468154520 0x7fb4680c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.647+0000 7fb465400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb46815e320 0x7fb4681e37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34020/0 (socket says 172.21.15.6:34020) 2024-03-22T12:57:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.647+0000 7fb465400640 1 -- 172.21.15.6:0/3222702854 learned_addr learned my addr 172.21.15.6:0/3222702854 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:22.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.650+0000 7fb466800640 1 -- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb468154520 msgr2=0x7fb4680c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.650+0000 7fb466800640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb468154520 0x7fb4680c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.650+0000 7fb466800640 1 -- 172.21.15.6:0/3222702854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb46815e320 msgr2=0x7fb4681e37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.650+0000 7fb466800640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb46815e320 0x7fb4681e37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.650+0000 7fb466800640 1 -- 172.21.15.6:0/3222702854 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4681f0400 con 0x7fb4680c5bb0 2024-03-22T12:57:22.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-22T12:57:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.653+0000 7fb466800640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4680c5bb0 0x7fb4681f08e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb460033000 tx=0x7fb460004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.657+0000 7fb45e000640 1 -- 172.21.15.6:0/3222702854 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460039050 con 0x7fb4680c5bb0 2024-03-22T12:57:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.657+0000 7fb45e000640 1 -- 172.21.15.6:0/3222702854 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb460045070 con 0x7fb4680c5bb0 2024-03-22T12:57:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.657+0000 7fb45e000640 1 -- 172.21.15.6:0/3222702854 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460009450 con 0x7fb4680c5bb0 2024-03-22T12:57:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.657+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46815cd40 con 0x7fb4680c5bb0 2024-03-22T12:57:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.657+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46815d300 con 0x7fb4680c5bb0 2024-03-22T12:57:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.658+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4681f05d0 con 0x7fb4680c5bb0 2024-03-22T12:57:22.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-22T12:57:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.824+0000 7f5369800640 1 Processor -- start 2024-03-22T12:57:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.824+0000 7f5369800640 1 -- start start 2024-03-22T12:57:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.825+0000 7f5369800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f53640cc400 0x7f53640cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.825+0000 7f5369800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536415a670 con 0x7f53640cf0e0 2024-03-22T12:57:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.825+0000 7f5369800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536415a930 con 0x7f53640cd7f0 2024-03-22T12:57:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.825+0000 7f5369800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536406dd30 con 0x7f53640cc400 2024-03-22T12:57:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.826+0000 7f5368e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f53640cf0e0 0x7f5364164c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55282/0 (socket says 172.21.15.6:55282) 2024-03-22T12:57:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.826+0000 7f5368e00640 1 -- 172.21.15.6:0/396183681 learned_addr learned my addr 172.21.15.6:0/396183681 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.826+0000 7f5363e00640 1 --2- 172.21.15.6:0/396183681 >> v2:172.21.15.92:3300/0 conn(0x7f53640cc400 0x7f53640cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.829+0000 7f5362a00640 1 -- 172.21.15.6:0/396183681 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193786831 0 0) 0x7f536415a670 con 0x7f53640cf0e0 2024-03-22T12:57:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.829+0000 7f5362a00640 1 -- 172.21.15.6:0/396183681 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5348003520 con 0x7f53640cf0e0 2024-03-22T12:57:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.830+0000 7f5363e00640 1 -- 172.21.15.6:0/396183681 >> v1:172.21.15.92:6789/0 conn(0x7f53640cd7f0 legacy=0x7f53640ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.830+0000 7f5363e00640 1 -- 172.21.15.6:0/396183681 >> v1:172.21.15.6:6789/0 conn(0x7f53640cf0e0 legacy=0x7f5364164c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.830+0000 7f5363e00640 1 -- 172.21.15.6:0/396183681 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536415ac10 con 0x7f53640cc400 2024-03-22T12:57:22.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.837+0000 7f5363e00640 1 --2- 172.21.15.6:0/396183681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cc400 0x7f53640cc7e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f53540097a0 tx=0x7f535402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a0b33d482c4c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.843+0000 7f5362a00640 1 -- 172.21.15.6:0/396183681 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53540315a0 con 0x7f53640cc400 2024-03-22T12:57:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.843+0000 7f5362a00640 1 -- 172.21.15.6:0/396183681 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5354037070 con 0x7f53640cc400 2024-03-22T12:57:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.843+0000 7f5362a00640 1 -- 172.21.15.6:0/396183681 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354031960 con 0x7f53640cc400 2024-03-22T12:57:22.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.843+0000 7f5369800640 1 -- 172.21.15.6:0/396183681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cc400 msgr2=0x7f53640cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.843+0000 7f5369800640 1 --2- 172.21.15.6:0/396183681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cc400 0x7f53640cc7e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f53540097a0 tx=0x7f535402b840 comp rx=0 tx=0).stop 2024-03-22T12:57:22.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.844+0000 7f5369800640 1 -- 172.21.15.6:0/396183681 shutdown_connections 2024-03-22T12:57:22.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.844+0000 7f5369800640 1 --2- 172.21.15.6:0/396183681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cc400 0x7f53640cc7e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.844+0000 7f5369800640 1 -- 172.21.15.6:0/396183681 >> 172.21.15.6:0/396183681 conn(0x7f53640cf9f0 msgr2=0x7f53640cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.844+0000 7f5369800640 1 -- 172.21.15.6:0/396183681 shutdown_connections 2024-03-22T12:57:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.844+0000 7f5369800640 1 -- 172.21.15.6:0/396183681 wait complete. 2024-03-22T12:57:22.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.846+0000 7f5369800640 1 Processor -- start 2024-03-22T12:57:22.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.847+0000 7f5369800640 1 -- start start 2024-03-22T12:57:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.848+0000 7f5369800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640cc400 0x7f5364159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.848+0000 7f5369800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53640cd7f0 0x7f536415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.848+0000 7f5369800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cf0e0 0x7f536415ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.848+0000 7f5369800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53640ce1d0 con 0x7f53640cc400 2024-03-22T12:57:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.848+0000 7f5369800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536415b1d0 con 0x7f53640cf0e0 2024-03-22T12:57:22.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.848+0000 7f5369800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53640ce4b0 con 0x7f53640cd7f0 2024-03-22T12:57:22.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.849+0000 7f5368e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cf0e0 0x7f536415ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.849+0000 7f5363e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640cc400 0x7f5364159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.849+0000 7f5363e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640cc400 0x7f5364159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34036/0 (socket says 172.21.15.6:34036) 2024-03-22T12:57:22.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.849+0000 7f5363e00640 1 -- 172.21.15.6:0/2522893742 learned_addr learned my addr 172.21.15.6:0/2522893742 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:22.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.853+0000 7f5363e00640 1 -- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53640cd7f0 msgr2=0x7f536415b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.853+0000 7f5363e00640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53640cd7f0 0x7f536415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.853+0000 7f5363e00640 1 -- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cf0e0 msgr2=0x7f536415ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.853+0000 7f5363e00640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cf0e0 0x7f536415ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.853+0000 7f5363e00640 1 -- 172.21.15.6:0/2522893742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53640c9470 con 0x7f53640cc400 2024-03-22T12:57:22.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.856+0000 7f5363e00640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640cc400 0x7f5364159940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f53540096f0 tx=0x7f53540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.861+0000 7f535be00640 1 -- 172.21.15.6:0/2522893742 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354032840 con 0x7f53640cc400 2024-03-22T12:57:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.862+0000 7f535be00640 1 -- 172.21.15.6:0/2522893742 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5354032a00 con 0x7f53640cc400 2024-03-22T12:57:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.862+0000 7f535be00640 1 -- 172.21.15.6:0/2522893742 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354032df0 con 0x7f53640cc400 2024-03-22T12:57:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.862+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53641f1000 con 0x7f53640cc400 2024-03-22T12:57:22.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.862+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536415cca0 con 0x7f53640cc400 2024-03-22T12:57:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.863+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5364075650 con 0x7f53640cc400 2024-03-22T12:57:22.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-22T12:57:22.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.942+0000 7fb45e000640 1 -- 172.21.15.6:0/3222702854 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb46003e610 con 0x7fb4680c5bb0 2024-03-22T12:57:22.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.944+0000 7fb45e000640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb44806bc50 0x7fb44806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.944+0000 7fb465e00640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb44806bc50 0x7fb44806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.945+0000 7fb465e00640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb44806bc50 0x7fb44806e110 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb458009b40 tx=0x7fb458009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:22.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.947+0000 7fb45e000640 1 -- 172.21.15.6:0/3222702854 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fb46007a330 con 0x7fb4680c5bb0 2024-03-22T12:57:22.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.958+0000 7f69cf000640 1 Processor -- start 2024-03-22T12:57:22.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.958+0000 7f69cf000640 1 -- start start 2024-03-22T12:57:22.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.960+0000 7f69cf000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f69c80cc400 0x7f69c80cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.960+0000 7f69cf000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cd7f0 0x7f69c80ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.960+0000 7f69cf000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c815a670 con 0x7f69c80cdc30 2024-03-22T12:57:22.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.960+0000 7f69cf000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c806dd30 con 0x7f69c80cc400 2024-03-22T12:57:22.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.960+0000 7f69cf000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c815a930 con 0x7f69c80cd7f0 2024-03-22T12:57:22.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.964+0000 7fb45e000640 1 -- 172.21.15.6:0/3222702854 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb46007c290 con 0x7fb4680c5bb0 2024-03-22T12:57:22.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.969+0000 7f69cd200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cd7f0 0x7f69c80ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.969+0000 7f69cd200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cd7f0 0x7f69c80ce9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46024/0 (socket says 172.21.15.6:46024) 2024-03-22T12:57:22.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.969+0000 7f69cd200640 1 -- 172.21.15.6:0/895804182 learned_addr learned my addr 172.21.15.6:0/895804182 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:22.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.971+0000 7f69cd200640 1 -- 172.21.15.6:0/895804182 >> v2:172.21.15.6:3300/0 conn(0x7f69c80cc400 msgr2=0x7f69c80cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:22.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.971+0000 7f69cd200640 1 --2- 172.21.15.6:0/895804182 >> v2:172.21.15.6:3300/0 conn(0x7f69c80cc400 0x7f69c80cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.971+0000 7f69cd200640 1 -- 172.21.15.6:0/895804182 >> v1:172.21.15.6:6789/0 conn(0x7f69c80cdc30 legacy=0x7f69c815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:22.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.971+0000 7f69cd200640 1 -- 172.21.15.6:0/895804182 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c815ac10 con 0x7f69c80cd7f0 2024-03-22T12:57:22.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.974+0000 7f69cd200640 1 --2- 172.21.15.6:0/895804182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cd7f0 0x7f69c80ce9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009870 tx=0x7f69b402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e5e0c4f277d00c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:22.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.979+0000 7f69c7e00640 1 -- 172.21.15.6:0/895804182 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b40317c0 con 0x7f69c80cd7f0 2024-03-22T12:57:22.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.979+0000 7f69c7e00640 1 -- 172.21.15.6:0/895804182 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b4037070 con 0x7f69c80cd7f0 2024-03-22T12:57:22.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.979+0000 7f69c7e00640 1 -- 172.21.15.6:0/895804182 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4031b80 con 0x7f69c80cd7f0 2024-03-22T12:57:22.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 -- 172.21.15.6:0/895804182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cd7f0 msgr2=0x7f69c80ce9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 --2- 172.21.15.6:0/895804182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cd7f0 0x7f69c80ce9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009870 tx=0x7f69b402b860 comp rx=0 tx=0).stop 2024-03-22T12:57:22.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 -- 172.21.15.6:0/895804182 shutdown_connections 2024-03-22T12:57:22.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 --2- 172.21.15.6:0/895804182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cd7f0 0x7f69c80ce9d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 --2- 172.21.15.6:0/895804182 >> v2:172.21.15.6:3300/0 conn(0x7f69c80cc400 0x7f69c80cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 -- 172.21.15.6:0/895804182 >> 172.21.15.6:0/895804182 conn(0x7f69c80cf9f0 msgr2=0x7f69c80cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:22.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 -- 172.21.15.6:0/895804182 shutdown_connections 2024-03-22T12:57:22.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.980+0000 7f69cf000640 1 -- 172.21.15.6:0/895804182 wait complete. 2024-03-22T12:57:22.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.983+0000 7f69cf000640 1 Processor -- start 2024-03-22T12:57:22.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.983+0000 7f69cf000640 1 -- start start 2024-03-22T12:57:22.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.985+0000 7f69cf000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cc400 0x7f69c81482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.985+0000 7f69cf000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69c80cd7f0 0x7f69c81e4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.985+0000 7f69cf000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69c80cdc30 0x7f69c81e5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:22.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.985+0000 7f69cf000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c80ce1d0 con 0x7f69c80cdc30 2024-03-22T12:57:22.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.985+0000 7f69cf000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c815b1d0 con 0x7f69c80cd7f0 2024-03-22T12:57:22.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.985+0000 7f69cf000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c80ce4b0 con 0x7f69c80cc400 2024-03-22T12:57:22.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-22T12:57:22.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.985+0000 7f69cdc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cc400 0x7f69c81482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.986+0000 7f69cdc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cc400 0x7f69c81482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46040/0 (socket says 172.21.15.6:46040) 2024-03-22T12:57:22.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.986+0000 7f69cd200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69c80cd7f0 0x7f69c81e4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:22.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.986+0000 7f69cdc00640 1 -- 172.21.15.6:0/541401335 learned_addr learned my addr 172.21.15.6:0/541401335 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:22.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.990+0000 7f69cdc00640 1 -- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69c80cd7f0 msgr2=0x7f69c81e4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.990+0000 7f69cdc00640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69c80cd7f0 0x7f69c81e4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.990+0000 7f69cdc00640 1 -- 172.21.15.6:0/541401335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69c80cdc30 msgr2=0x7f69c81e5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:22.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.990+0000 7f69cdc00640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69c80cdc30 0x7f69c81e5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:22.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.990+0000 7f69cdc00640 1 -- 172.21.15.6:0/541401335 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c81523d0 con 0x7f69c80cc400 2024-03-22T12:57:22.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.992+0000 7f69cdc00640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cc400 0x7f69c81482d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00b4d0 tx=0x7f69bc00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:22.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.996+0000 7f69c5600640 1 -- 172.21.15.6:0/541401335 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc014070 con 0x7f69c80cc400 2024-03-22T12:57:22.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.996+0000 7f69c5600640 1 -- 172.21.15.6:0/541401335 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc010870 con 0x7f69c80cc400 2024-03-22T12:57:22.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.996+0000 7f69c5600640 1 -- 172.21.15.6:0/541401335 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc010c60 con 0x7f69c80cc400 2024-03-22T12:57:22.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.996+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c814fa10 con 0x7f69c80cc400 2024-03-22T12:57:22.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.997+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c80b3740 con 0x7f69c80cc400 2024-03-22T12:57:22.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:22.998+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c8075650 con 0x7f69c80cc400 2024-03-22T12:57:23.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.018+0000 7f69c5600640 1 -- 172.21.15.6:0/541401335 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f69bc017070 con 0x7f69c80cc400 2024-03-22T12:57:23.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.019+0000 7f69c5600640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f69a806bc20 0x7f69a806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.020+0000 7f69cd200640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f69a806bc20 0x7f69a806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.021+0000 7f69cd200640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f69a806bc20 0x7f69a806e0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f69b4036000 tx=0x7f69b403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.055+0000 7f8bdd200640 1 Processor -- start 2024-03-22T12:57:23.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.055+0000 7f8bdd200640 1 -- start start 2024-03-22T12:57:23.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.056+0000 7f8bdd200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8bd80cc7e0 0x7f8bd80ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.056+0000 7f8bdd200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd8164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.056+0000 7f8bdd200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd815a670 con 0x7f8bd80cc400 2024-03-22T12:57:23.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.056+0000 7f8bdd200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd806dd30 con 0x7f8bd80cc7e0 2024-03-22T12:57:23.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.056+0000 7f8bdd200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80cf350 con 0x7f8bd80cef10 2024-03-22T12:57:23.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.091+0000 7f535be00640 1 -- 172.21.15.6:0/2522893742 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f535403b3f0 con 0x7f53640cc400 2024-03-22T12:57:23.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.092+0000 7f535be00640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f534406bc20 0x7f534406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.093+0000 7f5363400640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f534406bc20 0x7f534406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.094+0000 7f5363400640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f534406bc20 0x7f534406e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f536415c8d0 tx=0x7f5350007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.191+0000 7fabeb200640 1 Processor -- start 2024-03-22T12:57:23.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.191+0000 7fabeb200640 1 -- start start 2024-03-22T12:57:23.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.192+0000 7fabeb200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fabec0cc400 0x7fabec0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.192+0000 7fabeb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabec15a670 con 0x7fabec0cd7f0 2024-03-22T12:57:23.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.192+0000 7fabeb200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabec15a930 con 0x7fabec0cf0e0 2024-03-22T12:57:23.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.192+0000 7fabeb200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabec06dd30 con 0x7fabec0cc400 2024-03-22T12:57:23.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-22T12:57:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.193+0000 7fabe9e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fabec0cc400 0x7fabec0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.193+0000 7fabe9e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fabec0cc400 0x7fabec0cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50234/0 (socket says 172.21.15.6:50234) 2024-03-22T12:57:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.193+0000 7fabe9e00640 1 -- 172.21.15.6:0/2919886675 learned_addr learned my addr 172.21.15.6:0/2919886675 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.197+0000 7fabe9e00640 1 -- 172.21.15.6:0/2919886675 >> v1:172.21.15.92:6789/0 conn(0x7fabec0cf0e0 legacy=0x7fabec164c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.197+0000 7fabe9e00640 1 -- 172.21.15.6:0/2919886675 >> v1:172.21.15.6:6789/0 conn(0x7fabec0cd7f0 legacy=0x7fabec0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.197+0000 7fabe9e00640 1 -- 172.21.15.6:0/2919886675 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec15ac10 con 0x7fabec0cc400 2024-03-22T12:57:23.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.201+0000 7fabe9e00640 1 --2- 172.21.15.6:0/2919886675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cc400 0x7fabec0cc7e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009870 tx=0x7fabdc02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb3d143241012fe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.207+0000 7fabe8a00640 1 -- 172.21.15.6:0/2919886675 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc02e070 con 0x7fabec0cc400 2024-03-22T12:57:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.207+0000 7fabe8a00640 1 -- 172.21.15.6:0/2919886675 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabdc037070 con 0x7fabec0cc400 2024-03-22T12:57:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.207+0000 7fabe8a00640 1 -- 172.21.15.6:0/2919886675 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc031a00 con 0x7fabec0cc400 2024-03-22T12:57:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.208+0000 7fabeb200640 1 -- 172.21.15.6:0/2919886675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cc400 msgr2=0x7fabec0cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.208+0000 7fabeb200640 1 --2- 172.21.15.6:0/2919886675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cc400 0x7fabec0cc7e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009870 tx=0x7fabdc02b880 comp rx=0 tx=0).stop 2024-03-22T12:57:23.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.208+0000 7fabeb200640 1 -- 172.21.15.6:0/2919886675 shutdown_connections 2024-03-22T12:57:23.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.208+0000 7fabeb200640 1 --2- 172.21.15.6:0/2919886675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cc400 0x7fabec0cc7e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.208+0000 7fabeb200640 1 -- 172.21.15.6:0/2919886675 >> 172.21.15.6:0/2919886675 conn(0x7fabec0cf9f0 msgr2=0x7fabec0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:23.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.208+0000 7fabeb200640 1 -- 172.21.15.6:0/2919886675 shutdown_connections 2024-03-22T12:57:23.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.208+0000 7fabeb200640 1 -- 172.21.15.6:0/2919886675 wait complete. 2024-03-22T12:57:23.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.210+0000 7fabeb200640 1 Processor -- start 2024-03-22T12:57:23.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.210+0000 7fabeb200640 1 -- start start 2024-03-22T12:57:23.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.211+0000 7fabeb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec0cc400 0x7fabec1595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.211+0000 7fabeb200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabec0cd7f0 0x7fabec0b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.211+0000 7fabeb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cf0e0 0x7fabec0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.211+0000 7fabeb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0ce1d0 con 0x7fabec0cc400 2024-03-22T12:57:23.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.211+0000 7fabeb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec15b1d0 con 0x7fabec0cf0e0 2024-03-22T12:57:23.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.211+0000 7fabeb200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0ce4b0 con 0x7fabec0cd7f0 2024-03-22T12:57:23.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.212+0000 7fabea800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cf0e0 0x7fabec0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.213+0000 7fabea800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cf0e0 0x7fabec0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50250/0 (socket says 172.21.15.6:50250) 2024-03-22T12:57:23.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.213+0000 7fabea800640 1 -- 172.21.15.6:0/698251369 learned_addr learned my addr 172.21.15.6:0/698251369 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:23.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.218+0000 7fabea800640 1 -- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabec0cd7f0 msgr2=0x7fabec0b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.218+0000 7fabea800640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabec0cd7f0 0x7fabec0b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.218+0000 7fabea800640 1 -- 172.21.15.6:0/698251369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec0cc400 msgr2=0x7fabec1595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.218+0000 7fabea800640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec0cc400 0x7fabec1595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.218+0000 7fabea800640 1 -- 172.21.15.6:0/698251369 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec0b2a30 con 0x7fabec0cf0e0 2024-03-22T12:57:23.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.222+0000 7fabea800640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cf0e0 0x7fabec0b4c20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fabe400b990 tx=0x7fabe400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.227+0000 7fabe2000640 1 -- 172.21.15.6:0/698251369 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4019070 con 0x7fabec0cf0e0 2024-03-22T12:57:23.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.227+0000 7fabe2000640 1 -- 172.21.15.6:0/698251369 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe400cba0 con 0x7fabec0cf0e0 2024-03-22T12:57:23.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.227+0000 7fabe2000640 1 -- 172.21.15.6:0/698251369 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4014440 con 0x7fabec0cf0e0 2024-03-22T12:57:23.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.228+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabec0b3570 con 0x7fabec0cf0e0 2024-03-22T12:57:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.228+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabec0b2fb0 con 0x7fabec0cf0e0 2024-03-22T12:57:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.230+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabec075650 con 0x7fabec0cf0e0 2024-03-22T12:57:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.262+0000 7f69c5600640 1 -- 172.21.15.6:0/541401335 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f69bc057140 con 0x7f69c80cc400 2024-03-22T12:57:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.267+0000 7fabe2000640 1 -- 172.21.15.6:0/698251369 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fabe401c070 con 0x7fabec0cf0e0 2024-03-22T12:57:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.268+0000 7fabe2000640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabcc06bc20 0x7fabcc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.269+0000 7fabe9e00640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabcc06bc20 0x7fabcc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.270+0000 7fabe9e00640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabcc06bc20 0x7fabcc06e0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fabdc036000 tx=0x7fabdc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.271+0000 7fabe2000640 1 -- 172.21.15.6:0/698251369 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fabe4057430 con 0x7fabec0cf0e0 2024-03-22T12:57:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.275+0000 7f69c5600640 1 -- 172.21.15.6:0/541401335 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f69bc0590a0 con 0x7f69c80cc400 2024-03-22T12:57:23.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.277+0000 7f8bd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd8164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.277+0000 7f8bd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd8164c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46054/0 (socket says 172.21.15.6:46054) 2024-03-22T12:57:23.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.277+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1757025089 learned_addr learned my addr 172.21.15.6:0/1757025089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:23.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.282+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1757025089 >> v2:172.21.15.6:3300/0 conn(0x7f8bd80cc7e0 msgr2=0x7f8bd80ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:23.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.282+0000 7f8bd7e00640 1 --2- 172.21.15.6:0/1757025089 >> v2:172.21.15.6:3300/0 conn(0x7f8bd80cc7e0 0x7f8bd80ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.282+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1757025089 >> v1:172.21.15.6:6789/0 conn(0x7f8bd80cc400 legacy=0x7f8bd80cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:23.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.282+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1757025089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd815a930 con 0x7f8bd80cef10 2024-03-22T12:57:23.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.285+0000 7f8bd7e00640 1 --2- 172.21.15.6:0/1757025089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd8164c00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009e50 tx=0x7f8bd002be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2982650bb9af59 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.291+0000 7f8bd6000640 1 -- 172.21.15.6:0/1757025089 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd002ca10 con 0x7f8bd80cef10 2024-03-22T12:57:23.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.291+0000 7f8bd6000640 1 -- 172.21.15.6:0/1757025089 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd002cbb0 con 0x7f8bd80cef10 2024-03-22T12:57:23.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.291+0000 7f8bd6000640 1 -- 172.21.15.6:0/1757025089 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd00343f0 con 0x7f8bd80cef10 2024-03-22T12:57:23.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.292+0000 7f8bdd200640 1 -- 172.21.15.6:0/1757025089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 msgr2=0x7f8bd8164c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.292+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1757025089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd8164c00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009e50 tx=0x7f8bd002be70 comp rx=0 tx=0).stop 2024-03-22T12:57:23.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.292+0000 7f8bdd200640 1 -- 172.21.15.6:0/1757025089 shutdown_connections 2024-03-22T12:57:23.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.292+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1757025089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd8164c00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.292+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1757025089 >> v2:172.21.15.6:3300/0 conn(0x7f8bd80cc7e0 0x7f8bd80ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.293+0000 7f8bdd200640 1 -- 172.21.15.6:0/1757025089 >> 172.21.15.6:0/1757025089 conn(0x7f8bd80cf9f0 msgr2=0x7f8bd80cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:23.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.293+0000 7f8bdd200640 1 -- 172.21.15.6:0/1757025089 shutdown_connections 2024-03-22T12:57:23.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.293+0000 7f8bdd200640 1 -- 172.21.15.6:0/1757025089 wait complete. 2024-03-22T12:57:23.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.296+0000 7f8bdd200640 1 Processor -- start 2024-03-22T12:57:23.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.296+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb4681f01f0 con 0x7fb4680c5bb0 2024-03-22T12:57:23.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.297+0000 7f8bdd200640 1 -- start start 2024-03-22T12:57:23.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.299+0000 7f8bdd200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bd80cc400 0x7f8bd80b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.299+0000 7f8bdd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bd80cc7e0 0x7f8bd81e40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.299+0000 7f8bdd200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd81f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.299+0000 7f8bdd200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd815b1d0 con 0x7f8bd80cc7e0 2024-03-22T12:57:23.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.299+0000 7f8bdd200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd815aef0 con 0x7f8bd80cc400 2024-03-22T12:57:23.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.299+0000 7f8bdd200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80ce1d0 con 0x7f8bd80cef10 2024-03-22T12:57:23.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.304+0000 7f8bd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd81f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.305+0000 7fb45e000640 1 -- 172.21.15.6:0/3222702854 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb460080330 con 0x7fb4680c5bb0 2024-03-22T12:57:23.305 INFO:teuthology.orchestra.run.smithi006.stdout:60129542162 2024-03-22T12:57:23.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.306+0000 7f8bd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd81f2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46062/0 (socket says 172.21.15.6:46062) 2024-03-22T12:57:23.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.306+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1981158270 learned_addr learned my addr 172.21.15.6:0/1981158270 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb44806bc50 msgr2=0x7fb44806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb44806bc50 0x7fb44806e110 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb458009b40 tx=0x7fb458009190 comp rx=0 tx=0).stop 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4680c5bb0 msgr2=0x7fb4681f08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4680c5bb0 0x7fb4681f08e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb460033000 tx=0x7fb460004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 shutdown_connections 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4680c5bb0 0x7fb4681f08e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb46815e320 0x7fb4681e37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb44806bc50 0x7fb44806e110 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 --2- 172.21.15.6:0/3222702854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb468154520 0x7fb4680c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 >> 172.21.15.6:0/3222702854 conn(0x7fb46806f0e0 msgr2=0x7fb4681471c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:23.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 shutdown_connections 2024-03-22T12:57:23.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.308+0000 7fb467200640 1 -- 172.21.15.6:0/3222702854 wait complete. 2024-03-22T12:57:23.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.313+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bd80cc400 msgr2=0x7f8bd80b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.313+0000 7f8bd7e00640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bd80cc400 0x7f8bd80b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.313+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1981158270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bd80cc7e0 msgr2=0x7f8bd81e40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.313+0000 7f8bd7e00640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bd80cc7e0 0x7f8bd81e40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.314+0000 7f8bd7e00640 1 -- 172.21.15.6:0/1981158270 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd81f19a0 con 0x7f8bd80cef10 2024-03-22T12:57:23.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.319+0000 7f8bd7e00640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd81f2720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009f70 tx=0x7f8bd0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.325+0000 7f8bcf400640 1 -- 172.21.15.6:0/1981158270 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0043020 con 0x7f8bd80cef10 2024-03-22T12:57:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.325+0000 7f8bcf400640 1 -- 172.21.15.6:0/1981158270 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd003e450 con 0x7f8bd80cef10 2024-03-22T12:57:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.325+0000 7f8bcf400640 1 -- 172.21.15.6:0/1981158270 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd003e840 con 0x7f8bd80cef10 2024-03-22T12:57:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.326+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd81e61e0 con 0x7f8bd80cef10 2024-03-22T12:57:23.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.326+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd81f1f20 con 0x7f8bd80cef10 2024-03-22T12:57:23.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.328+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd8075650 con 0x7f8bd80cef10 2024-03-22T12:57:23.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542162 got 60129542162 for osd.6 2024-03-22T12:57:23.335 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.342+0000 7f535be00640 1 -- 172.21.15.6:0/2522893742 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f535402f030 con 0x7f53640cc400 2024-03-22T12:57:23.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.374+0000 7f535be00640 1 -- 172.21.15.6:0/2522893742 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53640c9470 con 0x7f53640cc400 2024-03-22T12:57:23.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.393+0000 7f034ba00640 1 Processor -- start 2024-03-22T12:57:23.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.393+0000 7f034ba00640 1 -- start start 2024-03-22T12:57:23.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.395+0000 7f034ba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03441547f0 0x7f03440c6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.395+0000 7f034ba00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f03440c5570 0x7f03440ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.395+0000 7f034ba00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03440c5270 con 0x7f03440c4b90 2024-03-22T12:57:23.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.395+0000 7f034ba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03440c5020 con 0x7f03441547f0 2024-03-22T12:57:23.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.395+0000 7f034ba00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034415e380 con 0x7f03440c5570 2024-03-22T12:57:23.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.396+0000 7f0349000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03441547f0 0x7f03440c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.397+0000 7f0349000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03441547f0 0x7f03440c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41302/0 (socket says 172.21.15.6:41302) 2024-03-22T12:57:23.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.397+0000 7f0349000640 1 -- 172.21.15.6:0/2264681115 learned_addr learned my addr 172.21.15.6:0/2264681115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:23.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.402+0000 7f0349000640 1 -- 172.21.15.6:0/2264681115 >> v2:172.21.15.92:3300/0 conn(0x7f03440c5570 msgr2=0x7f03440ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:23.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.402+0000 7f0349000640 1 --2- 172.21.15.6:0/2264681115 >> v2:172.21.15.92:3300/0 conn(0x7f03440c5570 0x7f03440ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.402+0000 7f0349000640 1 -- 172.21.15.6:0/2264681115 >> v1:172.21.15.6:6789/0 conn(0x7f03440c4b90 legacy=0x7f03440cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.402+0000 7f0349000640 1 -- 172.21.15.6:0/2264681115 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034415e660 con 0x7f03441547f0 2024-03-22T12:57:23.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.407+0000 7f0349000640 1 --2- 172.21.15.6:0/2264681115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03441547f0 0x7f03440c6450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f034408d860 tx=0x7f033802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c496a0f1e7faa2f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.413+0000 7f0343400640 1 -- 172.21.15.6:0/2264681115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338035020 con 0x7f03441547f0 2024-03-22T12:57:23.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.413+0000 7f0343400640 1 -- 172.21.15.6:0/2264681115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0338031890 con 0x7f03441547f0 2024-03-22T12:57:23.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.413+0000 7f0343400640 1 -- 172.21.15.6:0/2264681115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338031ca0 con 0x7f03441547f0 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 -- 172.21.15.6:0/2264681115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03441547f0 msgr2=0x7f03440c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 --2- 172.21.15.6:0/2264681115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03441547f0 0x7f03440c6450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f034408d860 tx=0x7f033802b930 comp rx=0 tx=0).stop 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 -- 172.21.15.6:0/2264681115 shutdown_connections 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 --2- 172.21.15.6:0/2264681115 >> v2:172.21.15.92:3300/0 conn(0x7f03440c5570 0x7f03440ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 --2- 172.21.15.6:0/2264681115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03441547f0 0x7f03440c6450 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 -- 172.21.15.6:0/2264681115 >> 172.21.15.6:0/2264681115 conn(0x7f0344066e40 msgr2=0x7f0344067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 -- 172.21.15.6:0/2264681115 shutdown_connections 2024-03-22T12:57:23.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.414+0000 7f034ba00640 1 -- 172.21.15.6:0/2264681115 wait complete. 2024-03-22T12:57:23.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.417+0000 7f034ba00640 1 Processor -- start 2024-03-22T12:57:23.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.417+0000 7f034ba00640 1 -- start start 2024-03-22T12:57:23.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.419+0000 7f034ba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03440c4b90 0x7f03440c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.419+0000 7f034ba00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03440c5570 0x7f0344155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.419+0000 7f034ba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03441547f0 0x7f034415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.419+0000 7f034ba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03440cd270 con 0x7f03440c4b90 2024-03-22T12:57:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.419+0000 7f034ba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03440c5f30 con 0x7f03441547f0 2024-03-22T12:57:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.419+0000 7f034ba00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03440c6990 con 0x7f03440c5570 2024-03-22T12:57:23.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.420+0000 7f0349000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03440c4b90 0x7f03440c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.420+0000 7f0349000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03440c4b90 0x7f03440c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41310/0 (socket says 172.21.15.6:41310) 2024-03-22T12:57:23.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.420+0000 7f0349000640 1 -- 172.21.15.6:0/2719900883 learned_addr learned my addr 172.21.15.6:0/2719900883 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:23.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.426+0000 7f0349000640 1 -- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03440c5570 msgr2=0x7f0344155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.426+0000 7f0349000640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03440c5570 0x7f0344155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.426+0000 7f0349000640 1 -- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03441547f0 msgr2=0x7f034415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.426+0000 7f0349000640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03441547f0 0x7f034415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.426+0000 7f0349000640 1 -- 172.21.15.6:0/2719900883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03441f0190 con 0x7f03440c4b90 2024-03-22T12:57:23.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.429+0000 7f0349000640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03440c4b90 0x7f03440c9300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0338000c00 tx=0x7f03380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.434+0000 7f0340c00640 1 -- 172.21.15.6:0/2719900883 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338049070 con 0x7f03440c4b90 2024-03-22T12:57:23.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.434+0000 7f0340c00640 1 -- 172.21.15.6:0/2719900883 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033803ab50 con 0x7f03440c4b90 2024-03-22T12:57:23.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.434+0000 7f0340c00640 1 -- 172.21.15.6:0/2719900883 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338044420 con 0x7f03440c4b90 2024-03-22T12:57:23.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.435+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03441f0470 con 0x7f03440c4b90 2024-03-22T12:57:23.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.435+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03441f0a30 con 0x7f03440c4b90 2024-03-22T12:57:23.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.437+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03440757d0 con 0x7f03440c4b90 2024-03-22T12:57:23.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.526+0000 7fabe2000640 1 -- 172.21.15.6:0/698251369 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fabe4023070 con 0x7fabec0cf0e0 2024-03-22T12:57:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.635+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f69c814f140 con 0x7f69c80cc400 2024-03-22T12:57:23.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.689+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f536406def0 con 0x7f53640cc400 2024-03-22T12:57:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.720+0000 7f0340c00640 1 -- 172.21.15.6:0/2719900883 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f033804c070 con 0x7f03440c4b90 2024-03-22T12:57:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.722+0000 7f0340c00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032406bc20 0x7f032406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.722+0000 7f0343e00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032406bc20 0x7f032406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.723+0000 7f0343e00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032406bc20 0x7f032406e0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0330000f60 tx=0x7f03300079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.732+0000 7f8bcf400640 1 -- 172.21.15.6:0/1981158270 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8bd0046020 con 0x7f8bd80cef10 2024-03-22T12:57:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.734+0000 7f8bcf400640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bb806bc20 0x7f8bb806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:23.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.734+0000 7f8bd7400640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bb806bc20 0x7f8bb806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:23.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.735+0000 7f8bd7400640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bb806bc20 0x7f8bb806e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009650 tx=0x7f8bc8006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.738+0000 7f8bcf400640 1 -- 172.21.15.6:0/1981158270 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f8bd0079ee0 con 0x7f8bd80cef10 2024-03-22T12:57:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.738+0000 7f0340c00640 1 -- 172.21.15.6:0/2719900883 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f03380795f0 con 0x7f03440c4b90 2024-03-22T12:57:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.739+0000 7f0340c00640 1 -- 172.21.15.6:0/2719900883 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0338053020 con 0x7f03440c4b90 2024-03-22T12:57:23.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.751+0000 7f535be00640 1 -- 172.21.15.6:0/2522893742 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5354040070 con 0x7f53640cc400 2024-03-22T12:57:23.752 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f534406bc20 msgr2=0x7f534406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f534406bc20 0x7f534406e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f536415c8d0 tx=0x7f5350007990 comp rx=0 tx=0).stop 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640cc400 msgr2=0x7f5364159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640cc400 0x7f5364159940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f53540096f0 tx=0x7f53540046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 shutdown_connections 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f534406bc20 0x7f534406e0e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53640cf0e0 0x7f536415ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53640cd7f0 0x7f536415b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 --2- 172.21.15.6:0/2522893742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f53640cc400 0x7f5364159940 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 >> 172.21.15.6:0/2522893742 conn(0x7f53640cf9f0 msgr2=0x7f536406d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:23.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.755+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 shutdown_connections 2024-03-22T12:57:23.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.756+0000 7f5369800640 1 -- 172.21.15.6:0/2522893742 wait complete. 2024-03-22T12:57:23.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.759+0000 7f8bcf400640 1 -- 172.21.15.6:0/1981158270 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8bd004d030 con 0x7f8bd80cef10 2024-03-22T12:57:23.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.773+0000 7f69c5600640 1 -- 172.21.15.6:0/541401335 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f69bc05d140 con 0x7f69c80cc400 2024-03-22T12:57:23.774 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:23.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f69a806bc20 msgr2=0x7f69a806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f69a806bc20 0x7f69a806e0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f69b4036000 tx=0x7f69b403d040 comp rx=0 tx=0).stop 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cc400 msgr2=0x7f69c81482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cc400 0x7f69c81482d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00b4d0 tx=0x7f69bc00b990 comp rx=0 tx=0).stop 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 shutdown_connections 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69c80cdc30 0x7f69c81e5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f69a806bc20 0x7f69a806e0e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69c80cd7f0 0x7f69c81e4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 --2- 172.21.15.6:0/541401335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69c80cc400 0x7f69c81482d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 >> 172.21.15.6:0/541401335 conn(0x7f69c80cf9f0 msgr2=0x7f69c8154050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.778+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 shutdown_connections 2024-03-22T12:57:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.780+0000 7f69cf000640 1 -- 172.21.15.6:0/541401335 wait complete. 2024-03-22T12:57:23.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509457 for osd.1 2024-03-22T12:57:23.785 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.793+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fabec06def0 con 0x7fabec0cf0e0 2024-03-22T12:57:23.803 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509457 for osd.5 2024-03-22T12:57:23.803 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.879+0000 7fabe2000640 1 -- 172.21.15.6:0/698251369 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fabe4059a90 con 0x7fabec0cf0e0 2024-03-22T12:57:23.880 INFO:teuthology.orchestra.run.smithi006.stdout:64424509456 2024-03-22T12:57:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabcc06bc20 msgr2=0x7fabcc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabcc06bc20 0x7fabcc06e0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fabdc036000 tx=0x7fabdc030000 comp rx=0 tx=0).stop 2024-03-22T12:57:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cf0e0 msgr2=0x7fabec0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cf0e0 0x7fabec0b4c20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fabe400b990 tx=0x7fabe400be50 comp rx=0 tx=0).stop 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 shutdown_connections 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabcc06bc20 0x7fabcc06e0e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabec0cf0e0 0x7fabec0b4c20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabec0cd7f0 0x7fabec0b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.882+0000 7fabeb200640 1 --2- 172.21.15.6:0/698251369 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec0cc400 0x7fabec1595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.883+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 >> 172.21.15.6:0/698251369 conn(0x7fabec0cf9f0 msgr2=0x7fabec0c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.883+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 shutdown_connections 2024-03-22T12:57:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:23.883+0000 7fabeb200640 1 -- 172.21.15.6:0/698251369 wait complete. 2024-03-22T12:57:23.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509456 got 64424509456 for osd.0 2024-03-22T12:57:23.901 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:24.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.035+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f0344067220 con 0x7f03440c4b90 2024-03-22T12:57:24.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.041+0000 7f0340c00640 1 -- 172.21.15.6:0/2719900883 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f033803f070 con 0x7f03440c4b90 2024-03-22T12:57:24.041 INFO:teuthology.orchestra.run.smithi006.stdout:64424509457 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032406bc20 msgr2=0x7f032406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032406bc20 0x7f032406e0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0330000f60 tx=0x7f03300079d0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03440c4b90 msgr2=0x7f03440c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03440c4b90 0x7f03440c9300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0338000c00 tx=0x7f03380046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 shutdown_connections 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032406bc20 0x7f032406e0e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03441547f0 0x7f034415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03440c5570 0x7f0344155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 --2- 172.21.15.6:0/2719900883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03440c4b90 0x7f03440c9300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 >> 172.21.15.6:0/2719900883 conn(0x7f0344066e40 msgr2=0x7f03440cf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 shutdown_connections 2024-03-22T12:57:24.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.044+0000 7f034ba00640 1 -- 172.21.15.6:0/2719900883 wait complete. 2024-03-22T12:57:24.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509457 for osd.3 2024-03-22T12:57:24.070 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:24.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.101+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8bd80cc1e0 con 0x7f8bd80cef10 2024-03-22T12:57:24.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.106+0000 7f8bcf400640 1 -- 172.21.15.6:0/1981158270 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8bd007c540 con 0x7f8bd80cef10 2024-03-22T12:57:24.106 INFO:teuthology.orchestra.run.smithi006.stdout:60129542162 2024-03-22T12:57:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bb806bc20 msgr2=0x7f8bb806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bb806bc20 0x7f8bb806e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009650 tx=0x7f8bc8006ba0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 msgr2=0x7f8bd81f2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd81f2720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009f70 tx=0x7f8bd0004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:24.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 shutdown_connections 2024-03-22T12:57:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bd80cef10 0x7f8bd81f2720 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bd80cc7e0 0x7f8bd81e40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bb806bc20 0x7f8bb806e0e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 --2- 172.21.15.6:0/1981158270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bd80cc400 0x7f8bd80b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 >> 172.21.15.6:0/1981158270 conn(0x7f8bd80cf9f0 msgr2=0x7f8bd8152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.109+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 shutdown_connections 2024-03-22T12:57:24.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.110+0000 7f8bdd200640 1 -- 172.21.15.6:0/1981158270 wait complete. 2024-03-22T12:57:24.125 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542162 got 60129542162 for osd.7 2024-03-22T12:57:24.126 DEBUG:teuthology.parallel:result is None 2024-03-22T12:57:24.126 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-22T12:57:24.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T12:57:24.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.368+0000 7f91f0c00640 1 Processor -- start 2024-03-22T12:57:24.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.368+0000 7f91f0c00640 1 -- start start 2024-03-22T12:57:24.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.370+0000 7f91f0c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f91ec06d840 0x7f91ec15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.370+0000 7f91f0c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec06c7a0 con 0x7f91ec0ca520 2024-03-22T12:57:24.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.370+0000 7f91f0c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec06ca80 con 0x7f91ec0ccad0 2024-03-22T12:57:24.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.370+0000 7f91f0c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec0cd000 con 0x7f91ec06d840 2024-03-22T12:57:24.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.371+0000 7f91e9400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f91ec0ccad0 0x7f91ec162da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:38650/0 (socket says 172.21.15.6:38650) 2024-03-22T12:57:24.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.371+0000 7f91e9400640 1 -- 172.21.15.6:0/3713413246 learned_addr learned my addr 172.21.15.6:0/3713413246 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:24.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.371+0000 7f91ea800640 1 --2- 172.21.15.6:0/3713413246 >> v2:172.21.15.6:3300/0 conn(0x7f91ec06d840 0x7f91ec15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:24.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.374+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060728350 0 0) 0x7f91ec06ca80 con 0x7f91ec0ccad0 2024-03-22T12:57:24.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.374+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d0003520 con 0x7f91ec0ccad0 2024-03-22T12:57:24.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.376+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992750582 0 0) 0x7f91ec06c7a0 con 0x7f91ec0ca520 2024-03-22T12:57:24.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.376+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ec06ca80 con 0x7f91ec0ca520 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.377+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2726595021 0 0) 0x7f91d0003520 con 0x7f91ec0ccad0 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 >> v2:172.21.15.6:3300/0 conn(0x7f91ec06d840 msgr2=0x7f91ec15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 --2- 172.21.15.6:0/3713413246 >> v2:172.21.15.6:3300/0 conn(0x7f91ec06d840 0x7f91ec15b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 >> v1:172.21.15.6:6789/0 conn(0x7f91ec0ca520 legacy=0x7f91ec0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec06cd60 con 0x7f91ec0ccad0 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f91d8001670 con 0x7f91ec0ccad0 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 >> v1:172.21.15.92:6789/0 conn(0x7f91ec0ccad0 legacy=0x7f91ec162da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91d0003b40 0x7f91d0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0004800 0x7f91d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91d00051c0 0x7f91d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0cd000 con 0x7f91d00051c0 2024-03-22T12:57:24.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d000c040 con 0x7f91d0003b40 2024-03-22T12:57:24.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.378+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d000c240 con 0x7f91d0004800 2024-03-22T12:57:24.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.379+0000 7f91e9400640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0004800 0x7f91d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:24.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.380+0000 7f91e9e00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91d0003b40 0x7f91d0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:24.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.382+0000 7f91e9e00640 1 -- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0004800 msgr2=0x7f91d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:24.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.382+0000 7f91e9e00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0004800 0x7f91d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.383+0000 7f91e9e00640 1 -- 172.21.15.6:0/3713413246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91d00051c0 msgr2=0x7f91d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:24.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.383+0000 7f91e9e00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91d00051c0 0x7f91d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.383+0000 7f91e9e00640 1 -- 172.21.15.6:0/3713413246 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec06cd60 con 0x7f91d0003b40 2024-03-22T12:57:24.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.383+0000 7f91ea800640 1 -- 172.21.15.6:0/3713413246 reap_dead start 2024-03-22T12:57:24.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.386+0000 7f91e9e00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91d0003b40 0x7f91d0003f40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0018d0 tx=0x7f91d0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=443a5b14957b1e02 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:24.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.390+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc037070 con 0x7f91d0003b40 2024-03-22T12:57:24.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.390+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc0052d0 con 0x7f91d0003b40 2024-03-22T12:57:24.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.390+0000 7f91e8a00640 1 -- 172.21.15.6:0/3713413246 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0014f0 con 0x7f91d0003b40 2024-03-22T12:57:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.391+0000 7f91f0c00640 1 -- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91d0003b40 msgr2=0x7f91d0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.391+0000 7f91f0c00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91d0003b40 0x7f91d0003f40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0018d0 tx=0x7f91d0005830 comp rx=0 tx=0).stop 2024-03-22T12:57:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.391+0000 7f91f0c00640 1 -- 172.21.15.6:0/3713413246 shutdown_connections 2024-03-22T12:57:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.391+0000 7f91f0c00640 1 --2- 172.21.15.6:0/3713413246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91d0003b40 0x7f91d0003f40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.391+0000 7f91f0c00640 1 -- 172.21.15.6:0/3713413246 >> 172.21.15.6:0/3713413246 conn(0x7f91ec0b16b0 msgr2=0x7f91ec0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.391+0000 7f91f0c00640 1 -- 172.21.15.6:0/3713413246 shutdown_connections 2024-03-22T12:57:24.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.391+0000 7f91f0c00640 1 -- 172.21.15.6:0/3713413246 wait complete. 2024-03-22T12:57:24.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.393+0000 7f91f0c00640 1 Processor -- start 2024-03-22T12:57:24.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.394+0000 7f91f0c00640 1 -- start start 2024-03-22T12:57:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.394+0000 7f91f0c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0003b40 0x7f91ec0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.394+0000 7f91f0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91ec159920 0x7f91ec1f1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.395+0000 7f91f0c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91ec155ed0 0x7f91ec1f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.395+0000 7f91f0c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec154c20 con 0x7f91ec159920 2024-03-22T12:57:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.395+0000 7f91f0c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec06d320 con 0x7f91ec155ed0 2024-03-22T12:57:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.395+0000 7f91f0c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec15b0e0 con 0x7f91d0003b40 2024-03-22T12:57:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.395+0000 7f91e9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91ec159920 0x7f91ec1f1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.395+0000 7f91e9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0003b40 0x7f91ec0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.395+0000 7f91ea800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91ec155ed0 0x7f91ec1f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.396+0000 7f91e9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91ec159920 0x7f91ec1f1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41330/0 (socket says 172.21.15.6:41330) 2024-03-22T12:57:24.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.396+0000 7f91e9400640 1 -- 172.21.15.6:0/1355236050 learned_addr learned my addr 172.21.15.6:0/1355236050 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:24.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.399+0000 7f91e9400640 1 -- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0003b40 msgr2=0x7f91ec0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.399+0000 7f91e9400640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0003b40 0x7f91ec0cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.399+0000 7f91e9400640 1 -- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91ec155ed0 msgr2=0x7f91ec1f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.399+0000 7f91e9400640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91ec155ed0 0x7f91ec1f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.399+0000 7f91e9400640 1 -- 172.21.15.6:0/1355236050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec15ca30 con 0x7f91ec159920 2024-03-22T12:57:24.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.402+0000 7f91e9400640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91ec159920 0x7f91ec1f1e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f91d80072f0 tx=0x7f91d80077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.407+0000 7f91e2000640 1 -- 172.21.15.6:0/1355236050 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d801a070 con 0x7f91ec159920 2024-03-22T12:57:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.407+0000 7f91e2000640 1 -- 172.21.15.6:0/1355236050 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d800f8e0 con 0x7f91ec159920 2024-03-22T12:57:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.407+0000 7f91e2000640 1 -- 172.21.15.6:0/1355236050 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d800fcd0 con 0x7f91ec159920 2024-03-22T12:57:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.408+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec157040 con 0x7f91ec159920 2024-03-22T12:57:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.408+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec1f13b0 con 0x7f91ec159920 2024-03-22T12:57:24.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.409+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ec156dd0 con 0x7f91ec159920 2024-03-22T12:57:24.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.436+0000 7f91e2000640 1 -- 172.21.15.6:0/1355236050 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f91d801d070 con 0x7f91ec159920 2024-03-22T12:57:24.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.437+0000 7f91e2000640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f91cc06bc50 0x7f91cc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.437+0000 7f91e9e00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f91cc06bc50 0x7f91cc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:24.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.438+0000 7f91e9e00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f91cc06bc50 0x7f91cc06e110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91dc034000 tx=0x7f91dc031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.717+0000 7f91e2000640 1 -- 172.21.15.6:0/1355236050 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f91d8056e20 con 0x7f91ec159920 2024-03-22T12:57:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.718+0000 7f91e2000640 1 -- 172.21.15.6:0/1355236050 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f91d8024020 con 0x7f91ec159920 2024-03-22T12:57:24.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.930+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f91ec0756c0 con 0x7f91cc06bc50 2024-03-22T12:57:24.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.940+0000 7f91e2000640 1 -- 172.21.15.6:0/1355236050 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332709 (secure 0 0 0) 0x7f91ec0756c0 con 0x7f91cc06bc50 2024-03-22T12:57:24.945 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:24.947 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T12:57:24.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f91cc06bc50 msgr2=0x7f91cc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f91cc06bc50 0x7f91cc06e110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91dc034000 tx=0x7f91dc031000 comp rx=0 tx=0).stop 2024-03-22T12:57:24.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91ec159920 msgr2=0x7f91ec1f1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91ec159920 0x7f91ec1f1e90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f91d80072f0 tx=0x7f91d80077b0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 shutdown_connections 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f91cc06bc50 0x7f91cc06e110 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91ec155ed0 0x7f91ec1f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91ec159920 0x7f91ec1f1e90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 --2- 172.21.15.6:0/1355236050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91d0003b40 0x7f91ec0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 >> 172.21.15.6:0/1355236050 conn(0x7f91ec0b16b0 msgr2=0x7f91ec0c9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.946+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 shutdown_connections 2024-03-22T12:57:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:24.947+0000 7f91f0c00640 1 -- 172.21.15.6:0/1355236050 wait complete. 2024-03-22T12:57:24.964 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-03-22T12:57:23.701847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":365,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219008,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755712,"statfs":{"total":773094113280,"available":772869849088,"internally_reserved":0,"allocated":2949120,"data_stored":2255496,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5624,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"2.1f","version":"30'13","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.081693+0000","last_change":"2024-03-22T12:56:28.778294+0000","last_active":"2024-03-22T12:57:18.081693+0000","last_peered":"2024-03-22T12:57:18.081693+0000","last_clean":"2024-03-22T12:57:18.081693+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T12:57:18.081693+0000","last_undegraded":"2024-03-22T12:57:18.081693+0000","last_fullsized":"2024-03-22T12:57:18.081693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174773+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T12:57:04.174773+0000","last_peered":"2024-03-22T12:57:04.174773+0000","last_clean":"2024-03-22T12:57:04.174773+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T12:57:04.174773+0000","last_undegraded":"2024-03-22T12:57:04.174773+0000","last_fullsized":"2024-03-22T12:57:04.174773+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173401+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T12:57:04.173401+0000","last_peered":"2024-03-22T12:57:04.173401+0000","last_clean":"2024-03-22T12:57:04.173401+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T12:57:04.173401+0000","last_undegraded":"2024-03-22T12:57:04.173401+0000","last_fullsized":"2024-03-22T12:57:04.173401+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:35:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.175013+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T12:57:04.175013+0000","last_peered":"2024-03-22T12:57:04.175013+0000","last_clean":"2024-03-22T12:57:04.175013+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T12:57:04.175013+0000","last_undegraded":"2024-03-22T12:57:04.175013+0000","last_fullsized":"2024-03-22T12:57:04.175013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172065+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T12:57:04.172065+0000","last_peered":"2024-03-22T12:57:04.172065+0000","last_clean":"2024-03-22T12:57:04.172065+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T12:57:04.172065+0000","last_undegraded":"2024-03-22T12:57:04.172065+0000","last_fullsized":"2024-03-22T12:57:04.172065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042070+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T12:57:05.042070+0000","last_peered":"2024-03-22T12:57:05.042070+0000","last_clean":"2024-03-22T12:57:05.042070+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T12:57:05.042070+0000","last_undegraded":"2024-03-22T12:57:05.042070+0000","last_fullsized":"2024-03-22T12:57:05.042070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:40.477732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'11","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.094735+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T12:57:18.094735+0000","last_peered":"2024-03-22T12:57:18.094735+0000","last_clean":"2024-03-22T12:57:18.094735+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T12:57:18.094735+0000","last_undegraded":"2024-03-22T12:57:18.094735+0000","last_fullsized":"2024-03-22T12:57:18.094735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946408+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T12:57:04.946408+0000","last_peered":"2024-03-22T12:57:04.946408+0000","last_clean":"2024-03-22T12:57:04.946408+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T12:57:04.946408+0000","last_undegraded":"2024-03-22T12:57:04.946408+0000","last_fullsized":"2024-03-22T12:57:04.946408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042290+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T12:57:05.042290+0000","last_peered":"2024-03-22T12:57:05.042290+0000","last_clean":"2024-03-22T12:57:05.042290+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T12:57:05.042290+0000","last_undegraded":"2024-03-22T12:57:05.042290+0000","last_fullsized":"2024-03-22T12:57:05.042290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:29.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.093644+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T12:57:18.093644+0000","last_peered":"2024-03-22T12:57:18.093644+0000","last_clean":"2024-03-22T12:57:18.093644+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T12:57:18.093644+0000","last_undegraded":"2024-03-22T12:57:18.093644+0000","last_fullsized":"2024-03-22T12:57:18.093644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946567+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T12:57:04.946567+0000","last_peered":"2024-03-22T12:57:04.946567+0000","last_clean":"2024-03-22T12:57:04.946567+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T12:57:04.946567+0000","last_undegraded":"2024-03-22T12:57:04.946567+0000","last_fullsized":"2024-03-22T12:57:04.946567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042408+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T12:57:05.042408+0000","last_peered":"2024-03-22T12:57:05.042408+0000","last_clean":"2024-03-22T12:57:05.042408+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T12:57:05.042408+0000","last_undegraded":"2024-03-22T12:57:05.042408+0000","last_fullsized":"2024-03-22T12:57:05.042408+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247053+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T12:57:04.247053+0000","last_peered":"2024-03-22T12:57:04.247053+0000","last_clean":"2024-03-22T12:57:04.247053+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T12:57:04.247053+0000","last_undegraded":"2024-03-22T12:57:04.247053+0000","last_fullsized":"2024-03-22T12:57:04.247053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.305489+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T12:57:12.305489+0000","last_peered":"2024-03-22T12:57:12.305489+0000","last_clean":"2024-03-22T12:57:12.305489+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T12:57:12.305489+0000","last_undegraded":"2024-03-22T12:57:12.305489+0000","last_fullsized":"2024-03-22T12:57:12.305489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042319+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T12:57:05.042319+0000","last_peered":"2024-03-22T12:57:05.042319+0000","last_clean":"2024-03-22T12:57:05.042319+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T12:57:05.042319+0000","last_undegraded":"2024-03-22T12:57:05.042319+0000","last_fullsized":"2024-03-22T12:57:05.042319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:29.130909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174615+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T12:57:04.174615+0000","last_peered":"2024-03-22T12:57:04.174615+0000","last_clean":"2024-03-22T12:57:04.174615+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T12:57:04.174615+0000","last_undegraded":"2024-03-22T12:57:04.174615+0000","last_fullsized":"2024-03-22T12:57:04.174615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.314072+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T12:57:16.314072+0000","last_peered":"2024-03-22T12:57:16.314072+0000","last_clean":"2024-03-22T12:57:16.314072+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T12:57:16.314072+0000","last_undegraded":"2024-03-22T12:57:16.314072+0000","last_fullsized":"2024-03-22T12:57:16.314072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042296+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T12:57:05.042296+0000","last_peered":"2024-03-22T12:57:05.042296+0000","last_clean":"2024-03-22T12:57:05.042296+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T12:57:05.042296+0000","last_undegraded":"2024-03-22T12:57:05.042296+0000","last_fullsized":"2024-03-22T12:57:05.042296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:40.776689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173279+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T12:57:04.173279+0000","last_peered":"2024-03-22T12:57:04.173279+0000","last_clean":"2024-03-22T12:57:04.173279+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T12:57:04.173279+0000","last_undegraded":"2024-03-22T12:57:04.173279+0000","last_fullsized":"2024-03-22T12:57:04.173279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:17.986612+0000","last_change":"2024-03-22T12:56:28.757253+0000","last_active":"2024-03-22T12:57:17.986612+0000","last_peered":"2024-03-22T12:57:17.986612+0000","last_clean":"2024-03-22T12:57:17.986612+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T12:57:17.986612+0000","last_undegraded":"2024-03-22T12:57:17.986612+0000","last_fullsized":"2024-03-22T12:57:17.986612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:48.506816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185414+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T12:57:04.185414+0000","last_peered":"2024-03-22T12:57:04.185414+0000","last_clean":"2024-03-22T12:57:04.185414+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T12:57:04.185414+0000","last_undegraded":"2024-03-22T12:57:04.185414+0000","last_fullsized":"2024-03-22T12:57:04.185414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042646+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T12:57:05.042646+0000","last_peered":"2024-03-22T12:57:05.042646+0000","last_clean":"2024-03-22T12:57:05.042646+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T12:57:05.042646+0000","last_undegraded":"2024-03-22T12:57:05.042646+0000","last_fullsized":"2024-03-22T12:57:05.042646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:34.508555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.306479+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T12:57:12.306479+0000","last_peered":"2024-03-22T12:57:12.306479+0000","last_clean":"2024-03-22T12:57:12.306479+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T12:57:12.306479+0000","last_undegraded":"2024-03-22T12:57:12.306479+0000","last_fullsized":"2024-03-22T12:57:12.306479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042323+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T12:57:05.042323+0000","last_peered":"2024-03-22T12:57:05.042323+0000","last_clean":"2024-03-22T12:57:05.042323+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T12:57:05.042323+0000","last_undegraded":"2024-03-22T12:57:05.042323+0000","last_fullsized":"2024-03-22T12:57:05.042323+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:09.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172429+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T12:57:04.172429+0000","last_peered":"2024-03-22T12:57:04.172429+0000","last_clean":"2024-03-22T12:57:04.172429+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T12:57:04.172429+0000","last_undegraded":"2024-03-22T12:57:04.172429+0000","last_fullsized":"2024-03-22T12:57:04.172429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.093203+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T12:57:18.093203+0000","last_peered":"2024-03-22T12:57:18.093203+0000","last_clean":"2024-03-22T12:57:18.093203+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T12:57:18.093203+0000","last_undegraded":"2024-03-22T12:57:18.093203+0000","last_fullsized":"2024-03-22T12:57:18.093203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042166+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T12:57:05.042166+0000","last_peered":"2024-03-22T12:57:05.042166+0000","last_clean":"2024-03-22T12:57:05.042166+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T12:57:05.042166+0000","last_undegraded":"2024-03-22T12:57:05.042166+0000","last_fullsized":"2024-03-22T12:57:05.042166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:13.307688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176050+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T12:57:04.176050+0000","last_peered":"2024-03-22T12:57:04.176050+0000","last_clean":"2024-03-22T12:57:04.176050+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T12:57:04.176050+0000","last_undegraded":"2024-03-22T12:57:04.176050+0000","last_fullsized":"2024-03-22T12:57:04.176050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:06.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.315957+0000","last_change":"2024-03-22T12:56:28.767925+0000","last_active":"2024-03-22T12:57:16.315957+0000","last_peered":"2024-03-22T12:57:16.315957+0000","last_clean":"2024-03-22T12:57:16.315957+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T12:57:16.315957+0000","last_undegraded":"2024-03-22T12:57:16.315957+0000","last_fullsized":"2024-03-22T12:57:16.315957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172825+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T12:57:04.172825+0000","last_peered":"2024-03-22T12:57:04.172825+0000","last_clean":"2024-03-22T12:57:04.172825+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T12:57:04.172825+0000","last_undegraded":"2024-03-22T12:57:04.172825+0000","last_fullsized":"2024-03-22T12:57:04.172825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:40:09.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185111+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T12:57:04.185111+0000","last_peered":"2024-03-22T12:57:04.185111+0000","last_clean":"2024-03-22T12:57:04.185111+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T12:57:04.185111+0000","last_undegraded":"2024-03-22T12:57:04.185111+0000","last_fullsized":"2024-03-22T12:57:04.185111+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.308370+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T12:57:16.308370+0000","last_peered":"2024-03-22T12:57:16.308370+0000","last_clean":"2024-03-22T12:57:16.308370+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T12:57:16.308370+0000","last_undegraded":"2024-03-22T12:57:16.308370+0000","last_fullsized":"2024-03-22T12:57:16.308370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170966+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T12:57:04.170966+0000","last_peered":"2024-03-22T12:57:04.170966+0000","last_clean":"2024-03-22T12:57:04.170966+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T12:57:04.170966+0000","last_undegraded":"2024-03-22T12:57:04.170966+0000","last_fullsized":"2024-03-22T12:57:04.170966+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946629+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T12:57:04.946629+0000","last_peered":"2024-03-22T12:57:04.946629+0000","last_clean":"2024-03-22T12:57:04.946629+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T12:57:04.946629+0000","last_undegraded":"2024-03-22T12:57:04.946629+0000","last_fullsized":"2024-03-22T12:57:04.946629+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185241+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T12:57:04.185241+0000","last_peered":"2024-03-22T12:57:04.185241+0000","last_clean":"2024-03-22T12:57:04.185241+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T12:57:04.185241+0000","last_undegraded":"2024-03-22T12:57:04.185241+0000","last_fullsized":"2024-03-22T12:57:04.185241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045655+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T12:57:05.045655+0000","last_peered":"2024-03-22T12:57:05.045655+0000","last_clean":"2024-03-22T12:57:05.045655+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T12:57:05.045655+0000","last_undegraded":"2024-03-22T12:57:05.045655+0000","last_fullsized":"2024-03-22T12:57:05.045655+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:45.197847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042741+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T12:57:05.042741+0000","last_peered":"2024-03-22T12:57:05.042741+0000","last_clean":"2024-03-22T12:57:05.042741+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T12:57:05.042741+0000","last_undegraded":"2024-03-22T12:57:05.042741+0000","last_fullsized":"2024-03-22T12:57:05.042741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:23.839356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.088903+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T12:57:18.088903+0000","last_peered":"2024-03-22T12:57:18.088903+0000","last_clean":"2024-03-22T12:57:18.088903+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T12:57:18.088903+0000","last_undegraded":"2024-03-22T12:57:18.088903+0000","last_fullsized":"2024-03-22T12:57:18.088903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184845+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T12:57:04.184845+0000","last_peered":"2024-03-22T12:57:04.184845+0000","last_clean":"2024-03-22T12:57:04.184845+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T12:57:04.184845+0000","last_undegraded":"2024-03-22T12:57:04.184845+0000","last_fullsized":"2024-03-22T12:57:04.184845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184792+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T12:57:04.184792+0000","last_peered":"2024-03-22T12:57:04.184792+0000","last_clean":"2024-03-22T12:57:04.184792+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T12:57:04.184792+0000","last_undegraded":"2024-03-22T12:57:04.184792+0000","last_fullsized":"2024-03-22T12:57:04.184792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:14.483147+0000","last_change":"2024-03-22T12:56:28.763384+0000","last_active":"2024-03-22T12:57:14.483147+0000","last_peered":"2024-03-22T12:57:14.483147+0000","last_clean":"2024-03-22T12:57:14.483147+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T12:57:14.483147+0000","last_undegraded":"2024-03-22T12:57:14.483147+0000","last_fullsized":"2024-03-22T12:57:14.483147+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:15:45.463056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.952755+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T12:57:04.952755+0000","last_peered":"2024-03-22T12:57:04.952755+0000","last_clean":"2024-03-22T12:57:04.952755+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T12:57:04.952755+0000","last_undegraded":"2024-03-22T12:57:04.952755+0000","last_fullsized":"2024-03-22T12:57:04.952755+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:48.651995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.169659+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T12:57:04.169659+0000","last_peered":"2024-03-22T12:57:04.169659+0000","last_clean":"2024-03-22T12:57:04.169659+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T12:57:04.169659+0000","last_undegraded":"2024-03-22T12:57:04.169659+0000","last_fullsized":"2024-03-22T12:57:04.169659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:36:50.048241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:14.391765+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T12:57:14.391765+0000","last_peered":"2024-03-22T12:57:14.391765+0000","last_clean":"2024-03-22T12:57:14.391765+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T12:57:14.391765+0000","last_undegraded":"2024-03-22T12:57:14.391765+0000","last_fullsized":"2024-03-22T12:57:14.391765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172286+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T12:57:04.172286+0000","last_peered":"2024-03-22T12:57:04.172286+0000","last_clean":"2024-03-22T12:57:04.172286+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T12:57:04.172286+0000","last_undegraded":"2024-03-22T12:57:04.172286+0000","last_fullsized":"2024-03-22T12:57:04.172286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:29:31.400078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174048+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T12:57:04.174048+0000","last_peered":"2024-03-22T12:57:04.174048+0000","last_clean":"2024-03-22T12:57:04.174048+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T12:57:04.174048+0000","last_undegraded":"2024-03-22T12:57:04.174048+0000","last_fullsized":"2024-03-22T12:57:04.174048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":34,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.312656+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T12:57:16.312656+0000","last_peered":"2024-03-22T12:57:16.312656+0000","last_clean":"2024-03-22T12:57:16.312656+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T12:57:16.312656+0000","last_undegraded":"2024-03-22T12:57:16.312656+0000","last_fullsized":"2024-03-22T12:57:16.312656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354430+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T12:57:04.354430+0000","last_peered":"2024-03-22T12:57:04.354430+0000","last_clean":"2024-03-22T12:57:04.354430+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T12:57:04.354430+0000","last_undegraded":"2024-03-22T12:57:04.354430+0000","last_fullsized":"2024-03-22T12:57:04.354430+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247232+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T12:57:04.247232+0000","last_peered":"2024-03-22T12:57:04.247232+0000","last_clean":"2024-03-22T12:57:04.247232+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T12:57:04.247232+0000","last_undegraded":"2024-03-22T12:57:04.247232+0000","last_fullsized":"2024-03-22T12:57:04.247232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.312265+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T12:57:16.312265+0000","last_peered":"2024-03-22T12:57:16.312265+0000","last_clean":"2024-03-22T12:57:16.312265+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T12:57:16.312265+0000","last_undegraded":"2024-03-22T12:57:16.312265+0000","last_fullsized":"2024-03-22T12:57:16.312265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247035+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T12:57:04.247035+0000","last_peered":"2024-03-22T12:57:04.247035+0000","last_clean":"2024-03-22T12:57:04.247035+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T12:57:04.247035+0000","last_undegraded":"2024-03-22T12:57:04.247035+0000","last_fullsized":"2024-03-22T12:57:04.247035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946466+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T12:57:04.946466+0000","last_peered":"2024-03-22T12:57:04.946466+0000","last_clean":"2024-03-22T12:57:04.946466+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T12:57:04.946466+0000","last_undegraded":"2024-03-22T12:57:04.946466+0000","last_fullsized":"2024-03-22T12:57:04.946466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.314152+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T12:57:16.314152+0000","last_peered":"2024-03-22T12:57:16.314152+0000","last_clean":"2024-03-22T12:57:16.314152+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T12:57:16.314152+0000","last_undegraded":"2024-03-22T12:57:16.314152+0000","last_fullsized":"2024-03-22T12:57:16.314152+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:22.329365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171343+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T12:57:04.171343+0000","last_peered":"2024-03-22T12:57:04.171343+0000","last_clean":"2024-03-22T12:57:04.171343+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T12:57:04.171343+0000","last_undegraded":"2024-03-22T12:57:04.171343+0000","last_fullsized":"2024-03-22T12:57:04.171343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355029+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T12:57:04.355029+0000","last_peered":"2024-03-22T12:57:04.355029+0000","last_clean":"2024-03-22T12:57:04.355029+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T12:57:04.355029+0000","last_undegraded":"2024-03-22T12:57:04.355029+0000","last_fullsized":"2024-03-22T12:57:04.355029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:17.985421+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T12:57:17.985421+0000","last_peered":"2024-03-22T12:57:17.985421+0000","last_clean":"2024-03-22T12:57:17.985421+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T12:57:17.985421+0000","last_undegraded":"2024-03-22T12:57:17.985421+0000","last_fullsized":"2024-03-22T12:57:17.985421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946109+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T12:57:04.946109+0000","last_peered":"2024-03-22T12:57:04.946109+0000","last_clean":"2024-03-22T12:57:04.946109+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T12:57:04.946109+0000","last_undegraded":"2024-03-22T12:57:04.946109+0000","last_fullsized":"2024-03-22T12:57:04.946109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042539+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T12:57:05.042539+0000","last_peered":"2024-03-22T12:57:05.042539+0000","last_clean":"2024-03-22T12:57:05.042539+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T12:57:05.042539+0000","last_undegraded":"2024-03-22T12:57:05.042539+0000","last_fullsized":"2024-03-22T12:57:05.042539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:24:46.021828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042059+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T12:57:05.042059+0000","last_peered":"2024-03-22T12:57:05.042059+0000","last_clean":"2024-03-22T12:57:05.042059+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T12:57:05.042059+0000","last_undegraded":"2024-03-22T12:57:05.042059+0000","last_fullsized":"2024-03-22T12:57:05.042059+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:38.316319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354615+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T12:57:04.354615+0000","last_peered":"2024-03-22T12:57:04.354615+0000","last_clean":"2024-03-22T12:57:04.354615+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T12:57:04.354615+0000","last_undegraded":"2024-03-22T12:57:04.354615+0000","last_fullsized":"2024-03-22T12:57:04.354615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'17","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.314563+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T12:57:16.314563+0000","last_peered":"2024-03-22T12:57:16.314563+0000","last_clean":"2024-03-22T12:57:16.314563+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T12:57:16.314563+0000","last_undegraded":"2024-03-22T12:57:16.314563+0000","last_fullsized":"2024-03-22T12:57:16.314563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:20:57.722062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172194+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T12:57:04.172194+0000","last_peered":"2024-03-22T12:57:04.172194+0000","last_clean":"2024-03-22T12:57:04.172194+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T12:57:04.172194+0000","last_undegraded":"2024-03-22T12:57:04.172194+0000","last_fullsized":"2024-03-22T12:57:04.172194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247088+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T12:57:04.247088+0000","last_peered":"2024-03-22T12:57:04.247088+0000","last_clean":"2024-03-22T12:57:04.247088+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T12:57:04.247088+0000","last_undegraded":"2024-03-22T12:57:04.247088+0000","last_fullsized":"2024-03-22T12:57:04.247088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.302102+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T12:57:16.302102+0000","last_peered":"2024-03-22T12:57:16.302102+0000","last_clean":"2024-03-22T12:57:16.302102+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T12:57:16.302102+0000","last_undegraded":"2024-03-22T12:57:16.302102+0000","last_fullsized":"2024-03-22T12:57:16.302102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176522+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T12:57:04.176522+0000","last_peered":"2024-03-22T12:57:04.176522+0000","last_clean":"2024-03-22T12:57:04.176522+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T12:57:04.176522+0000","last_undegraded":"2024-03-22T12:57:04.176522+0000","last_fullsized":"2024-03-22T12:57:04.176522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:51.788457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354739+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T12:57:04.354739+0000","last_peered":"2024-03-22T12:57:04.354739+0000","last_clean":"2024-03-22T12:57:04.354739+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T12:57:04.354739+0000","last_undegraded":"2024-03-22T12:57:04.354739+0000","last_fullsized":"2024-03-22T12:57:04.354739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.310176+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T12:57:16.310176+0000","last_peered":"2024-03-22T12:57:16.310176+0000","last_clean":"2024-03-22T12:57:16.310176+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T12:57:16.310176+0000","last_undegraded":"2024-03-22T12:57:16.310176+0000","last_fullsized":"2024-03-22T12:57:16.310176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.949816+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T12:57:04.949816+0000","last_peered":"2024-03-22T12:57:04.949816+0000","last_clean":"2024-03-22T12:57:04.949816+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T12:57:04.949816+0000","last_undegraded":"2024-03-22T12:57:04.949816+0000","last_fullsized":"2024-03-22T12:57:04.949816+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:08:12.860933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170220+0000","last_change":"2024-03-22T12:56:30.299969+0000","last_active":"2024-03-22T12:57:04.170220+0000","last_peered":"2024-03-22T12:57:04.170220+0000","last_clean":"2024-03-22T12:57:04.170220+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T12:57:04.170220+0000","last_undegraded":"2024-03-22T12:57:04.170220+0000","last_fullsized":"2024-03-22T12:57:04.170220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'9","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.080641+0000","last_change":"2024-03-22T12:56:28.777142+0000","last_active":"2024-03-22T12:57:18.080641+0000","last_peered":"2024-03-22T12:57:18.080641+0000","last_clean":"2024-03-22T12:57:18.080641+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T12:57:18.080641+0000","last_undegraded":"2024-03-22T12:57:18.080641+0000","last_fullsized":"2024-03-22T12:57:18.080641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:49.411808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946528+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T12:57:04.946528+0000","last_peered":"2024-03-22T12:57:04.946528+0000","last_clean":"2024-03-22T12:57:04.946528+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T12:57:04.946528+0000","last_undegraded":"2024-03-22T12:57:04.946528+0000","last_fullsized":"2024-03-22T12:57:04.946528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354901+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T12:57:04.354901+0000","last_peered":"2024-03-22T12:57:04.354901+0000","last_clean":"2024-03-22T12:57:04.354901+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T12:57:04.354901+0000","last_undegraded":"2024-03-22T12:57:04.354901+0000","last_fullsized":"2024-03-22T12:57:04.354901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'14","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.092975+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T12:57:18.092975+0000","last_peered":"2024-03-22T12:57:18.092975+0000","last_clean":"2024-03-22T12:57:18.092975+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T12:57:18.092975+0000","last_undegraded":"2024-03-22T12:57:18.092975+0000","last_fullsized":"2024-03-22T12:57:18.092975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184655+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T12:57:04.184655+0000","last_peered":"2024-03-22T12:57:04.184655+0000","last_clean":"2024-03-22T12:57:04.184655+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T12:57:04.184655+0000","last_undegraded":"2024-03-22T12:57:04.184655+0000","last_fullsized":"2024-03-22T12:57:04.184655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042567+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T12:57:05.042567+0000","last_peered":"2024-03-22T12:57:05.042567+0000","last_clean":"2024-03-22T12:57:05.042567+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T12:57:05.042567+0000","last_undegraded":"2024-03-22T12:57:05.042567+0000","last_fullsized":"2024-03-22T12:57:05.042567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:49.776407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.413187+0000","last_change":"2024-03-22T12:56:28.769041+0000","last_active":"2024-03-22T12:57:12.413187+0000","last_peered":"2024-03-22T12:57:12.413187+0000","last_clean":"2024-03-22T12:57:12.413187+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T12:57:12.413187+0000","last_undegraded":"2024-03-22T12:57:12.413187+0000","last_fullsized":"2024-03-22T12:57:12.413187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042461+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T12:57:05.042461+0000","last_peered":"2024-03-22T12:57:05.042461+0000","last_clean":"2024-03-22T12:57:05.042461+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T12:57:05.042461+0000","last_undegraded":"2024-03-22T12:57:05.042461+0000","last_fullsized":"2024-03-22T12:57:05.042461+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:56.815720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172275+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T12:57:04.172275+0000","last_peered":"2024-03-22T12:57:04.172275+0000","last_clean":"2024-03-22T12:57:04.172275+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T12:57:04.172275+0000","last_undegraded":"2024-03-22T12:57:04.172275+0000","last_fullsized":"2024-03-22T12:57:04.172275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.085201+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T12:57:18.085201+0000","last_peered":"2024-03-22T12:57:18.085201+0000","last_clean":"2024-03-22T12:57:18.085201+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T12:57:18.085201+0000","last_undegraded":"2024-03-22T12:57:18.085201+0000","last_fullsized":"2024-03-22T12:57:18.085201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173079+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T12:57:04.173079+0000","last_peered":"2024-03-22T12:57:04.173079+0000","last_clean":"2024-03-22T12:57:04.173079+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T12:57:04.173079+0000","last_undegraded":"2024-03-22T12:57:04.173079+0000","last_fullsized":"2024-03-22T12:57:04.173079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:06:47.716384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946555+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T12:57:04.946555+0000","last_peered":"2024-03-22T12:57:04.946555+0000","last_clean":"2024-03-22T12:57:04.946555+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T12:57:04.946555+0000","last_undegraded":"2024-03-22T12:57:04.946555+0000","last_fullsized":"2024-03-22T12:57:04.946555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.310332+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T12:57:12.310332+0000","last_peered":"2024-03-22T12:57:12.310332+0000","last_clean":"2024-03-22T12:57:12.310332+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T12:57:12.310332+0000","last_undegraded":"2024-03-22T12:57:12.310332+0000","last_fullsized":"2024-03-22T12:57:12.310332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174378+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T12:57:04.174378+0000","last_peered":"2024-03-22T12:57:04.174378+0000","last_clean":"2024-03-22T12:57:04.174378+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T12:57:04.174378+0000","last_undegraded":"2024-03-22T12:57:04.174378+0000","last_fullsized":"2024-03-22T12:57:04.174378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172090+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T12:57:04.172090+0000","last_peered":"2024-03-22T12:57:04.172090+0000","last_clean":"2024-03-22T12:57:04.172090+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T12:57:04.172090+0000","last_undegraded":"2024-03-22T12:57:04.172090+0000","last_fullsized":"2024-03-22T12:57:04.172090+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184400+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T12:57:04.184400+0000","last_peered":"2024-03-22T12:57:04.184400+0000","last_clean":"2024-03-22T12:57:04.184400+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T12:57:04.184400+0000","last_undegraded":"2024-03-22T12:57:04.184400+0000","last_fullsized":"2024-03-22T12:57:04.184400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.307298+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T12:57:16.307298+0000","last_peered":"2024-03-22T12:57:16.307298+0000","last_clean":"2024-03-22T12:57:16.307298+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T12:57:16.307298+0000","last_undegraded":"2024-03-22T12:57:16.307298+0000","last_fullsized":"2024-03-22T12:57:16.307298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247413+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T12:57:04.247413+0000","last_peered":"2024-03-22T12:57:04.247413+0000","last_clean":"2024-03-22T12:57:04.247413+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T12:57:04.247413+0000","last_undegraded":"2024-03-22T12:57:04.247413+0000","last_fullsized":"2024-03-22T12:57:04.247413+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173181+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T12:57:04.173181+0000","last_peered":"2024-03-22T12:57:04.173181+0000","last_clean":"2024-03-22T12:57:04.173181+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T12:57:04.173181+0000","last_undegraded":"2024-03-22T12:57:04.173181+0000","last_fullsized":"2024-03-22T12:57:04.173181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:14.480598+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T12:57:14.480598+0000","last_peered":"2024-03-22T12:57:14.480598+0000","last_clean":"2024-03-22T12:57:14.480598+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T12:57:14.480598+0000","last_undegraded":"2024-03-22T12:57:14.480598+0000","last_fullsized":"2024-03-22T12:57:14.480598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:31:41.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.304889+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T12:57:16.304889+0000","last_peered":"2024-03-22T12:57:16.304889+0000","last_clean":"2024-03-22T12:57:16.304889+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T12:57:16.304889+0000","last_undegraded":"2024-03-22T12:57:16.304889+0000","last_fullsized":"2024-03-22T12:57:16.304889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.047312+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T12:57:05.047312+0000","last_peered":"2024-03-22T12:57:05.047312+0000","last_clean":"2024-03-22T12:57:05.047312+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T12:57:05.047312+0000","last_undegraded":"2024-03-22T12:57:05.047312+0000","last_fullsized":"2024-03-22T12:57:05.047312+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.180347+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T12:57:04.180347+0000","last_peered":"2024-03-22T12:57:04.180347+0000","last_clean":"2024-03-22T12:57:04.180347+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T12:57:04.180347+0000","last_undegraded":"2024-03-22T12:57:04.180347+0000","last_fullsized":"2024-03-22T12:57:04.180347+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:43.256977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.481695+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T12:57:12.481695+0000","last_peered":"2024-03-22T12:57:12.481695+0000","last_clean":"2024-03-22T12:57:12.481695+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T12:57:12.481695+0000","last_undegraded":"2024-03-22T12:57:12.481695+0000","last_fullsized":"2024-03-22T12:57:12.481695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042643+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T12:57:05.042643+0000","last_peered":"2024-03-22T12:57:05.042643+0000","last_clean":"2024-03-22T12:57:05.042643+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T12:57:05.042643+0000","last_undegraded":"2024-03-22T12:57:05.042643+0000","last_fullsized":"2024-03-22T12:57:05.042643+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:48.882251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174116+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T12:57:04.174116+0000","last_peered":"2024-03-22T12:57:04.174116+0000","last_clean":"2024-03-22T12:57:04.174116+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T12:57:04.174116+0000","last_undegraded":"2024-03-22T12:57:04.174116+0000","last_fullsized":"2024-03-22T12:57:04.174116+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247032+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T12:57:04.247032+0000","last_peered":"2024-03-22T12:57:04.247032+0000","last_clean":"2024-03-22T12:57:04.247032+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T12:57:04.247032+0000","last_undegraded":"2024-03-22T12:57:04.247032+0000","last_fullsized":"2024-03-22T12:57:04.247032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.188189+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T12:57:04.188189+0000","last_peered":"2024-03-22T12:57:04.188189+0000","last_clean":"2024-03-22T12:57:04.188189+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T12:57:04.188189+0000","last_undegraded":"2024-03-22T12:57:04.188189+0000","last_fullsized":"2024-03-22T12:57:04.188189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946357+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T12:57:04.946357+0000","last_peered":"2024-03-22T12:57:04.946357+0000","last_clean":"2024-03-22T12:57:04.946357+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T12:57:04.946357+0000","last_undegraded":"2024-03-22T12:57:04.946357+0000","last_fullsized":"2024-03-22T12:57:04.946357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171397+0000","last_change":"2024-03-22T12:56:30.300244+0000","last_active":"2024-03-22T12:57:04.171397+0000","last_peered":"2024-03-22T12:57:04.171397+0000","last_clean":"2024-03-22T12:57:04.171397+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T12:57:04.171397+0000","last_undegraded":"2024-03-22T12:57:04.171397+0000","last_fullsized":"2024-03-22T12:57:04.171397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174764+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T12:57:04.174764+0000","last_peered":"2024-03-22T12:57:04.174764+0000","last_clean":"2024-03-22T12:57:04.174764+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T12:57:04.174764+0000","last_undegraded":"2024-03-22T12:57:04.174764+0000","last_fullsized":"2024-03-22T12:57:04.174764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185804+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T12:57:04.185804+0000","last_peered":"2024-03-22T12:57:04.185804+0000","last_clean":"2024-03-22T12:57:04.185804+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T12:57:04.185804+0000","last_undegraded":"2024-03-22T12:57:04.185804+0000","last_fullsized":"2024-03-22T12:57:04.185804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247526+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T12:57:04.247526+0000","last_peered":"2024-03-22T12:57:04.247526+0000","last_clean":"2024-03-22T12:57:04.247526+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T12:57:04.247526+0000","last_undegraded":"2024-03-22T12:57:04.247526+0000","last_fullsized":"2024-03-22T12:57:04.247526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.175763+0000","last_change":"2024-03-22T12:56:30.308903+0000","last_active":"2024-03-22T12:57:04.175763+0000","last_peered":"2024-03-22T12:57:04.175763+0000","last_clean":"2024-03-22T12:57:04.175763+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T12:57:04.175763+0000","last_undegraded":"2024-03-22T12:57:04.175763+0000","last_fullsized":"2024-03-22T12:57:04.175763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185929+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T12:57:04.185929+0000","last_peered":"2024-03-22T12:57:04.185929+0000","last_clean":"2024-03-22T12:57:04.185929+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T12:57:04.185929+0000","last_undegraded":"2024-03-22T12:57:04.185929+0000","last_fullsized":"2024-03-22T12:57:04.185929+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:13:57.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045995+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T12:57:05.045995+0000","last_peered":"2024-03-22T12:57:05.045995+0000","last_clean":"2024-03-22T12:57:05.045995+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T12:57:05.045995+0000","last_undegraded":"2024-03-22T12:57:05.045995+0000","last_fullsized":"2024-03-22T12:57:05.045995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:19.797978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946253+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T12:57:04.946253+0000","last_peered":"2024-03-22T12:57:04.946253+0000","last_clean":"2024-03-22T12:57:04.946253+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T12:57:04.946253+0000","last_undegraded":"2024-03-22T12:57:04.946253+0000","last_fullsized":"2024-03-22T12:57:04.946253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171180+0000","last_change":"2024-03-22T12:56:30.299214+0000","last_active":"2024-03-22T12:57:04.171180+0000","last_peered":"2024-03-22T12:57:04.171180+0000","last_clean":"2024-03-22T12:57:04.171180+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T12:57:04.171180+0000","last_undegraded":"2024-03-22T12:57:04.171180+0000","last_fullsized":"2024-03-22T12:57:04.171180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:35.356723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247391+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T12:57:04.247391+0000","last_peered":"2024-03-22T12:57:04.247391+0000","last_clean":"2024-03-22T12:57:04.247391+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T12:57:04.247391+0000","last_undegraded":"2024-03-22T12:57:04.247391+0000","last_fullsized":"2024-03-22T12:57:04.247391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.189179+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T12:57:04.189179+0000","last_peered":"2024-03-22T12:57:04.189179+0000","last_clean":"2024-03-22T12:57:04.189179+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T12:57:04.189179+0000","last_undegraded":"2024-03-22T12:57:04.189179+0000","last_fullsized":"2024-03-22T12:57:04.189179+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:57:33.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.182435+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T12:57:04.182435+0000","last_peered":"2024-03-22T12:57:04.182435+0000","last_clean":"2024-03-22T12:57:04.182435+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T12:57:04.182435+0000","last_undegraded":"2024-03-22T12:57:04.182435+0000","last_fullsized":"2024-03-22T12:57:04.182435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:38.987151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946819+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T12:57:04.946819+0000","last_peered":"2024-03-22T12:57:04.946819+0000","last_clean":"2024-03-22T12:57:04.946819+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T12:57:04.946819+0000","last_undegraded":"2024-03-22T12:57:04.946819+0000","last_fullsized":"2024-03-22T12:57:04.946819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946220+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T12:57:04.946220+0000","last_peered":"2024-03-22T12:57:04.946220+0000","last_clean":"2024-03-22T12:57:04.946220+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T12:57:04.946220+0000","last_undegraded":"2024-03-22T12:57:04.946220+0000","last_fullsized":"2024-03-22T12:57:04.946220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184477+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T12:57:04.184477+0000","last_peered":"2024-03-22T12:57:04.184477+0000","last_clean":"2024-03-22T12:57:04.184477+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T12:57:04.184477+0000","last_undegraded":"2024-03-22T12:57:04.184477+0000","last_fullsized":"2024-03-22T12:57:04.184477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170174+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T12:57:04.170174+0000","last_peered":"2024-03-22T12:57:04.170174+0000","last_clean":"2024-03-22T12:57:04.170174+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T12:57:04.170174+0000","last_undegraded":"2024-03-22T12:57:04.170174+0000","last_fullsized":"2024-03-22T12:57:04.170174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174348+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T12:57:04.174348+0000","last_peered":"2024-03-22T12:57:04.174348+0000","last_clean":"2024-03-22T12:57:04.174348+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T12:57:04.174348+0000","last_undegraded":"2024-03-22T12:57:04.174348+0000","last_fullsized":"2024-03-22T12:57:04.174348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042563+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T12:57:05.042563+0000","last_peered":"2024-03-22T12:57:05.042563+0000","last_clean":"2024-03-22T12:57:05.042563+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T12:57:05.042563+0000","last_undegraded":"2024-03-22T12:57:05.042563+0000","last_fullsized":"2024-03-22T12:57:05.042563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:23.598956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.251063+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T12:57:04.251063+0000","last_peered":"2024-03-22T12:57:04.251063+0000","last_clean":"2024-03-22T12:57:04.251063+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T12:57:04.251063+0000","last_undegraded":"2024-03-22T12:57:04.251063+0000","last_fullsized":"2024-03-22T12:57:04.251063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247414+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T12:57:04.247414+0000","last_peered":"2024-03-22T12:57:04.247414+0000","last_clean":"2024-03-22T12:57:04.247414+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T12:57:04.247414+0000","last_undegraded":"2024-03-22T12:57:04.247414+0000","last_fullsized":"2024-03-22T12:57:04.247414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355010+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T12:57:04.355010+0000","last_peered":"2024-03-22T12:57:04.355010+0000","last_clean":"2024-03-22T12:57:04.355010+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T12:57:04.355010+0000","last_undegraded":"2024-03-22T12:57:04.355010+0000","last_fullsized":"2024-03-22T12:57:04.355010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172977+0000","last_change":"2024-03-22T12:56:30.309258+0000","last_active":"2024-03-22T12:57:04.172977+0000","last_peered":"2024-03-22T12:57:04.172977+0000","last_clean":"2024-03-22T12:57:04.172977+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T12:57:04.172977+0000","last_undegraded":"2024-03-22T12:57:04.172977+0000","last_fullsized":"2024-03-22T12:57:04.172977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354578+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T12:57:04.354578+0000","last_peered":"2024-03-22T12:57:04.354578+0000","last_clean":"2024-03-22T12:57:04.354578+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T12:57:04.354578+0000","last_undegraded":"2024-03-22T12:57:04.354578+0000","last_fullsized":"2024-03-22T12:57:04.354578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946219+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T12:57:04.946219+0000","last_peered":"2024-03-22T12:57:04.946219+0000","last_clean":"2024-03-22T12:57:04.946219+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T12:57:04.946219+0000","last_undegraded":"2024-03-22T12:57:04.946219+0000","last_fullsized":"2024-03-22T12:57:04.946219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354584+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T12:57:04.354584+0000","last_peered":"2024-03-22T12:57:04.354584+0000","last_clean":"2024-03-22T12:57:04.354584+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T12:57:04.354584+0000","last_undegraded":"2024-03-22T12:57:04.354584+0000","last_fullsized":"2024-03-22T12:57:04.354584+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946376+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T12:57:04.946376+0000","last_peered":"2024-03-22T12:57:04.946376+0000","last_clean":"2024-03-22T12:57:04.946376+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T12:57:04.946376+0000","last_undegraded":"2024-03-22T12:57:04.946376+0000","last_fullsized":"2024-03-22T12:57:04.946376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171812+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T12:57:04.171812+0000","last_peered":"2024-03-22T12:57:04.171812+0000","last_clean":"2024-03-22T12:57:04.171812+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T12:57:04.171812+0000","last_undegraded":"2024-03-22T12:57:04.171812+0000","last_fullsized":"2024-03-22T12:57:04.171812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:14.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185524+0000","last_change":"2024-03-22T12:56:30.303254+0000","last_active":"2024-03-22T12:57:04.185524+0000","last_peered":"2024-03-22T12:57:04.185524+0000","last_clean":"2024-03-22T12:57:04.185524+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T12:57:04.185524+0000","last_undegraded":"2024-03-22T12:57:04.185524+0000","last_fullsized":"2024-03-22T12:57:04.185524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354841+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T12:57:04.354841+0000","last_peered":"2024-03-22T12:57:04.354841+0000","last_clean":"2024-03-22T12:57:04.354841+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T12:57:04.354841+0000","last_undegraded":"2024-03-22T12:57:04.354841+0000","last_fullsized":"2024-03-22T12:57:04.354841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":95,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":135711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":133109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509457,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":136445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":133311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509456,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":413696,"data_stored":723021,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":421888,"data_stored":724247,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":136411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":133241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T12:57:24.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T12:57:25.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.197+0000 7f6942e00640 1 Processor -- start 2024-03-22T12:57:25.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.197+0000 7f6942e00640 1 -- start start 2024-03-22T12:57:25.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.199+0000 7f6942e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f693c1589f0 0x7f693c15f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:25.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.199+0000 7f6942e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c0c7640 0x7f693c162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:25.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.199+0000 7f6942e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c158dd0 con 0x7f693c1628c0 2024-03-22T12:57:25.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.199+0000 7f6942e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693c161c10 con 0x7f693c1589f0 2024-03-22T12:57:25.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.199+0000 7f6942e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0c7b40 con 0x7f693c0c7640 2024-03-22T12:57:25.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.200+0000 7f6940e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f693c1628c0 0x7f693c06b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46794/0 (socket says 172.21.15.6:46794) 2024-03-22T12:57:25.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.200+0000 7f6940e00640 1 -- 172.21.15.6:0/3303975812 learned_addr learned my addr 172.21.15.6:0/3303975812 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:25.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.200+0000 7f693be00640 1 --2- 172.21.15.6:0/3303975812 >> v2:172.21.15.92:3300/0 conn(0x7f693c1589f0 0x7f693c15f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:25.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.200+0000 7f693b400640 1 --2- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c0c7640 0x7f693c162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.202+0000 7f693be00640 1 -- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c0c7640 msgr2=0x7f693c162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.202+0000 7f693be00640 1 --2- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c0c7640 0x7f693c162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.202+0000 7f693be00640 1 -- 172.21.15.6:0/3303975812 >> v1:172.21.15.6:6789/0 conn(0x7f693c1628c0 legacy=0x7f693c06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.202+0000 7f693be00640 1 -- 172.21.15.6:0/3303975812 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c0c6b10 con 0x7f693c1589f0 2024-03-22T12:57:25.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.205+0000 7f693be00640 1 --2- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c1589f0 0x7f693c15f0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f692c0097a0 tx=0x7f692c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3bd91f7f6945c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:25.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.207+0000 7f693aa00640 1 -- 172.21.15.6:0/3303975812 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c034070 con 0x7f693c1589f0 2024-03-22T12:57:25.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.209+0000 7f693aa00640 1 -- 172.21.15.6:0/3303975812 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c030610 con 0x7f693c1589f0 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.209+0000 7f6942e00640 1 -- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c1589f0 msgr2=0x7f693c15f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.209+0000 7f6942e00640 1 --2- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c1589f0 0x7f693c15f0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f692c0097a0 tx=0x7f692c02b960 comp rx=0 tx=0).stop 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.210+0000 7f6942e00640 1 -- 172.21.15.6:0/3303975812 shutdown_connections 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.210+0000 7f6942e00640 1 --2- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c0c7640 0x7f693c162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.210+0000 7f6942e00640 1 --2- 172.21.15.6:0/3303975812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c1589f0 0x7f693c15f0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.210+0000 7f6942e00640 1 -- 172.21.15.6:0/3303975812 >> 172.21.15.6:0/3303975812 conn(0x7f693c0cb930 msgr2=0x7f693c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.210+0000 7f6942e00640 1 -- 172.21.15.6:0/3303975812 shutdown_connections 2024-03-22T12:57:25.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.210+0000 7f6942e00640 1 -- 172.21.15.6:0/3303975812 wait complete. 2024-03-22T12:57:25.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.212+0000 7f6942e00640 1 Processor -- start 2024-03-22T12:57:25.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.212+0000 7f6942e00640 1 -- start start 2024-03-22T12:57:25.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.213+0000 7f6942e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0c7640 0x7f693c15d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:25.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.213+0000 7f6942e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f693c1589f0 0x7f693c160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:25.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.213+0000 7f6942e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c1628c0 0x7f693c1614a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:25.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.213+0000 7f6942e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c161e60 con 0x7f693c1589f0 2024-03-22T12:57:25.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.213+0000 7f6942e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0c70d0 con 0x7f693c0c7640 2024-03-22T12:57:25.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.213+0000 7f6942e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693c072380 con 0x7f693c1628c0 2024-03-22T12:57:25.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.213+0000 7f693b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f693c1589f0 0x7f693c160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:25.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.214+0000 7f6940e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c1628c0 0x7f693c1614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:25.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.214+0000 7f693be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0c7640 0x7f693c15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:25.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.214+0000 7f693b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f693c1589f0 0x7f693c160f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41336/0 (socket says 172.21.15.6:41336) 2024-03-22T12:57:25.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.214+0000 7f693b400640 1 -- 172.21.15.6:0/3043466660 learned_addr learned my addr 172.21.15.6:0/3043466660 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:25.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.218+0000 7f693be00640 1 -- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c1628c0 msgr2=0x7f693c1614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:25.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.218+0000 7f693be00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c1628c0 0x7f693c1614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.218+0000 7f693be00640 1 -- 172.21.15.6:0/3043466660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f693c1589f0 msgr2=0x7f693c160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:25.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.218+0000 7f693be00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f693c1589f0 0x7f693c160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.218+0000 7f693be00640 1 -- 172.21.15.6:0/3043466660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c1f45f0 con 0x7f693c0c7640 2024-03-22T12:57:25.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.220+0000 7f693be00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0c7640 0x7f693c15d530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f692c033000 tx=0x7f692c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:25.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.224+0000 7f6933e00640 1 -- 172.21.15.6:0/3043466660 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c048070 con 0x7f693c0c7640 2024-03-22T12:57:25.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.224+0000 7f6933e00640 1 -- 172.21.15.6:0/3043466660 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c030410 con 0x7f693c0c7640 2024-03-22T12:57:25.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.224+0000 7f6933e00640 1 -- 172.21.15.6:0/3043466660 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c034050 con 0x7f693c0c7640 2024-03-22T12:57:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.224+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c1f48d0 con 0x7f693c0c7640 2024-03-22T12:57:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.225+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c1f4e90 con 0x7f693c0c7640 2024-03-22T12:57:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.226+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c0756c0 con 0x7f693c0c7640 2024-03-22T12:57:25.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.262+0000 7f6933e00640 1 -- 172.21.15.6:0/3043466660 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f692c04b070 con 0x7f693c0c7640 2024-03-22T12:57:25.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.263+0000 7f6933e00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f691c06bc20 0x7f691c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:25.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.263+0000 7f693b400640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f691c06bc20 0x7f691c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:25.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.264+0000 7f693b400640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f691c06bc20 0x7f691c06e0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f693c14d930 tx=0x7f6928009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:25.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.266+0000 7f6933e00640 1 -- 172.21.15.6:0/3043466660 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f692c079250 con 0x7f693c0c7640 2024-03-22T12:57:25.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.575+0000 7f6933e00640 1 -- 172.21.15.6:0/3043466660 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f692c052020 con 0x7f693c0c7640 2024-03-22T12:57:25.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.795+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f693c0c7420 con 0x7f691c06bc20 2024-03-22T12:57:25.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.804+0000 7f6933e00640 1 -- 172.21.15.6:0/3043466660 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332745 (secure 0 0 0) 0x7f693c0c7420 con 0x7f691c06bc20 2024-03-22T12:57:25.807 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:25.809 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T12:57:25.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.809+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f691c06bc20 msgr2=0x7f691c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:25.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.809+0000 7f6942e00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f691c06bc20 0x7f691c06e0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f693c14d930 tx=0x7f6928009190 comp rx=0 tx=0).stop 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0c7640 msgr2=0x7f693c15d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0c7640 0x7f693c15d530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f692c033000 tx=0x7f692c0046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 shutdown_connections 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c1628c0 0x7f693c1614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f693c1589f0 0x7f693c160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f691c06bc20 0x7f691c06e0e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 --2- 172.21.15.6:0/3043466660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0c7640 0x7f693c15d530 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 >> 172.21.15.6:0/3043466660 conn(0x7f693c0cb930 msgr2=0x7f693c154ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 shutdown_connections 2024-03-22T12:57:25.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:25.810+0000 7f6942e00640 1 -- 172.21.15.6:0/3043466660 wait complete. 2024-03-22T12:57:25.826 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-22T12:57:25.702395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":366,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":298,"ondisk_log_size":298,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219008,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755712,"statfs":{"total":773094113280,"available":772869849088,"internally_reserved":0,"allocated":2949120,"data_stored":2255496,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5512,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003390"},"pg_stats":[{"pgid":"2.1f","version":"30'13","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.081693+0000","last_change":"2024-03-22T12:56:28.778294+0000","last_active":"2024-03-22T12:57:18.081693+0000","last_peered":"2024-03-22T12:57:18.081693+0000","last_clean":"2024-03-22T12:57:18.081693+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T12:57:18.081693+0000","last_undegraded":"2024-03-22T12:57:18.081693+0000","last_fullsized":"2024-03-22T12:57:18.081693+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174773+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T12:57:04.174773+0000","last_peered":"2024-03-22T12:57:04.174773+0000","last_clean":"2024-03-22T12:57:04.174773+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T12:57:04.174773+0000","last_undegraded":"2024-03-22T12:57:04.174773+0000","last_fullsized":"2024-03-22T12:57:04.174773+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173401+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T12:57:04.173401+0000","last_peered":"2024-03-22T12:57:04.173401+0000","last_clean":"2024-03-22T12:57:04.173401+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T12:57:04.173401+0000","last_undegraded":"2024-03-22T12:57:04.173401+0000","last_fullsized":"2024-03-22T12:57:04.173401+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:35:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.175013+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T12:57:04.175013+0000","last_peered":"2024-03-22T12:57:04.175013+0000","last_clean":"2024-03-22T12:57:04.175013+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T12:57:04.175013+0000","last_undegraded":"2024-03-22T12:57:04.175013+0000","last_fullsized":"2024-03-22T12:57:04.175013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172065+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T12:57:04.172065+0000","last_peered":"2024-03-22T12:57:04.172065+0000","last_clean":"2024-03-22T12:57:04.172065+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T12:57:04.172065+0000","last_undegraded":"2024-03-22T12:57:04.172065+0000","last_fullsized":"2024-03-22T12:57:04.172065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042070+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T12:57:05.042070+0000","last_peered":"2024-03-22T12:57:05.042070+0000","last_clean":"2024-03-22T12:57:05.042070+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T12:57:05.042070+0000","last_undegraded":"2024-03-22T12:57:05.042070+0000","last_fullsized":"2024-03-22T12:57:05.042070+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:40.477732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'11","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.094735+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T12:57:18.094735+0000","last_peered":"2024-03-22T12:57:18.094735+0000","last_clean":"2024-03-22T12:57:18.094735+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T12:57:18.094735+0000","last_undegraded":"2024-03-22T12:57:18.094735+0000","last_fullsized":"2024-03-22T12:57:18.094735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946408+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T12:57:04.946408+0000","last_peered":"2024-03-22T12:57:04.946408+0000","last_clean":"2024-03-22T12:57:04.946408+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T12:57:04.946408+0000","last_undegraded":"2024-03-22T12:57:04.946408+0000","last_fullsized":"2024-03-22T12:57:04.946408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042290+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T12:57:05.042290+0000","last_peered":"2024-03-22T12:57:05.042290+0000","last_clean":"2024-03-22T12:57:05.042290+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T12:57:05.042290+0000","last_undegraded":"2024-03-22T12:57:05.042290+0000","last_fullsized":"2024-03-22T12:57:05.042290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:29.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.093644+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T12:57:18.093644+0000","last_peered":"2024-03-22T12:57:18.093644+0000","last_clean":"2024-03-22T12:57:18.093644+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T12:57:18.093644+0000","last_undegraded":"2024-03-22T12:57:18.093644+0000","last_fullsized":"2024-03-22T12:57:18.093644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946567+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T12:57:04.946567+0000","last_peered":"2024-03-22T12:57:04.946567+0000","last_clean":"2024-03-22T12:57:04.946567+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T12:57:04.946567+0000","last_undegraded":"2024-03-22T12:57:04.946567+0000","last_fullsized":"2024-03-22T12:57:04.946567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042408+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T12:57:05.042408+0000","last_peered":"2024-03-22T12:57:05.042408+0000","last_clean":"2024-03-22T12:57:05.042408+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T12:57:05.042408+0000","last_undegraded":"2024-03-22T12:57:05.042408+0000","last_fullsized":"2024-03-22T12:57:05.042408+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247053+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T12:57:04.247053+0000","last_peered":"2024-03-22T12:57:04.247053+0000","last_clean":"2024-03-22T12:57:04.247053+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T12:57:04.247053+0000","last_undegraded":"2024-03-22T12:57:04.247053+0000","last_fullsized":"2024-03-22T12:57:04.247053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.305489+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T12:57:12.305489+0000","last_peered":"2024-03-22T12:57:12.305489+0000","last_clean":"2024-03-22T12:57:12.305489+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T12:57:12.305489+0000","last_undegraded":"2024-03-22T12:57:12.305489+0000","last_fullsized":"2024-03-22T12:57:12.305489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042319+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T12:57:05.042319+0000","last_peered":"2024-03-22T12:57:05.042319+0000","last_clean":"2024-03-22T12:57:05.042319+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T12:57:05.042319+0000","last_undegraded":"2024-03-22T12:57:05.042319+0000","last_fullsized":"2024-03-22T12:57:05.042319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:29.130909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174615+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T12:57:04.174615+0000","last_peered":"2024-03-22T12:57:04.174615+0000","last_clean":"2024-03-22T12:57:04.174615+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T12:57:04.174615+0000","last_undegraded":"2024-03-22T12:57:04.174615+0000","last_fullsized":"2024-03-22T12:57:04.174615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:17.014943+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T12:57:17.014943+0000","last_peered":"2024-03-22T12:57:17.014943+0000","last_clean":"2024-03-22T12:57:17.014943+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T12:57:17.014943+0000","last_undegraded":"2024-03-22T12:57:17.014943+0000","last_fullsized":"2024-03-22T12:57:17.014943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042296+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T12:57:05.042296+0000","last_peered":"2024-03-22T12:57:05.042296+0000","last_clean":"2024-03-22T12:57:05.042296+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T12:57:05.042296+0000","last_undegraded":"2024-03-22T12:57:05.042296+0000","last_fullsized":"2024-03-22T12:57:05.042296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:40.776689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173279+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T12:57:04.173279+0000","last_peered":"2024-03-22T12:57:04.173279+0000","last_clean":"2024-03-22T12:57:04.173279+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T12:57:04.173279+0000","last_undegraded":"2024-03-22T12:57:04.173279+0000","last_fullsized":"2024-03-22T12:57:04.173279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:17.986612+0000","last_change":"2024-03-22T12:56:28.757253+0000","last_active":"2024-03-22T12:57:17.986612+0000","last_peered":"2024-03-22T12:57:17.986612+0000","last_clean":"2024-03-22T12:57:17.986612+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T12:57:17.986612+0000","last_undegraded":"2024-03-22T12:57:17.986612+0000","last_fullsized":"2024-03-22T12:57:17.986612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:48.506816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185414+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T12:57:04.185414+0000","last_peered":"2024-03-22T12:57:04.185414+0000","last_clean":"2024-03-22T12:57:04.185414+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T12:57:04.185414+0000","last_undegraded":"2024-03-22T12:57:04.185414+0000","last_fullsized":"2024-03-22T12:57:04.185414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042646+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T12:57:05.042646+0000","last_peered":"2024-03-22T12:57:05.042646+0000","last_clean":"2024-03-22T12:57:05.042646+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T12:57:05.042646+0000","last_undegraded":"2024-03-22T12:57:05.042646+0000","last_fullsized":"2024-03-22T12:57:05.042646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:34.508555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.306479+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T12:57:12.306479+0000","last_peered":"2024-03-22T12:57:12.306479+0000","last_clean":"2024-03-22T12:57:12.306479+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T12:57:12.306479+0000","last_undegraded":"2024-03-22T12:57:12.306479+0000","last_fullsized":"2024-03-22T12:57:12.306479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042323+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T12:57:05.042323+0000","last_peered":"2024-03-22T12:57:05.042323+0000","last_clean":"2024-03-22T12:57:05.042323+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T12:57:05.042323+0000","last_undegraded":"2024-03-22T12:57:05.042323+0000","last_fullsized":"2024-03-22T12:57:05.042323+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:09.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172429+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T12:57:04.172429+0000","last_peered":"2024-03-22T12:57:04.172429+0000","last_clean":"2024-03-22T12:57:04.172429+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T12:57:04.172429+0000","last_undegraded":"2024-03-22T12:57:04.172429+0000","last_fullsized":"2024-03-22T12:57:04.172429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.093203+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T12:57:18.093203+0000","last_peered":"2024-03-22T12:57:18.093203+0000","last_clean":"2024-03-22T12:57:18.093203+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T12:57:18.093203+0000","last_undegraded":"2024-03-22T12:57:18.093203+0000","last_fullsized":"2024-03-22T12:57:18.093203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042166+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T12:57:05.042166+0000","last_peered":"2024-03-22T12:57:05.042166+0000","last_clean":"2024-03-22T12:57:05.042166+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T12:57:05.042166+0000","last_undegraded":"2024-03-22T12:57:05.042166+0000","last_fullsized":"2024-03-22T12:57:05.042166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:13.307688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176050+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T12:57:04.176050+0000","last_peered":"2024-03-22T12:57:04.176050+0000","last_clean":"2024-03-22T12:57:04.176050+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T12:57:04.176050+0000","last_undegraded":"2024-03-22T12:57:04.176050+0000","last_fullsized":"2024-03-22T12:57:04.176050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:06.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.315957+0000","last_change":"2024-03-22T12:56:28.767925+0000","last_active":"2024-03-22T12:57:16.315957+0000","last_peered":"2024-03-22T12:57:16.315957+0000","last_clean":"2024-03-22T12:57:16.315957+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T12:57:16.315957+0000","last_undegraded":"2024-03-22T12:57:16.315957+0000","last_fullsized":"2024-03-22T12:57:16.315957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172825+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T12:57:04.172825+0000","last_peered":"2024-03-22T12:57:04.172825+0000","last_clean":"2024-03-22T12:57:04.172825+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T12:57:04.172825+0000","last_undegraded":"2024-03-22T12:57:04.172825+0000","last_fullsized":"2024-03-22T12:57:04.172825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:40:09.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185111+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T12:57:04.185111+0000","last_peered":"2024-03-22T12:57:04.185111+0000","last_clean":"2024-03-22T12:57:04.185111+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T12:57:04.185111+0000","last_undegraded":"2024-03-22T12:57:04.185111+0000","last_fullsized":"2024-03-22T12:57:04.185111+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.308370+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T12:57:16.308370+0000","last_peered":"2024-03-22T12:57:16.308370+0000","last_clean":"2024-03-22T12:57:16.308370+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T12:57:16.308370+0000","last_undegraded":"2024-03-22T12:57:16.308370+0000","last_fullsized":"2024-03-22T12:57:16.308370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170966+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T12:57:04.170966+0000","last_peered":"2024-03-22T12:57:04.170966+0000","last_clean":"2024-03-22T12:57:04.170966+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T12:57:04.170966+0000","last_undegraded":"2024-03-22T12:57:04.170966+0000","last_fullsized":"2024-03-22T12:57:04.170966+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946629+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T12:57:04.946629+0000","last_peered":"2024-03-22T12:57:04.946629+0000","last_clean":"2024-03-22T12:57:04.946629+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T12:57:04.946629+0000","last_undegraded":"2024-03-22T12:57:04.946629+0000","last_fullsized":"2024-03-22T12:57:04.946629+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185241+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T12:57:04.185241+0000","last_peered":"2024-03-22T12:57:04.185241+0000","last_clean":"2024-03-22T12:57:04.185241+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T12:57:04.185241+0000","last_undegraded":"2024-03-22T12:57:04.185241+0000","last_fullsized":"2024-03-22T12:57:04.185241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045655+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T12:57:05.045655+0000","last_peered":"2024-03-22T12:57:05.045655+0000","last_clean":"2024-03-22T12:57:05.045655+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T12:57:05.045655+0000","last_undegraded":"2024-03-22T12:57:05.045655+0000","last_fullsized":"2024-03-22T12:57:05.045655+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:45.197847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042741+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T12:57:05.042741+0000","last_peered":"2024-03-22T12:57:05.042741+0000","last_clean":"2024-03-22T12:57:05.042741+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T12:57:05.042741+0000","last_undegraded":"2024-03-22T12:57:05.042741+0000","last_fullsized":"2024-03-22T12:57:05.042741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:23.839356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.088903+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T12:57:18.088903+0000","last_peered":"2024-03-22T12:57:18.088903+0000","last_clean":"2024-03-22T12:57:18.088903+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T12:57:18.088903+0000","last_undegraded":"2024-03-22T12:57:18.088903+0000","last_fullsized":"2024-03-22T12:57:18.088903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184845+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T12:57:04.184845+0000","last_peered":"2024-03-22T12:57:04.184845+0000","last_clean":"2024-03-22T12:57:04.184845+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T12:57:04.184845+0000","last_undegraded":"2024-03-22T12:57:04.184845+0000","last_fullsized":"2024-03-22T12:57:04.184845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184792+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T12:57:04.184792+0000","last_peered":"2024-03-22T12:57:04.184792+0000","last_clean":"2024-03-22T12:57:04.184792+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T12:57:04.184792+0000","last_undegraded":"2024-03-22T12:57:04.184792+0000","last_fullsized":"2024-03-22T12:57:04.184792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:14.483147+0000","last_change":"2024-03-22T12:56:28.763384+0000","last_active":"2024-03-22T12:57:14.483147+0000","last_peered":"2024-03-22T12:57:14.483147+0000","last_clean":"2024-03-22T12:57:14.483147+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T12:57:14.483147+0000","last_undegraded":"2024-03-22T12:57:14.483147+0000","last_fullsized":"2024-03-22T12:57:14.483147+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:15:45.463056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.952755+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T12:57:04.952755+0000","last_peered":"2024-03-22T12:57:04.952755+0000","last_clean":"2024-03-22T12:57:04.952755+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T12:57:04.952755+0000","last_undegraded":"2024-03-22T12:57:04.952755+0000","last_fullsized":"2024-03-22T12:57:04.952755+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:48.651995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.169659+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T12:57:04.169659+0000","last_peered":"2024-03-22T12:57:04.169659+0000","last_clean":"2024-03-22T12:57:04.169659+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T12:57:04.169659+0000","last_undegraded":"2024-03-22T12:57:04.169659+0000","last_fullsized":"2024-03-22T12:57:04.169659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:36:50.048241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:14.391765+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T12:57:14.391765+0000","last_peered":"2024-03-22T12:57:14.391765+0000","last_clean":"2024-03-22T12:57:14.391765+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T12:57:14.391765+0000","last_undegraded":"2024-03-22T12:57:14.391765+0000","last_fullsized":"2024-03-22T12:57:14.391765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172286+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T12:57:04.172286+0000","last_peered":"2024-03-22T12:57:04.172286+0000","last_clean":"2024-03-22T12:57:04.172286+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T12:57:04.172286+0000","last_undegraded":"2024-03-22T12:57:04.172286+0000","last_fullsized":"2024-03-22T12:57:04.172286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:29:31.400078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174048+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T12:57:04.174048+0000","last_peered":"2024-03-22T12:57:04.174048+0000","last_clean":"2024-03-22T12:57:04.174048+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T12:57:04.174048+0000","last_undegraded":"2024-03-22T12:57:04.174048+0000","last_fullsized":"2024-03-22T12:57:04.174048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":34,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.312656+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T12:57:16.312656+0000","last_peered":"2024-03-22T12:57:16.312656+0000","last_clean":"2024-03-22T12:57:16.312656+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T12:57:16.312656+0000","last_undegraded":"2024-03-22T12:57:16.312656+0000","last_fullsized":"2024-03-22T12:57:16.312656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354430+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T12:57:04.354430+0000","last_peered":"2024-03-22T12:57:04.354430+0000","last_clean":"2024-03-22T12:57:04.354430+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T12:57:04.354430+0000","last_undegraded":"2024-03-22T12:57:04.354430+0000","last_fullsized":"2024-03-22T12:57:04.354430+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247232+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T12:57:04.247232+0000","last_peered":"2024-03-22T12:57:04.247232+0000","last_clean":"2024-03-22T12:57:04.247232+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T12:57:04.247232+0000","last_undegraded":"2024-03-22T12:57:04.247232+0000","last_fullsized":"2024-03-22T12:57:04.247232+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.312265+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T12:57:16.312265+0000","last_peered":"2024-03-22T12:57:16.312265+0000","last_clean":"2024-03-22T12:57:16.312265+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T12:57:16.312265+0000","last_undegraded":"2024-03-22T12:57:16.312265+0000","last_fullsized":"2024-03-22T12:57:16.312265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247035+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T12:57:04.247035+0000","last_peered":"2024-03-22T12:57:04.247035+0000","last_clean":"2024-03-22T12:57:04.247035+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T12:57:04.247035+0000","last_undegraded":"2024-03-22T12:57:04.247035+0000","last_fullsized":"2024-03-22T12:57:04.247035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946466+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T12:57:04.946466+0000","last_peered":"2024-03-22T12:57:04.946466+0000","last_clean":"2024-03-22T12:57:04.946466+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T12:57:04.946466+0000","last_undegraded":"2024-03-22T12:57:04.946466+0000","last_fullsized":"2024-03-22T12:57:04.946466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.314152+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T12:57:16.314152+0000","last_peered":"2024-03-22T12:57:16.314152+0000","last_clean":"2024-03-22T12:57:16.314152+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T12:57:16.314152+0000","last_undegraded":"2024-03-22T12:57:16.314152+0000","last_fullsized":"2024-03-22T12:57:16.314152+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:22.329365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171343+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T12:57:04.171343+0000","last_peered":"2024-03-22T12:57:04.171343+0000","last_clean":"2024-03-22T12:57:04.171343+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T12:57:04.171343+0000","last_undegraded":"2024-03-22T12:57:04.171343+0000","last_fullsized":"2024-03-22T12:57:04.171343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355029+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T12:57:04.355029+0000","last_peered":"2024-03-22T12:57:04.355029+0000","last_clean":"2024-03-22T12:57:04.355029+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T12:57:04.355029+0000","last_undegraded":"2024-03-22T12:57:04.355029+0000","last_fullsized":"2024-03-22T12:57:04.355029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:17.985421+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T12:57:17.985421+0000","last_peered":"2024-03-22T12:57:17.985421+0000","last_clean":"2024-03-22T12:57:17.985421+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T12:57:17.985421+0000","last_undegraded":"2024-03-22T12:57:17.985421+0000","last_fullsized":"2024-03-22T12:57:17.985421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946109+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T12:57:04.946109+0000","last_peered":"2024-03-22T12:57:04.946109+0000","last_clean":"2024-03-22T12:57:04.946109+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T12:57:04.946109+0000","last_undegraded":"2024-03-22T12:57:04.946109+0000","last_fullsized":"2024-03-22T12:57:04.946109+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355037+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T12:57:04.355037+0000","last_peered":"2024-03-22T12:57:04.355037+0000","last_clean":"2024-03-22T12:57:04.355037+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T12:57:04.355037+0000","last_undegraded":"2024-03-22T12:57:04.355037+0000","last_fullsized":"2024-03-22T12:57:04.355037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042539+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T12:57:05.042539+0000","last_peered":"2024-03-22T12:57:05.042539+0000","last_clean":"2024-03-22T12:57:05.042539+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T12:57:05.042539+0000","last_undegraded":"2024-03-22T12:57:05.042539+0000","last_fullsized":"2024-03-22T12:57:05.042539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:24:46.021828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042059+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T12:57:05.042059+0000","last_peered":"2024-03-22T12:57:05.042059+0000","last_clean":"2024-03-22T12:57:05.042059+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T12:57:05.042059+0000","last_undegraded":"2024-03-22T12:57:05.042059+0000","last_fullsized":"2024-03-22T12:57:05.042059+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:38.316319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354615+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T12:57:04.354615+0000","last_peered":"2024-03-22T12:57:04.354615+0000","last_clean":"2024-03-22T12:57:04.354615+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T12:57:04.354615+0000","last_undegraded":"2024-03-22T12:57:04.354615+0000","last_fullsized":"2024-03-22T12:57:04.354615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'17","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.314563+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T12:57:16.314563+0000","last_peered":"2024-03-22T12:57:16.314563+0000","last_clean":"2024-03-22T12:57:16.314563+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T12:57:16.314563+0000","last_undegraded":"2024-03-22T12:57:16.314563+0000","last_fullsized":"2024-03-22T12:57:16.314563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:20:57.722062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172194+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T12:57:04.172194+0000","last_peered":"2024-03-22T12:57:04.172194+0000","last_clean":"2024-03-22T12:57:04.172194+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T12:57:04.172194+0000","last_undegraded":"2024-03-22T12:57:04.172194+0000","last_fullsized":"2024-03-22T12:57:04.172194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247088+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T12:57:04.247088+0000","last_peered":"2024-03-22T12:57:04.247088+0000","last_clean":"2024-03-22T12:57:04.247088+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T12:57:04.247088+0000","last_undegraded":"2024-03-22T12:57:04.247088+0000","last_fullsized":"2024-03-22T12:57:04.247088+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.302102+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T12:57:16.302102+0000","last_peered":"2024-03-22T12:57:16.302102+0000","last_clean":"2024-03-22T12:57:16.302102+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T12:57:16.302102+0000","last_undegraded":"2024-03-22T12:57:16.302102+0000","last_fullsized":"2024-03-22T12:57:16.302102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.176522+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T12:57:04.176522+0000","last_peered":"2024-03-22T12:57:04.176522+0000","last_clean":"2024-03-22T12:57:04.176522+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T12:57:04.176522+0000","last_undegraded":"2024-03-22T12:57:04.176522+0000","last_fullsized":"2024-03-22T12:57:04.176522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:51.788457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354739+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T12:57:04.354739+0000","last_peered":"2024-03-22T12:57:04.354739+0000","last_clean":"2024-03-22T12:57:04.354739+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T12:57:04.354739+0000","last_undegraded":"2024-03-22T12:57:04.354739+0000","last_fullsized":"2024-03-22T12:57:04.354739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.310176+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T12:57:16.310176+0000","last_peered":"2024-03-22T12:57:16.310176+0000","last_clean":"2024-03-22T12:57:16.310176+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T12:57:16.310176+0000","last_undegraded":"2024-03-22T12:57:16.310176+0000","last_fullsized":"2024-03-22T12:57:16.310176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.949816+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T12:57:04.949816+0000","last_peered":"2024-03-22T12:57:04.949816+0000","last_clean":"2024-03-22T12:57:04.949816+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T12:57:04.949816+0000","last_undegraded":"2024-03-22T12:57:04.949816+0000","last_fullsized":"2024-03-22T12:57:04.949816+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:08:12.860933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170220+0000","last_change":"2024-03-22T12:56:30.299969+0000","last_active":"2024-03-22T12:57:04.170220+0000","last_peered":"2024-03-22T12:57:04.170220+0000","last_clean":"2024-03-22T12:57:04.170220+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T12:57:04.170220+0000","last_undegraded":"2024-03-22T12:57:04.170220+0000","last_fullsized":"2024-03-22T12:57:04.170220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'9","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.080641+0000","last_change":"2024-03-22T12:56:28.777142+0000","last_active":"2024-03-22T12:57:18.080641+0000","last_peered":"2024-03-22T12:57:18.080641+0000","last_clean":"2024-03-22T12:57:18.080641+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T12:57:18.080641+0000","last_undegraded":"2024-03-22T12:57:18.080641+0000","last_fullsized":"2024-03-22T12:57:18.080641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:49.411808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946528+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T12:57:04.946528+0000","last_peered":"2024-03-22T12:57:04.946528+0000","last_clean":"2024-03-22T12:57:04.946528+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T12:57:04.946528+0000","last_undegraded":"2024-03-22T12:57:04.946528+0000","last_fullsized":"2024-03-22T12:57:04.946528+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354901+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T12:57:04.354901+0000","last_peered":"2024-03-22T12:57:04.354901+0000","last_clean":"2024-03-22T12:57:04.354901+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T12:57:04.354901+0000","last_undegraded":"2024-03-22T12:57:04.354901+0000","last_fullsized":"2024-03-22T12:57:04.354901+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'14","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.092975+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T12:57:18.092975+0000","last_peered":"2024-03-22T12:57:18.092975+0000","last_clean":"2024-03-22T12:57:18.092975+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T12:57:18.092975+0000","last_undegraded":"2024-03-22T12:57:18.092975+0000","last_fullsized":"2024-03-22T12:57:18.092975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184655+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T12:57:04.184655+0000","last_peered":"2024-03-22T12:57:04.184655+0000","last_clean":"2024-03-22T12:57:04.184655+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T12:57:04.184655+0000","last_undegraded":"2024-03-22T12:57:04.184655+0000","last_fullsized":"2024-03-22T12:57:04.184655+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042567+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T12:57:05.042567+0000","last_peered":"2024-03-22T12:57:05.042567+0000","last_clean":"2024-03-22T12:57:05.042567+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T12:57:05.042567+0000","last_undegraded":"2024-03-22T12:57:05.042567+0000","last_fullsized":"2024-03-22T12:57:05.042567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:49.776407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.413187+0000","last_change":"2024-03-22T12:56:28.769041+0000","last_active":"2024-03-22T12:57:12.413187+0000","last_peered":"2024-03-22T12:57:12.413187+0000","last_clean":"2024-03-22T12:57:12.413187+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T12:57:12.413187+0000","last_undegraded":"2024-03-22T12:57:12.413187+0000","last_fullsized":"2024-03-22T12:57:12.413187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042461+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T12:57:05.042461+0000","last_peered":"2024-03-22T12:57:05.042461+0000","last_clean":"2024-03-22T12:57:05.042461+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T12:57:05.042461+0000","last_undegraded":"2024-03-22T12:57:05.042461+0000","last_fullsized":"2024-03-22T12:57:05.042461+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:56.815720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172275+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T12:57:04.172275+0000","last_peered":"2024-03-22T12:57:04.172275+0000","last_clean":"2024-03-22T12:57:04.172275+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T12:57:04.172275+0000","last_undegraded":"2024-03-22T12:57:04.172275+0000","last_fullsized":"2024-03-22T12:57:04.172275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:18.085201+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T12:57:18.085201+0000","last_peered":"2024-03-22T12:57:18.085201+0000","last_clean":"2024-03-22T12:57:18.085201+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T12:57:18.085201+0000","last_undegraded":"2024-03-22T12:57:18.085201+0000","last_fullsized":"2024-03-22T12:57:18.085201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173079+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T12:57:04.173079+0000","last_peered":"2024-03-22T12:57:04.173079+0000","last_clean":"2024-03-22T12:57:04.173079+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T12:57:04.173079+0000","last_undegraded":"2024-03-22T12:57:04.173079+0000","last_fullsized":"2024-03-22T12:57:04.173079+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:06:47.716384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946555+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T12:57:04.946555+0000","last_peered":"2024-03-22T12:57:04.946555+0000","last_clean":"2024-03-22T12:57:04.946555+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T12:57:04.946555+0000","last_undegraded":"2024-03-22T12:57:04.946555+0000","last_fullsized":"2024-03-22T12:57:04.946555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'4","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.310332+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T12:57:12.310332+0000","last_peered":"2024-03-22T12:57:12.310332+0000","last_clean":"2024-03-22T12:57:12.310332+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T12:57:12.310332+0000","last_undegraded":"2024-03-22T12:57:12.310332+0000","last_fullsized":"2024-03-22T12:57:12.310332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174378+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T12:57:04.174378+0000","last_peered":"2024-03-22T12:57:04.174378+0000","last_clean":"2024-03-22T12:57:04.174378+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T12:57:04.174378+0000","last_undegraded":"2024-03-22T12:57:04.174378+0000","last_fullsized":"2024-03-22T12:57:04.174378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172090+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T12:57:04.172090+0000","last_peered":"2024-03-22T12:57:04.172090+0000","last_clean":"2024-03-22T12:57:04.172090+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T12:57:04.172090+0000","last_undegraded":"2024-03-22T12:57:04.172090+0000","last_fullsized":"2024-03-22T12:57:04.172090+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184400+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T12:57:04.184400+0000","last_peered":"2024-03-22T12:57:04.184400+0000","last_clean":"2024-03-22T12:57:04.184400+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T12:57:04.184400+0000","last_undegraded":"2024-03-22T12:57:04.184400+0000","last_fullsized":"2024-03-22T12:57:04.184400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.307298+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T12:57:16.307298+0000","last_peered":"2024-03-22T12:57:16.307298+0000","last_clean":"2024-03-22T12:57:16.307298+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T12:57:16.307298+0000","last_undegraded":"2024-03-22T12:57:16.307298+0000","last_fullsized":"2024-03-22T12:57:16.307298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247413+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T12:57:04.247413+0000","last_peered":"2024-03-22T12:57:04.247413+0000","last_clean":"2024-03-22T12:57:04.247413+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T12:57:04.247413+0000","last_undegraded":"2024-03-22T12:57:04.247413+0000","last_fullsized":"2024-03-22T12:57:04.247413+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.173181+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T12:57:04.173181+0000","last_peered":"2024-03-22T12:57:04.173181+0000","last_clean":"2024-03-22T12:57:04.173181+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T12:57:04.173181+0000","last_undegraded":"2024-03-22T12:57:04.173181+0000","last_fullsized":"2024-03-22T12:57:04.173181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'3","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:14.480598+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T12:57:14.480598+0000","last_peered":"2024-03-22T12:57:14.480598+0000","last_clean":"2024-03-22T12:57:14.480598+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T12:57:14.480598+0000","last_undegraded":"2024-03-22T12:57:14.480598+0000","last_fullsized":"2024-03-22T12:57:14.480598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:31:41.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'7","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:16.304889+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T12:57:16.304889+0000","last_peered":"2024-03-22T12:57:16.304889+0000","last_clean":"2024-03-22T12:57:16.304889+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T12:57:16.304889+0000","last_undegraded":"2024-03-22T12:57:16.304889+0000","last_fullsized":"2024-03-22T12:57:16.304889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.047312+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T12:57:05.047312+0000","last_peered":"2024-03-22T12:57:05.047312+0000","last_clean":"2024-03-22T12:57:05.047312+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T12:57:05.047312+0000","last_undegraded":"2024-03-22T12:57:05.047312+0000","last_fullsized":"2024-03-22T12:57:05.047312+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.180347+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T12:57:04.180347+0000","last_peered":"2024-03-22T12:57:04.180347+0000","last_clean":"2024-03-22T12:57:04.180347+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T12:57:04.180347+0000","last_undegraded":"2024-03-22T12:57:04.180347+0000","last_fullsized":"2024-03-22T12:57:04.180347+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:43.256977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":33,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:12.481695+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T12:57:12.481695+0000","last_peered":"2024-03-22T12:57:12.481695+0000","last_clean":"2024-03-22T12:57:12.481695+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T12:57:12.481695+0000","last_undegraded":"2024-03-22T12:57:12.481695+0000","last_fullsized":"2024-03-22T12:57:12.481695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042643+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T12:57:05.042643+0000","last_peered":"2024-03-22T12:57:05.042643+0000","last_clean":"2024-03-22T12:57:05.042643+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T12:57:05.042643+0000","last_undegraded":"2024-03-22T12:57:05.042643+0000","last_fullsized":"2024-03-22T12:57:05.042643+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:48.882251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174116+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T12:57:04.174116+0000","last_peered":"2024-03-22T12:57:04.174116+0000","last_clean":"2024-03-22T12:57:04.174116+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T12:57:04.174116+0000","last_undegraded":"2024-03-22T12:57:04.174116+0000","last_fullsized":"2024-03-22T12:57:04.174116+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247032+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T12:57:04.247032+0000","last_peered":"2024-03-22T12:57:04.247032+0000","last_clean":"2024-03-22T12:57:04.247032+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T12:57:04.247032+0000","last_undegraded":"2024-03-22T12:57:04.247032+0000","last_fullsized":"2024-03-22T12:57:04.247032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.188189+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T12:57:04.188189+0000","last_peered":"2024-03-22T12:57:04.188189+0000","last_clean":"2024-03-22T12:57:04.188189+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T12:57:04.188189+0000","last_undegraded":"2024-03-22T12:57:04.188189+0000","last_fullsized":"2024-03-22T12:57:04.188189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946357+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T12:57:04.946357+0000","last_peered":"2024-03-22T12:57:04.946357+0000","last_clean":"2024-03-22T12:57:04.946357+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T12:57:04.946357+0000","last_undegraded":"2024-03-22T12:57:04.946357+0000","last_fullsized":"2024-03-22T12:57:04.946357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171397+0000","last_change":"2024-03-22T12:56:30.300244+0000","last_active":"2024-03-22T12:57:04.171397+0000","last_peered":"2024-03-22T12:57:04.171397+0000","last_clean":"2024-03-22T12:57:04.171397+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T12:57:04.171397+0000","last_undegraded":"2024-03-22T12:57:04.171397+0000","last_fullsized":"2024-03-22T12:57:04.171397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174764+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T12:57:04.174764+0000","last_peered":"2024-03-22T12:57:04.174764+0000","last_clean":"2024-03-22T12:57:04.174764+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T12:57:04.174764+0000","last_undegraded":"2024-03-22T12:57:04.174764+0000","last_fullsized":"2024-03-22T12:57:04.174764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185804+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T12:57:04.185804+0000","last_peered":"2024-03-22T12:57:04.185804+0000","last_clean":"2024-03-22T12:57:04.185804+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T12:57:04.185804+0000","last_undegraded":"2024-03-22T12:57:04.185804+0000","last_fullsized":"2024-03-22T12:57:04.185804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247526+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T12:57:04.247526+0000","last_peered":"2024-03-22T12:57:04.247526+0000","last_clean":"2024-03-22T12:57:04.247526+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T12:57:04.247526+0000","last_undegraded":"2024-03-22T12:57:04.247526+0000","last_fullsized":"2024-03-22T12:57:04.247526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.175763+0000","last_change":"2024-03-22T12:56:30.308903+0000","last_active":"2024-03-22T12:57:04.175763+0000","last_peered":"2024-03-22T12:57:04.175763+0000","last_clean":"2024-03-22T12:57:04.175763+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T12:57:04.175763+0000","last_undegraded":"2024-03-22T12:57:04.175763+0000","last_fullsized":"2024-03-22T12:57:04.175763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185929+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T12:57:04.185929+0000","last_peered":"2024-03-22T12:57:04.185929+0000","last_clean":"2024-03-22T12:57:04.185929+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T12:57:04.185929+0000","last_undegraded":"2024-03-22T12:57:04.185929+0000","last_fullsized":"2024-03-22T12:57:04.185929+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:13:57.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.045995+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T12:57:05.045995+0000","last_peered":"2024-03-22T12:57:05.045995+0000","last_clean":"2024-03-22T12:57:05.045995+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T12:57:05.045995+0000","last_undegraded":"2024-03-22T12:57:05.045995+0000","last_fullsized":"2024-03-22T12:57:05.045995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:19.797978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946253+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T12:57:04.946253+0000","last_peered":"2024-03-22T12:57:04.946253+0000","last_clean":"2024-03-22T12:57:04.946253+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T12:57:04.946253+0000","last_undegraded":"2024-03-22T12:57:04.946253+0000","last_fullsized":"2024-03-22T12:57:04.946253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171180+0000","last_change":"2024-03-22T12:56:30.299214+0000","last_active":"2024-03-22T12:57:04.171180+0000","last_peered":"2024-03-22T12:57:04.171180+0000","last_clean":"2024-03-22T12:57:04.171180+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T12:57:04.171180+0000","last_undegraded":"2024-03-22T12:57:04.171180+0000","last_fullsized":"2024-03-22T12:57:04.171180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:35.356723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247391+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T12:57:04.247391+0000","last_peered":"2024-03-22T12:57:04.247391+0000","last_clean":"2024-03-22T12:57:04.247391+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T12:57:04.247391+0000","last_undegraded":"2024-03-22T12:57:04.247391+0000","last_fullsized":"2024-03-22T12:57:04.247391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.189179+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T12:57:04.189179+0000","last_peered":"2024-03-22T12:57:04.189179+0000","last_clean":"2024-03-22T12:57:04.189179+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T12:57:04.189179+0000","last_undegraded":"2024-03-22T12:57:04.189179+0000","last_fullsized":"2024-03-22T12:57:04.189179+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:57:33.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.182435+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T12:57:04.182435+0000","last_peered":"2024-03-22T12:57:04.182435+0000","last_clean":"2024-03-22T12:57:04.182435+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T12:57:04.182435+0000","last_undegraded":"2024-03-22T12:57:04.182435+0000","last_fullsized":"2024-03-22T12:57:04.182435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:38.987151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946819+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T12:57:04.946819+0000","last_peered":"2024-03-22T12:57:04.946819+0000","last_clean":"2024-03-22T12:57:04.946819+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T12:57:04.946819+0000","last_undegraded":"2024-03-22T12:57:04.946819+0000","last_fullsized":"2024-03-22T12:57:04.946819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946220+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T12:57:04.946220+0000","last_peered":"2024-03-22T12:57:04.946220+0000","last_clean":"2024-03-22T12:57:04.946220+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T12:57:04.946220+0000","last_undegraded":"2024-03-22T12:57:04.946220+0000","last_fullsized":"2024-03-22T12:57:04.946220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.184477+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T12:57:04.184477+0000","last_peered":"2024-03-22T12:57:04.184477+0000","last_clean":"2024-03-22T12:57:04.184477+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T12:57:04.184477+0000","last_undegraded":"2024-03-22T12:57:04.184477+0000","last_fullsized":"2024-03-22T12:57:04.184477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.170174+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T12:57:04.170174+0000","last_peered":"2024-03-22T12:57:04.170174+0000","last_clean":"2024-03-22T12:57:04.170174+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T12:57:04.170174+0000","last_undegraded":"2024-03-22T12:57:04.170174+0000","last_fullsized":"2024-03-22T12:57:04.170174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.174348+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T12:57:04.174348+0000","last_peered":"2024-03-22T12:57:04.174348+0000","last_clean":"2024-03-22T12:57:04.174348+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T12:57:04.174348+0000","last_undegraded":"2024-03-22T12:57:04.174348+0000","last_fullsized":"2024-03-22T12:57:04.174348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:05.042563+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T12:57:05.042563+0000","last_peered":"2024-03-22T12:57:05.042563+0000","last_clean":"2024-03-22T12:57:05.042563+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T12:57:05.042563+0000","last_undegraded":"2024-03-22T12:57:05.042563+0000","last_fullsized":"2024-03-22T12:57:05.042563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:23.598956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.251063+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T12:57:04.251063+0000","last_peered":"2024-03-22T12:57:04.251063+0000","last_clean":"2024-03-22T12:57:04.251063+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T12:57:04.251063+0000","last_undegraded":"2024-03-22T12:57:04.251063+0000","last_fullsized":"2024-03-22T12:57:04.251063+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.247414+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T12:57:04.247414+0000","last_peered":"2024-03-22T12:57:04.247414+0000","last_clean":"2024-03-22T12:57:04.247414+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T12:57:04.247414+0000","last_undegraded":"2024-03-22T12:57:04.247414+0000","last_fullsized":"2024-03-22T12:57:04.247414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.355010+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T12:57:04.355010+0000","last_peered":"2024-03-22T12:57:04.355010+0000","last_clean":"2024-03-22T12:57:04.355010+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T12:57:04.355010+0000","last_undegraded":"2024-03-22T12:57:04.355010+0000","last_fullsized":"2024-03-22T12:57:04.355010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.172977+0000","last_change":"2024-03-22T12:56:30.309258+0000","last_active":"2024-03-22T12:57:04.172977+0000","last_peered":"2024-03-22T12:57:04.172977+0000","last_clean":"2024-03-22T12:57:04.172977+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T12:57:04.172977+0000","last_undegraded":"2024-03-22T12:57:04.172977+0000","last_fullsized":"2024-03-22T12:57:04.172977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354578+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T12:57:04.354578+0000","last_peered":"2024-03-22T12:57:04.354578+0000","last_clean":"2024-03-22T12:57:04.354578+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T12:57:04.354578+0000","last_undegraded":"2024-03-22T12:57:04.354578+0000","last_fullsized":"2024-03-22T12:57:04.354578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946219+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T12:57:04.946219+0000","last_peered":"2024-03-22T12:57:04.946219+0000","last_clean":"2024-03-22T12:57:04.946219+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T12:57:04.946219+0000","last_undegraded":"2024-03-22T12:57:04.946219+0000","last_fullsized":"2024-03-22T12:57:04.946219+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354584+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T12:57:04.354584+0000","last_peered":"2024-03-22T12:57:04.354584+0000","last_clean":"2024-03-22T12:57:04.354584+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T12:57:04.354584+0000","last_undegraded":"2024-03-22T12:57:04.354584+0000","last_fullsized":"2024-03-22T12:57:04.354584+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.946376+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T12:57:04.946376+0000","last_peered":"2024-03-22T12:57:04.946376+0000","last_clean":"2024-03-22T12:57:04.946376+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T12:57:04.946376+0000","last_undegraded":"2024-03-22T12:57:04.946376+0000","last_fullsized":"2024-03-22T12:57:04.946376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.171812+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T12:57:04.171812+0000","last_peered":"2024-03-22T12:57:04.171812+0000","last_clean":"2024-03-22T12:57:04.171812+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T12:57:04.171812+0000","last_undegraded":"2024-03-22T12:57:04.171812+0000","last_fullsized":"2024-03-22T12:57:04.171812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:14.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.185524+0000","last_change":"2024-03-22T12:56:30.303254+0000","last_active":"2024-03-22T12:57:04.185524+0000","last_peered":"2024-03-22T12:57:04.185524+0000","last_clean":"2024-03-22T12:57:04.185524+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T12:57:04.185524+0000","last_undegraded":"2024-03-22T12:57:04.185524+0000","last_fullsized":"2024-03-22T12:57:04.185524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-22T12:57:04.354841+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T12:57:04.354841+0000","last_peered":"2024-03-22T12:57:04.354841+0000","last_clean":"2024-03-22T12:57:04.354841+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T12:57:04.354841+0000","last_undegraded":"2024-03-22T12:57:04.354841+0000","last_fullsized":"2024-03-22T12:57:04.354841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":207,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":135711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509458,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":133109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509458,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":136445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509458,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":133311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":413696,"data_stored":723021,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":421888,"data_stored":724247,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":136411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":133241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T12:57:25.830 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-22T12:57:25.830 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-22T12:57:25.830 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-22T12:57:25.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-22T12:57:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.059+0000 7fe696800640 1 Processor -- start 2024-03-22T12:57:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.059+0000 7fe696800640 1 -- start start 2024-03-22T12:57:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.060+0000 7fe696800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6900cc380 0x7fe69006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.060+0000 7fe696800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe69015ae30 0x7fe6900cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.060+0000 7fe696800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69015b260 con 0x7fe6900ce8a0 2024-03-22T12:57:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.060+0000 7fe696800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6900cee00 con 0x7fe69015ae30 2024-03-22T12:57:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.060+0000 7fe696800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6900cf0c0 con 0x7fe6900cc380 2024-03-22T12:57:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.061+0000 7fe695400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6900cc380 0x7fe69006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.061+0000 7fe695e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe69015ae30 0x7fe6900cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.062+0000 7fe695400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6900cc380 0x7fe69006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46092/0 (socket says 172.21.15.6:46092) 2024-03-22T12:57:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.062+0000 7fe695400640 1 -- 172.21.15.6:0/3282589046 learned_addr learned my addr 172.21.15.6:0/3282589046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.065+0000 7fe695400640 1 -- 172.21.15.6:0/3282589046 >> v2:172.21.15.6:3300/0 conn(0x7fe69015ae30 msgr2=0x7fe6900cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.065+0000 7fe695400640 1 --2- 172.21.15.6:0/3282589046 >> v2:172.21.15.6:3300/0 conn(0x7fe69015ae30 0x7fe6900cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.065+0000 7fe695400640 1 -- 172.21.15.6:0/3282589046 >> v1:172.21.15.92:6789/0 conn(0x7fe6900ce8a0 legacy=0x7fe6900cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.065+0000 7fe695400640 1 -- 172.21.15.6:0/3282589046 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900cf3a0 con 0x7fe6900cc380 2024-03-22T12:57:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.068+0000 7fe695400640 1 --2- 172.21.15.6:0/3282589046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6900cc380 0x7fe69006c230 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe684009870 tx=0x7fe68402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25628d5c3cd5243 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.072+0000 7fe68fe00640 1 -- 172.21.15.6:0/3282589046 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684035020 con 0x7fe6900cc380 2024-03-22T12:57:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.072+0000 7fe68fe00640 1 -- 172.21.15.6:0/3282589046 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe684031640 con 0x7fe6900cc380 2024-03-22T12:57:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.072+0000 7fe68fe00640 1 -- 172.21.15.6:0/3282589046 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68402d040 con 0x7fe6900cc380 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 -- 172.21.15.6:0/3282589046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6900cc380 msgr2=0x7fe69006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 --2- 172.21.15.6:0/3282589046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6900cc380 0x7fe69006c230 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe684009870 tx=0x7fe68402b8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 -- 172.21.15.6:0/3282589046 shutdown_connections 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 --2- 172.21.15.6:0/3282589046 >> v2:172.21.15.6:3300/0 conn(0x7fe69015ae30 0x7fe6900cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 --2- 172.21.15.6:0/3282589046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6900cc380 0x7fe69006c230 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 -- 172.21.15.6:0/3282589046 >> 172.21.15.6:0/3282589046 conn(0x7fe6900b1480 msgr2=0x7fe6900b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 -- 172.21.15.6:0/3282589046 shutdown_connections 2024-03-22T12:57:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.073+0000 7fe696800640 1 -- 172.21.15.6:0/3282589046 wait complete. 2024-03-22T12:57:26.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.076+0000 7fe696800640 1 Processor -- start 2024-03-22T12:57:26.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.076+0000 7fe696800640 1 -- start start 2024-03-22T12:57:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.077+0000 7fe696800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6900cc380 0x7fe690144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.077+0000 7fe696800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6900ce8a0 0x7fe6901f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.077+0000 7fe696800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe69015ae30 0x7fe690155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.078+0000 7fe696800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69015a910 con 0x7fe6900cc380 2024-03-22T12:57:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.078+0000 7fe696800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69015a630 con 0x7fe6900ce8a0 2024-03-22T12:57:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.078+0000 7fe696800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6901549d0 con 0x7fe69015ae30 2024-03-22T12:57:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.078+0000 7fe694a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6900ce8a0 0x7fe6901f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.078+0000 7fe695e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe69015ae30 0x7fe690155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.079+0000 7fe694a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6900ce8a0 0x7fe6901f4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50316/0 (socket says 172.21.15.6:50316) 2024-03-22T12:57:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.079+0000 7fe694a00640 1 -- 172.21.15.6:0/1581468075 learned_addr learned my addr 172.21.15.6:0/1581468075 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.084+0000 7fe694a00640 1 -- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe69015ae30 msgr2=0x7fe690155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.084+0000 7fe694a00640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe69015ae30 0x7fe690155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.084+0000 7fe694a00640 1 -- 172.21.15.6:0/1581468075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6900cc380 msgr2=0x7fe690144b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.084+0000 7fe694a00640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6900cc380 0x7fe690144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.084+0000 7fe694a00640 1 -- 172.21.15.6:0/1581468075 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6901f4ab0 con 0x7fe6900ce8a0 2024-03-22T12:57:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.088+0000 7fe694a00640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6900ce8a0 0x7fe6901f4180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00c430 tx=0x7fe67c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.093+0000 7fe68d600640 1 -- 172.21.15.6:0/1581468075 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c017070 con 0x7fe6900ce8a0 2024-03-22T12:57:26.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.093+0000 7fe68d600640 1 -- 172.21.15.6:0/1581468075 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c0035f0 con 0x7fe6900ce8a0 2024-03-22T12:57:26.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.093+0000 7fe68d600640 1 -- 172.21.15.6:0/1581468075 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c0039e0 con 0x7fe6900ce8a0 2024-03-22T12:57:26.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.093+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe690156e80 con 0x7fe6900ce8a0 2024-03-22T12:57:26.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.093+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe690154e00 con 0x7fe6900ce8a0 2024-03-22T12:57:26.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.095+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe690075650 con 0x7fe6900ce8a0 2024-03-22T12:57:26.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.126+0000 7fe68d600640 1 -- 172.21.15.6:0/1581468075 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe67c01a070 con 0x7fe6900ce8a0 2024-03-22T12:57:26.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.126+0000 7fe68d600640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe67006bc20 0x7fe67006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.127+0000 7fe695400640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe67006bc20 0x7fe67006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.128+0000 7fe695400640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe67006bc20 0x7fe67006e0e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe684036000 tx=0x7fe684030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:26.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.130+0000 7fe68d600640 1 -- 172.21.15.6:0/1581468075 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fe67c056bd0 con 0x7fe6900ce8a0 2024-03-22T12:57:26.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.417+0000 7fe68d600640 1 -- 172.21.15.6:0/1581468075 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe67c058b30 con 0x7fe6900ce8a0 2024-03-22T12:57:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.668+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe690158d80 con 0x7fe6900ce8a0 2024-03-22T12:57:26.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.676+0000 7fe68d600640 1 -- 172.21.15.6:0/1581468075 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe67c05cbd0 con 0x7fe6900ce8a0 2024-03-22T12:57:26.676 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:26.676 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe67006bc20 msgr2=0x7fe67006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe67006bc20 0x7fe67006e0e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe684036000 tx=0x7fe684030000 comp rx=0 tx=0).stop 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6900ce8a0 msgr2=0x7fe6901f4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6900ce8a0 0x7fe6901f4180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00c430 tx=0x7fe67c00c8f0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 shutdown_connections 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe69015ae30 0x7fe690155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe67006bc20 0x7fe67006e0e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6900ce8a0 0x7fe6901f4180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.678+0000 7fe696800640 1 --2- 172.21.15.6:0/1581468075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6900cc380 0x7fe690144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.679+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 >> 172.21.15.6:0/1581468075 conn(0x7fe6900b1480 msgr2=0x7fe690153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.679+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 shutdown_connections 2024-03-22T12:57:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.679+0000 7fe696800640 1 -- 172.21.15.6:0/1581468075 wait complete. 2024-03-22T12:57:26.697 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-22T12:57:26.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:26.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.884+0000 7f10e9c00640 1 Processor -- start 2024-03-22T12:57:26.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.885+0000 7f10e9c00640 1 -- start start 2024-03-22T12:57:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.886+0000 7f10e9c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f10e40ccad0 0x7f10e406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.886+0000 7f10e9c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e406c7a0 con 0x7f10e40ca520 2024-03-22T12:57:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.886+0000 7f10e9c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e406ca80 con 0x7f10e415b0e0 2024-03-22T12:57:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.886+0000 7f10e9c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e40cd000 con 0x7f10e40ccad0 2024-03-22T12:57:26.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.888+0000 7f10e2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f10e40ca520 0x7f10e40cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46808/0 (socket says 172.21.15.6:46808) 2024-03-22T12:57:26.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.888+0000 7f10e2a00640 1 -- 172.21.15.6:0/1194000376 learned_addr learned my addr 172.21.15.6:0/1194000376 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.889+0000 7f10dbe00640 1 --2- 172.21.15.6:0/1194000376 >> v2:172.21.15.92:3300/0 conn(0x7f10e40ccad0 0x7f10e406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.893+0000 7f10e2000640 1 -- 172.21.15.6:0/1194000376 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185824457 0 0) 0x7f10e406c7a0 con 0x7f10e40ca520 2024-03-22T12:57:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.893+0000 7f10e2000640 1 -- 172.21.15.6:0/1194000376 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c8003520 con 0x7f10e40ca520 2024-03-22T12:57:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.896+0000 7f10dbe00640 1 -- 172.21.15.6:0/1194000376 >> v1:172.21.15.92:6789/0 conn(0x7f10e415b0e0 legacy=0x7f10e4164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.896+0000 7f10dbe00640 1 -- 172.21.15.6:0/1194000376 >> v1:172.21.15.6:6789/0 conn(0x7f10e40ca520 legacy=0x7f10e40cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.896+0000 7f10dbe00640 1 -- 172.21.15.6:0/1194000376 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e406cd60 con 0x7f10e40ccad0 2024-03-22T12:57:26.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.900+0000 7f10dbe00640 1 --2- 172.21.15.6:0/1194000376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e406d840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0097a0 tx=0x7f10cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7220dad5cdb8c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.902+0000 7f10e2000640 1 -- 172.21.15.6:0/1194000376 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc035020 con 0x7f10e40ccad0 2024-03-22T12:57:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.904+0000 7f10e2000640 1 -- 172.21.15.6:0/1194000376 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc031830 con 0x7f10e40ccad0 2024-03-22T12:57:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.904+0000 7f10e9c00640 1 -- 172.21.15.6:0/1194000376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 msgr2=0x7f10e406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.904+0000 7f10e9c00640 1 --2- 172.21.15.6:0/1194000376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e406d840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0097a0 tx=0x7f10cc02b8d0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.905+0000 7f10e9c00640 1 -- 172.21.15.6:0/1194000376 shutdown_connections 2024-03-22T12:57:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.905+0000 7f10e9c00640 1 --2- 172.21.15.6:0/1194000376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e406d840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.905+0000 7f10e9c00640 1 -- 172.21.15.6:0/1194000376 >> 172.21.15.6:0/1194000376 conn(0x7f10e40b16b0 msgr2=0x7f10e40b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.905+0000 7f10e9c00640 1 -- 172.21.15.6:0/1194000376 shutdown_connections 2024-03-22T12:57:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.905+0000 7f10e9c00640 1 -- 172.21.15.6:0/1194000376 wait complete. 2024-03-22T12:57:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.906+0000 7f10e9c00640 1 Processor -- start 2024-03-22T12:57:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.907+0000 7f10e9c00640 1 -- start start 2024-03-22T12:57:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.907+0000 7f10e9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e40ca520 0x7f10e40c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.907+0000 7f10e9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e41f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.907+0000 7f10e9c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10e415b0e0 0x7f10e41f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.908+0000 7f10e9c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e4154c20 con 0x7f10e40ca520 2024-03-22T12:57:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.908+0000 7f10e9c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e406d320 con 0x7f10e40ccad0 2024-03-22T12:57:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.908+0000 7f10e9c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e415b980 con 0x7f10e415b0e0 2024-03-22T12:57:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.908+0000 7f10e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e40ca520 0x7f10e40c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.908+0000 7f10dbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e41f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.908+0000 7f10e3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10e415b0e0 0x7f10e41f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.908+0000 7f10e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e40ca520 0x7f10e40c61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41370/0 (socket says 172.21.15.6:41370) 2024-03-22T12:57:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.909+0000 7f10e2a00640 1 -- 172.21.15.6:0/3310678587 learned_addr learned my addr 172.21.15.6:0/3310678587 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.912+0000 7f10dbe00640 1 -- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10e415b0e0 msgr2=0x7f10e41f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.912+0000 7f10dbe00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10e415b0e0 0x7f10e41f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.912+0000 7f10dbe00640 1 -- 172.21.15.6:0/3310678587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e40ca520 msgr2=0x7f10e40c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.912+0000 7f10dbe00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e40ca520 0x7f10e40c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.912+0000 7f10dbe00640 1 -- 172.21.15.6:0/3310678587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e4154f00 con 0x7f10e40ccad0 2024-03-22T12:57:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.914+0000 7f10dbe00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e41f2530 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f10cc033000 tx=0x7f10cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:26.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.917+0000 7f10daa00640 1 -- 172.21.15.6:0/3310678587 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc035040 con 0x7f10e40ccad0 2024-03-22T12:57:26.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.917+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e41551e0 con 0x7f10e40ccad0 2024-03-22T12:57:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.917+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e41f1d30 con 0x7f10e40ccad0 2024-03-22T12:57:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.918+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e41f1b20 con 0x7f10e40ccad0 2024-03-22T12:57:26.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.919+0000 7f10daa00640 1 -- 172.21.15.6:0/3310678587 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc031800 con 0x7f10e40ccad0 2024-03-22T12:57:26.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:26.919+0000 7f10daa00640 1 -- 172.21.15.6:0/3310678587 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc031bb0 con 0x7f10e40ccad0 2024-03-22T12:57:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.203+0000 7f10daa00640 1 -- 172.21.15.6:0/3310678587 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f10cc038020 con 0x7f10e40ccad0 2024-03-22T12:57:27.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.206+0000 7f10daa00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f10c006bc00 0x7f10c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:27.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.206+0000 7f10daa00640 1 -- 172.21.15.6:0/3310678587 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f10cc0ad300 con 0x7f10e40ccad0 2024-03-22T12:57:27.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.206+0000 7f10e2a00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f10c006bc00 0x7f10c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.208+0000 7f10e2a00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f10c006bc00 0x7f10c006e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f10d4005880 tx=0x7f10d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:27.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.228+0000 7f10daa00640 1 -- 172.21.15.6:0/3310678587 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f10cc07b440 con 0x7f10e40ccad0 2024-03-22T12:57:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.442+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10e40ce170 con 0x7f10e40ccad0 2024-03-22T12:57:27.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.507+0000 7f10daa00640 1 -- 172.21.15.6:0/3310678587 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7f10cc07f4e0 con 0x7f10e40ccad0 2024-03-22T12:57:27.508 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:27.508 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f10c006bc00 msgr2=0x7f10c006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f10c006bc00 0x7f10c006e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f10d4005880 tx=0x7f10d4009040 comp rx=0 tx=0).stop 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 msgr2=0x7f10e41f2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e41f2530 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f10cc033000 tx=0x7f10cc0046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 shutdown_connections 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10e415b0e0 0x7f10e41f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f10c006bc00 0x7f10c006e0c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10e40ccad0 0x7f10e41f2530 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 --2- 172.21.15.6:0/3310678587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10e40ca520 0x7f10e40c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 >> 172.21.15.6:0/3310678587 conn(0x7f10e40b16b0 msgr2=0x7f10e415bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:27.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 shutdown_connections 2024-03-22T12:57:27.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.510+0000 7f10e9c00640 1 -- 172.21.15.6:0/3310678587 wait complete. 2024-03-22T12:57:27.528 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T12:56:14.196164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T12:56:26.857509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T12:56:28.394044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T12:56:34.623081+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T12:57:27.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T12:57:27.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.722+0000 7fb4a1e00640 1 Processor -- start 2024-03-22T12:57:27.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.722+0000 7fb4a1e00640 1 -- start start 2024-03-22T12:57:27.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.724+0000 7fb4a1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:27.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.724+0000 7fb4a1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb49c067550 0x7fb49c0cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:27.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.724+0000 7fb4a1e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c0c5470 con 0x7fb49c0cc700 2024-03-22T12:57:27.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.724+0000 7fb4a1e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c0c4ef0 con 0x7fb49c067550 2024-03-22T12:57:27.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.724+0000 7fb4a1e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c15b640 con 0x7fb49c1548b0 2024-03-22T12:57:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.725+0000 7fb49a000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb49c067550 0x7fb49c0cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.725+0000 7fb49aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.725+0000 7fb49a000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb49c067550 0x7fb49c0cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50326/0 (socket says 172.21.15.6:50326) 2024-03-22T12:57:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.725+0000 7fb49a000640 1 -- 172.21.15.6:0/132834635 learned_addr learned my addr 172.21.15.6:0/132834635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.728+0000 7fb49a000640 1 -- 172.21.15.6:0/132834635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 msgr2=0x7fb49c15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.728+0000 7fb49a000640 1 --2- 172.21.15.6:0/132834635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.729+0000 7fb49a000640 1 -- 172.21.15.6:0/132834635 >> v1:172.21.15.92:6789/0 conn(0x7fb49c0cc700 legacy=0x7fb49c15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.729+0000 7fb49a000640 1 -- 172.21.15.6:0/132834635 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c0c5190 con 0x7fb49c067550 2024-03-22T12:57:27.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.735+0000 7fb49a000640 1 --2- 172.21.15.6:0/132834635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c067550 0x7fb49c0cb340 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb488009870 tx=0x7fb48802b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af736920462ac60e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:27.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.739+0000 7fb499600640 1 -- 172.21.15.6:0/132834635 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488038070 con 0x7fb49c067550 2024-03-22T12:57:27.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb499600640 1 -- 172.21.15.6:0/132834635 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb488031740 con 0x7fb49c067550 2024-03-22T12:57:27.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb499600640 1 -- 172.21.15.6:0/132834635 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48802f050 con 0x7fb49c067550 2024-03-22T12:57:27.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb4a1e00640 1 -- 172.21.15.6:0/132834635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c067550 msgr2=0x7fb49c0cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:27.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/132834635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c067550 0x7fb49c0cb340 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb488009870 tx=0x7fb48802b7e0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb4a1e00640 1 -- 172.21.15.6:0/132834635 shutdown_connections 2024-03-22T12:57:27.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/132834635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c067550 0x7fb49c0cb340 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/132834635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.740+0000 7fb4a1e00640 1 -- 172.21.15.6:0/132834635 >> 172.21.15.6:0/132834635 conn(0x7fb49c06cbc0 msgr2=0x7fb49c066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.741+0000 7fb4a1e00640 1 -- 172.21.15.6:0/132834635 shutdown_connections 2024-03-22T12:57:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.741+0000 7fb4a1e00640 1 -- 172.21.15.6:0/132834635 wait complete. 2024-03-22T12:57:27.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.743+0000 7fb4a1e00640 1 Processor -- start 2024-03-22T12:57:27.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.743+0000 7fb4a1e00640 1 -- start start 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.745+0000 7fb4a1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb49c067550 0x7fb49c1e60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.745+0000 7fb4a1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c0cc700 0x7fb49c1f0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.745+0000 7fb4a1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.745+0000 7fb4a1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0c5880 con 0x7fb49c067550 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.745+0000 7fb4a1e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0cab40 con 0x7fb49c0cc700 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.745+0000 7fb4a1e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c154c90 con 0x7fb49c1548b0 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.746+0000 7fb49b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:27.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.746+0000 7fb49aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb49c067550 0x7fb49c1e60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:27.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.746+0000 7fb49b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46132/0 (socket says 172.21.15.6:46132) 2024-03-22T12:57:27.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.746+0000 7fb49b400640 1 -- 172.21.15.6:0/1383414778 learned_addr learned my addr 172.21.15.6:0/1383414778 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:27.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.746+0000 7fb49a000640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c0cc700 0x7fb49c1f0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:27.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.750+0000 7fb49aa00640 1 -- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 msgr2=0x7fb49c159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:27.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.750+0000 7fb49aa00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.750+0000 7fb49aa00640 1 -- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c0cc700 msgr2=0x7fb49c1f0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:27.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.750+0000 7fb49aa00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c0cc700 0x7fb49c1f0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:27.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.750+0000 7fb49aa00640 1 -- 172.21.15.6:0/1383414778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c157a00 con 0x7fb49c067550 2024-03-22T12:57:27.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.752+0000 7fb49aa00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb49c067550 0x7fb49c1e60c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00e8a0 tx=0x7fb48c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.756+0000 7fb493400640 1 -- 172.21.15.6:0/1383414778 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c017070 con 0x7fb49c067550 2024-03-22T12:57:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.756+0000 7fb493400640 1 -- 172.21.15.6:0/1383414778 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c00c940 con 0x7fb49c067550 2024-03-22T12:57:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.756+0000 7fb493400640 1 -- 172.21.15.6:0/1383414778 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c009e60 con 0x7fb49c067550 2024-03-22T12:57:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.757+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c1e12a0 con 0x7fb49c067550 2024-03-22T12:57:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.757+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c1e1860 con 0x7fb49c067550 2024-03-22T12:57:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.759+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c06b740 con 0x7fb49c067550 2024-03-22T12:57:27.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.793+0000 7fb493400640 1 -- 172.21.15.6:0/1383414778 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb48c01a070 con 0x7fb49c067550 2024-03-22T12:57:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.795+0000 7fb493400640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb47c06bc20 0x7fb47c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.795+0000 7fb49a000640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb47c06bc20 0x7fb47c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:27.796+0000 7fb49a000640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb47c06bc20 0x7fb47c06e0e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb488035000 tx=0x7fb488008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.115+0000 7fb493400640 1 -- 172.21.15.6:0/1383414778 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fb48c0574b0 con 0x7fb49c067550 2024-03-22T12:57:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.116+0000 7fb493400640 1 -- 172.21.15.6:0/1383414778 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb48c059410 con 0x7fb49c067550 2024-03-22T12:57:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.337+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb49c0b35d0 con 0x7fb49c067550 2024-03-22T12:57:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.366+0000 7fb493400640 1 -- 172.21.15.6:0/1383414778 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb48c05d4b0 con 0x7fb49c067550 2024-03-22T12:57:28.366 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb47c06bc20 msgr2=0x7fb47c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb47c06bc20 0x7fb47c06e0e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb488035000 tx=0x7fb488008030 comp rx=0 tx=0).stop 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb49c067550 msgr2=0x7fb49c1e60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb49c067550 0x7fb49c1e60c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00e8a0 tx=0x7fb48c00ed60 comp rx=0 tx=0).stop 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 shutdown_connections 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb49c1548b0 0x7fb49c159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb47c06bc20 0x7fb47c06e0e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb49c0cc700 0x7fb49c1f0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 --2- 172.21.15.6:0/1383414778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb49c067550 0x7fb49c1e60c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 >> 172.21.15.6:0/1383414778 conn(0x7fb49c06cbc0 msgr2=0x7fb49c0b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 shutdown_connections 2024-03-22T12:57:28.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.369+0000 7fb4a1e00640 1 -- 172.21.15.6:0/1383414778 wait complete. 2024-03-22T12:57:28.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T12:57:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.573+0000 7f423d200640 1 Processor -- start 2024-03-22T12:57:28.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.573+0000 7f423d200640 1 -- start start 2024-03-22T12:57:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.575+0000 7f423d200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f42380cf0e0 0x7f4238165c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.575+0000 7f423d200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423815a670 con 0x7f42380cc400 2024-03-22T12:57:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.575+0000 7f423d200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423815a930 con 0x7f42380cc7e0 2024-03-22T12:57:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.575+0000 7f423d200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423806dd30 con 0x7f42380cf0e0 2024-03-22T12:57:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.575+0000 7f4236a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f42380cc7e0 0x7f42380ce9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:38672/0 (socket says 172.21.15.6:38672) 2024-03-22T12:57:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.576+0000 7f4236a00640 1 -- 172.21.15.6:0/2034717222 learned_addr learned my addr 172.21.15.6:0/2034717222 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.576+0000 7f4237e00640 1 --2- 172.21.15.6:0/2034717222 >> v2:172.21.15.6:3300/0 conn(0x7f42380cf0e0 0x7f4238165c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.578+0000 7f4236000640 1 -- 172.21.15.6:0/2034717222 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042797054 0 0) 0x7f423815a930 con 0x7f42380cc7e0 2024-03-22T12:57:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.578+0000 7f4236000640 1 -- 172.21.15.6:0/2034717222 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c003520 con 0x7f42380cc7e0 2024-03-22T12:57:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.581+0000 7f4237e00640 1 -- 172.21.15.6:0/2034717222 >> v1:172.21.15.92:6789/0 conn(0x7f42380cc7e0 legacy=0x7f42380ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:28.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.581+0000 7f4237e00640 1 -- 172.21.15.6:0/2034717222 >> v1:172.21.15.6:6789/0 conn(0x7f42380cc400 legacy=0x7f42380cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:28.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.581+0000 7f4237e00640 1 -- 172.21.15.6:0/2034717222 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423815ac10 con 0x7f42380cf0e0 2024-03-22T12:57:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.587+0000 7f4237e00640 1 --2- 172.21.15.6:0/2034717222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f4238165c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4230009d90 tx=0x7f423002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2d0ce350a1fc25 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.592+0000 7f4236000640 1 -- 172.21.15.6:0/2034717222 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230039070 con 0x7f42380cf0e0 2024-03-22T12:57:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.592+0000 7f4236000640 1 -- 172.21.15.6:0/2034717222 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423002cc60 con 0x7f42380cf0e0 2024-03-22T12:57:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.592+0000 7f4236000640 1 -- 172.21.15.6:0/2034717222 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230034660 con 0x7f42380cf0e0 2024-03-22T12:57:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.593+0000 7f423d200640 1 -- 172.21.15.6:0/2034717222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 msgr2=0x7f4238165c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.593+0000 7f423d200640 1 --2- 172.21.15.6:0/2034717222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f4238165c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4230009d90 tx=0x7f423002beb0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.593+0000 7f423d200640 1 -- 172.21.15.6:0/2034717222 shutdown_connections 2024-03-22T12:57:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.593+0000 7f423d200640 1 --2- 172.21.15.6:0/2034717222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f4238165c10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.593+0000 7f423d200640 1 -- 172.21.15.6:0/2034717222 >> 172.21.15.6:0/2034717222 conn(0x7f42380cf9f0 msgr2=0x7f42380cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.594+0000 7f423d200640 1 -- 172.21.15.6:0/2034717222 shutdown_connections 2024-03-22T12:57:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.594+0000 7f423d200640 1 -- 172.21.15.6:0/2034717222 wait complete. 2024-03-22T12:57:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.595+0000 7f423d200640 1 Processor -- start 2024-03-22T12:57:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.595+0000 7f423d200640 1 -- start start 2024-03-22T12:57:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.596+0000 7f423d200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42380cc400 0x7f423815d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.596+0000 7f423d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42380cc7e0 0x7f423815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.596+0000 7f423d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f423815ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.596+0000 7f423d200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380ce1d0 con 0x7f42380cf0e0 2024-03-22T12:57:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.596+0000 7f423d200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423815b1d0 con 0x7f42380cc400 2024-03-22T12:57:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.596+0000 7f423d200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42380ce4b0 con 0x7f42380cc7e0 2024-03-22T12:57:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.597+0000 7f4237e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f423815ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.597+0000 7f4236a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42380cc7e0 0x7f423815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.598+0000 7f4237e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f423815ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41416/0 (socket says 172.21.15.6:41416) 2024-03-22T12:57:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.598+0000 7f4237e00640 1 -- 172.21.15.6:0/1463728534 learned_addr learned my addr 172.21.15.6:0/1463728534 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.598+0000 7f4237400640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42380cc400 0x7f423815d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:28.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.602+0000 7f4237400640 1 -- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42380cc7e0 msgr2=0x7f423815b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:28.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.602+0000 7f4237400640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42380cc7e0 0x7f423815b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.602+0000 7f4237400640 1 -- 172.21.15.6:0/1463728534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 msgr2=0x7f423815ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:28.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.602+0000 7f4237400640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f423815ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:28.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.602+0000 7f4237400640 1 -- 172.21.15.6:0/1463728534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380b4110 con 0x7f42380cc400 2024-03-22T12:57:28.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.604+0000 7f4237400640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42380cc400 0x7f423815d530 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f422800c340 tx=0x7f422800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:28.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.608+0000 7f422f400640 1 -- 172.21.15.6:0/1463728534 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280052d0 con 0x7f42380cc400 2024-03-22T12:57:28.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.608+0000 7f422f400640 1 -- 172.21.15.6:0/1463728534 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228005490 con 0x7f42380cc400 2024-03-22T12:57:28.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.608+0000 7f422f400640 1 -- 172.21.15.6:0/1463728534 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228007260 con 0x7f42380cc400 2024-03-22T12:57:28.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.608+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42380b4f30 con 0x7f42380cc400 2024-03-22T12:57:28.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.608+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42380b4690 con 0x7f42380cc400 2024-03-22T12:57:28.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.609+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4238075650 con 0x7f42380cc400 2024-03-22T12:57:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.630+0000 7f422f400640 1 -- 172.21.15.6:0/1463728534 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4228007400 con 0x7f42380cc400 2024-03-22T12:57:28.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.631+0000 7f422f400640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f421806bc20 0x7f421806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:28.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.632+0000 7f4236a00640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f421806bc20 0x7f421806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:28.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.632+0000 7f4236a00640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f421806bc20 0x7f421806e0e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f42380b5610 tx=0x7f4224007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:28.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.633+0000 7f422f400640 1 -- 172.21.15.6:0/1463728534 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f4228056c00 con 0x7f42380cc400 2024-03-22T12:57:28.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:28.881+0000 7f422f400640 1 -- 172.21.15.6:0/1463728534 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4228020070 con 0x7f42380cc400 2024-03-22T12:57:29.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.092+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42380b22d0 con 0x7f42380cc400 2024-03-22T12:57:29.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.100+0000 7f422f400640 1 -- 172.21.15.6:0/1463728534 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4228058b60 con 0x7f42380cc400 2024-03-22T12:57:29.099 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:57:29.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f421806bc20 msgr2=0x7f421806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:29.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f421806bc20 0x7f421806e0e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f42380b5610 tx=0x7f4224007d30 comp rx=0 tx=0).stop 2024-03-22T12:57:29.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42380cc400 msgr2=0x7f423815d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42380cc400 0x7f423815d530 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f422800c340 tx=0x7f422800c800 comp rx=0 tx=0).stop 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 shutdown_connections 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42380cf0e0 0x7f423815ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42380cc7e0 0x7f423815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f421806bc20 0x7f421806e0e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 --2- 172.21.15.6:0/1463728534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42380cc400 0x7f423815d530 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.102+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 >> 172.21.15.6:0/1463728534 conn(0x7f42380cf9f0 msgr2=0x7f42380744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.103+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 shutdown_connections 2024-03-22T12:57:29.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.103+0000 7f423d200640 1 -- 172.21.15.6:0/1463728534 wait complete. 2024-03-22T12:57:29.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T12:57:29.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.350+0000 7f5862800640 1 Processor -- start 2024-03-22T12:57:29.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.351+0000 7f5862800640 1 -- start start 2024-03-22T12:57:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.352+0000 7f5862800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f585c0cc400 0x7f585c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.352+0000 7f5862800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f585c0cd7f0 0x7f585c0ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.352+0000 7f5862800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cdc30 0x7f585c0cef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.352+0000 7f5862800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585c06dd30 con 0x7f585c0cc400 2024-03-22T12:57:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.352+0000 7f5862800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585c15a670 con 0x7f585c0cd7f0 2024-03-22T12:57:29.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.352+0000 7f5862800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585c15a930 con 0x7f585c0cdc30 2024-03-22T12:57:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.353+0000 7f5861e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cdc30 0x7f585c0cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.353+0000 7f5860a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f585c0cd7f0 0x7f585c0ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.353+0000 7f5861400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f585c0cc400 0x7f585c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.353+0000 7f5861e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cdc30 0x7f585c0cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46162/0 (socket says 172.21.15.6:46162) 2024-03-22T12:57:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.353+0000 7f5861e00640 1 -- 172.21.15.6:0/846970335 learned_addr learned my addr 172.21.15.6:0/846970335 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.356+0000 7f5861e00640 1 -- 172.21.15.6:0/846970335 >> v2:172.21.15.92:3300/0 conn(0x7f585c0cd7f0 msgr2=0x7f585c0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.356+0000 7f5861e00640 1 --2- 172.21.15.6:0/846970335 >> v2:172.21.15.92:3300/0 conn(0x7f585c0cd7f0 0x7f585c0ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.356+0000 7f5861e00640 1 -- 172.21.15.6:0/846970335 >> v2:172.21.15.6:3300/0 conn(0x7f585c0cc400 msgr2=0x7f585c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.356+0000 7f5861e00640 1 --2- 172.21.15.6:0/846970335 >> v2:172.21.15.6:3300/0 conn(0x7f585c0cc400 0x7f585c0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.356+0000 7f5861e00640 1 -- 172.21.15.6:0/846970335 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c15ac10 con 0x7f585c0cdc30 2024-03-22T12:57:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.359+0000 7f5861e00640 1 --2- 172.21.15.6:0/846970335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cdc30 0x7f585c0cef10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f585c0d0af0 tx=0x7f585402be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5e1eedfd08ec692 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.362+0000 7f585be00640 1 -- 172.21.15.6:0/846970335 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585402c9f0 con 0x7f585c0cdc30 2024-03-22T12:57:29.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.363+0000 7f585be00640 1 -- 172.21.15.6:0/846970335 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585402cb90 con 0x7f585c0cdc30 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 -- 172.21.15.6:0/846970335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cdc30 msgr2=0x7f585c0cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 --2- 172.21.15.6:0/846970335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cdc30 0x7f585c0cef10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f585c0d0af0 tx=0x7f585402be50 comp rx=0 tx=0).stop 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 -- 172.21.15.6:0/846970335 shutdown_connections 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 --2- 172.21.15.6:0/846970335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cdc30 0x7f585c0cef10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 --2- 172.21.15.6:0/846970335 >> v2:172.21.15.92:3300/0 conn(0x7f585c0cd7f0 0x7f585c0ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 --2- 172.21.15.6:0/846970335 >> v2:172.21.15.6:3300/0 conn(0x7f585c0cc400 0x7f585c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 -- 172.21.15.6:0/846970335 >> 172.21.15.6:0/846970335 conn(0x7f585c0cf9f0 msgr2=0x7f585c0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 -- 172.21.15.6:0/846970335 shutdown_connections 2024-03-22T12:57:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.364+0000 7f5862800640 1 -- 172.21.15.6:0/846970335 wait complete. 2024-03-22T12:57:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.366+0000 7f5862800640 1 Processor -- start 2024-03-22T12:57:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.366+0000 7f5862800640 1 -- start start 2024-03-22T12:57:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.367+0000 7f5862800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f585c0cc400 0x7f585c0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.367+0000 7f5862800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cd7f0 0x7f585c15e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.367+0000 7f5862800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585c0cdc30 0x7f585c1f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.367+0000 7f5862800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c0ce1d0 con 0x7f585c0cdc30 2024-03-22T12:57:29.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.367+0000 7f5862800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c15b1d0 con 0x7f585c0cc400 2024-03-22T12:57:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.367+0000 7f5862800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585c0ce4b0 con 0x7f585c0cd7f0 2024-03-22T12:57:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.369+0000 7f5861e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585c0cdc30 0x7f585c1f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.369+0000 7f5861400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f585c0cc400 0x7f585c0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.369+0000 7f5860a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cd7f0 0x7f585c15e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.369+0000 7f5861e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585c0cdc30 0x7f585c1f1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41428/0 (socket says 172.21.15.6:41428) 2024-03-22T12:57:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.369+0000 7f5861e00640 1 -- 172.21.15.6:0/1193252918 learned_addr learned my addr 172.21.15.6:0/1193252918 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.373+0000 7f5861e00640 1 -- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cd7f0 msgr2=0x7f585c15e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.373+0000 7f5861e00640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cd7f0 0x7f585c15e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.373+0000 7f5861e00640 1 -- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f585c0cc400 msgr2=0x7f585c0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.373+0000 7f5861e00640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f585c0cc400 0x7f585c0b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.373+0000 7f5861e00640 1 -- 172.21.15.6:0/1193252918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c154f50 con 0x7f585c0cdc30 2024-03-22T12:57:29.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.378+0000 7f5861e00640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585c0cdc30 0x7f585c1f1640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5854009f40 tx=0x7f5854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.383+0000 7f5859600640 1 -- 172.21.15.6:0/1193252918 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5854043020 con 0x7f585c0cdc30 2024-03-22T12:57:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.383+0000 7f5859600640 1 -- 172.21.15.6:0/1193252918 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5854003e10 con 0x7f585c0cdc30 2024-03-22T12:57:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.383+0000 7f5859600640 1 -- 172.21.15.6:0/1193252918 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5854009510 con 0x7f585c0cdc30 2024-03-22T12:57:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.383+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585c0c7e20 con 0x7f585c0cdc30 2024-03-22T12:57:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.383+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585c1554f0 con 0x7f585c0cdc30 2024-03-22T12:57:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.384+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585c075650 con 0x7f585c0cdc30 2024-03-22T12:57:29.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.409+0000 7f5859600640 1 -- 172.21.15.6:0/1193252918 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5854046020 con 0x7f585c0cdc30 2024-03-22T12:57:29.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.411+0000 7f5859600640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f583c06bc50 0x7f583c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:29.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.411+0000 7f5861400640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f583c06bc50 0x7f583c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:29.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.412+0000 7f5861400640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f583c06bc50 0x7f583c06e110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5850009ac0 tx=0x7f5850009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:29.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.413+0000 7f5859600640 1 -- 172.21.15.6:0/1193252918 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f5854079940 con 0x7f585c0cdc30 2024-03-22T12:57:29.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:57:29.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:57:29.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.694+0000 7f5859600640 1 -- 172.21.15.6:0/1193252918 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f585404d070 con 0x7f585c0cdc30 2024-03-22T12:57:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.899+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f585c06def0 con 0x7f585c0cdc30 2024-03-22T12:57:29.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.953+0000 7f5859600640 1 -- 172.21.15.6:0/1193252918 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) v1 ==== 100+0+11 (secure 0 0 0) 0x7f585407b8a0 con 0x7f585c0cdc30 2024-03-22T12:57:29.954 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-22T12:57:29.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f583c06bc50 msgr2=0x7f583c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:29.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f583c06bc50 0x7f583c06e110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5850009ac0 tx=0x7f5850009190 comp rx=0 tx=0).stop 2024-03-22T12:57:29.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585c0cdc30 msgr2=0x7f585c1f1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:29.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585c0cdc30 0x7f585c1f1640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5854009f40 tx=0x7f5854004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 shutdown_connections 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f585c0cdc30 0x7f585c1f1640 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f585c0cd7f0 0x7f585c15e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f583c06bc50 0x7f583c06e110 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 --2- 172.21.15.6:0/1193252918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f585c0cc400 0x7f585c0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 >> 172.21.15.6:0/1193252918 conn(0x7f585c0cf9f0 msgr2=0x7f585c0c70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.956+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 shutdown_connections 2024-03-22T12:57:29.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:29.957+0000 7f5862800640 1 -- 172.21.15.6:0/1193252918 wait complete. 2024-03-22T12:57:29.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T12:57:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.215+0000 7f8c00a00640 1 Processor -- start 2024-03-22T12:57:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.215+0000 7f8c00a00640 1 -- start start 2024-03-22T12:57:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.217+0000 7f8c00a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.217+0000 7f8c00a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8bfc0cef10 0x7f8bfc164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.217+0000 7f8c00a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0cf3a0 con 0x7f8bfc0cc400 2024-03-22T12:57:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.217+0000 7f8c00a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc06dd30 con 0x7f8bfc0cef10 2024-03-22T12:57:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.217+0000 7f8c00a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc15a670 con 0x7f8bfc0cc7e0 2024-03-22T12:57:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.218+0000 7f8bfa800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8bfc0cc400 0x7f8bfc0cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46838/0 (socket says 172.21.15.6:46838) 2024-03-22T12:57:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.218+0000 7f8bfa800640 1 -- 172.21.15.6:0/392138116 learned_addr learned my addr 172.21.15.6:0/392138116 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.218+0000 7f8bf9e00640 1 --2- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.218+0000 7f8bfb200640 1 --2- 172.21.15.6:0/392138116 >> v2:172.21.15.92:3300/0 conn(0x7f8bfc0cef10 0x7f8bfc164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.220+0000 7f8bf9400640 1 -- 172.21.15.6:0/392138116 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785768706 0 0) 0x7f8bfc0cf3a0 con 0x7f8bfc0cc400 2024-03-22T12:57:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.220+0000 7f8bf9400640 1 -- 172.21.15.6:0/392138116 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be0003520 con 0x7f8bfc0cc400 2024-03-22T12:57:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.221+0000 7f8bfb200640 1 -- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 msgr2=0x7f8bfc0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.221+0000 7f8bfb200640 1 --2- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.221+0000 7f8bfb200640 1 -- 172.21.15.6:0/392138116 >> v1:172.21.15.6:6789/0 conn(0x7f8bfc0cc400 legacy=0x7f8bfc0cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.221+0000 7f8bfb200640 1 -- 172.21.15.6:0/392138116 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc15a930 con 0x7f8bfc0cef10 2024-03-22T12:57:30.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.225+0000 7f8bfb200640 1 --2- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cef10 0x7f8bfc164c00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009d90 tx=0x7f8bf402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a0209f86fcbc11 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.229+0000 7f8bf9400640 1 -- 172.21.15.6:0/392138116 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4039070 con 0x7f8bfc0cef10 2024-03-22T12:57:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.229+0000 7f8bf9400640 1 -- 172.21.15.6:0/392138116 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf402cbc0 con 0x7f8bfc0cef10 2024-03-22T12:57:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.229+0000 7f8bf9400640 1 -- 172.21.15.6:0/392138116 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4034440 con 0x7f8bfc0cef10 2024-03-22T12:57:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 -- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cef10 msgr2=0x7f8bfc164c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 --2- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cef10 0x7f8bfc164c00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009d90 tx=0x7f8bf402be10 comp rx=0 tx=0).stop 2024-03-22T12:57:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 -- 172.21.15.6:0/392138116 shutdown_connections 2024-03-22T12:57:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 --2- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cef10 0x7f8bfc164c00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 --2- 172.21.15.6:0/392138116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 -- 172.21.15.6:0/392138116 >> 172.21.15.6:0/392138116 conn(0x7f8bfc0cf9f0 msgr2=0x7f8bfc0cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 -- 172.21.15.6:0/392138116 shutdown_connections 2024-03-22T12:57:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.230+0000 7f8c00a00640 1 -- 172.21.15.6:0/392138116 wait complete. 2024-03-22T12:57:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.232+0000 7f8c00a00640 1 Processor -- start 2024-03-22T12:57:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.232+0000 7f8c00a00640 1 -- start start 2024-03-22T12:57:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.233+0000 7f8c00a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cc400 0x7f8bfc0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.233+0000 7f8c00a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.233+0000 7f8c00a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0cef10 0x7f8bfc15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.233+0000 7f8c00a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc15b1d0 con 0x7f8bfc0cef10 2024-03-22T12:57:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.233+0000 7f8c00a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc15aef0 con 0x7f8bfc0cc400 2024-03-22T12:57:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.233+0000 7f8c00a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0ce1d0 con 0x7f8bfc0cc7e0 2024-03-22T12:57:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.234+0000 7f8bfa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cc400 0x7f8bfc0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.234+0000 7f8bf9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.234+0000 7f8bfb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0cef10 0x7f8bfc15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.234+0000 7f8bfa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cc400 0x7f8bfc0b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50358/0 (socket says 172.21.15.6:50358) 2024-03-22T12:57:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.234+0000 7f8bfa800640 1 -- 172.21.15.6:0/3714133096 learned_addr learned my addr 172.21.15.6:0/3714133096 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.237+0000 7f8bfa800640 1 -- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 msgr2=0x7f8bfc0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.237+0000 7f8bfa800640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.237+0000 7f8bfa800640 1 -- 172.21.15.6:0/3714133096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0cef10 msgr2=0x7f8bfc15b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.237+0000 7f8bfa800640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0cef10 0x7f8bfc15b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.237+0000 7f8bfa800640 1 -- 172.21.15.6:0/3714133096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc15e150 con 0x7f8bfc0cc400 2024-03-22T12:57:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.240+0000 7f8bfa800640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cc400 0x7f8bfc0b02d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00c460 tx=0x7f8bec00c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.244+0000 7f8bf2a00640 1 -- 172.21.15.6:0/3714133096 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec017070 con 0x7f8bfc0cc400 2024-03-22T12:57:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.244+0000 7f8bf2a00640 1 -- 172.21.15.6:0/3714133096 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec005ce0 con 0x7f8bfc0cc400 2024-03-22T12:57:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.244+0000 7f8bf2a00640 1 -- 172.21.15.6:0/3714133096 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec007290 con 0x7f8bfc0cc400 2024-03-22T12:57:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.244+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc0b13e0 con 0x7f8bfc0cc400 2024-03-22T12:57:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.244+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc0b16a0 con 0x7f8bfc0cc400 2024-03-22T12:57:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.245+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc0ca040 con 0x7f8bfc0cc400 2024-03-22T12:57:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.266+0000 7f8bf2a00640 1 -- 172.21.15.6:0/3714133096 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8bec01a070 con 0x7f8bfc0cc400 2024-03-22T12:57:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.267+0000 7f8bf2a00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bdc06bc50 0x7f8bdc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.267+0000 7f8bf9e00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bdc06bc50 0x7f8bdc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.267+0000 7f8bf2a00640 1 -- 172.21.15.6:0/3714133096 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f8bec056710 con 0x7f8bfc0cc400 2024-03-22T12:57:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.268+0000 7f8bf9e00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bdc06bc50 0x7f8bdc06e110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0c2ef0 tx=0x7f8be8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:30.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.515+0000 7f8bf2a00640 1 -- 172.21.15.6:0/3714133096 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8bfc0ca040 con 0x7f8bfc0cc400 2024-03-22T12:57:30.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.724+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8bfc1f3c90 con 0x7f8bfc0cc400 2024-03-22T12:57:30.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.731+0000 7f8bf2a00640 1 -- 172.21.15.6:0/3714133096 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v30) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8bec021020 con 0x7f8bfc0cc400 2024-03-22T12:57:30.731 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:57:30.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bdc06bc50 msgr2=0x7f8bdc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bdc06bc50 0x7f8bdc06e110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0c2ef0 tx=0x7f8be8009190 comp rx=0 tx=0).stop 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cc400 msgr2=0x7f8bfc0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cc400 0x7f8bfc0b02d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00c460 tx=0x7f8bec00c920 comp rx=0 tx=0).stop 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 shutdown_connections 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8bfc0cef10 0x7f8bfc15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bfc0cc7e0 0x7f8bfc0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8bdc06bc50 0x7f8bdc06e110 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 --2- 172.21.15.6:0/3714133096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bfc0cc400 0x7f8bfc0b02d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.734+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 >> 172.21.15.6:0/3714133096 conn(0x7f8bfc0cf9f0 msgr2=0x7f8bfc159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.735+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 shutdown_connections 2024-03-22T12:57:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.735+0000 7f8c00a00640 1 -- 172.21.15.6:0/3714133096 wait complete. 2024-03-22T12:57:30.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:57:30.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.934+0000 7f80a0e00640 1 Processor -- start 2024-03-22T12:57:30.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.934+0000 7f80a0e00640 1 -- start start 2024-03-22T12:57:30.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.935+0000 7f80a0e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.936+0000 7f80a0e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f809c0ccad0 0x7f809c15b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.936+0000 7f80a0e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809c06c7a0 con 0x7f809c15b0e0 2024-03-22T12:57:30.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.936+0000 7f80a0e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809c0cd000 con 0x7f809c0ccad0 2024-03-22T12:57:30.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.936+0000 7f80a0e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c06ca80 con 0x7f809c0ca520 2024-03-22T12:57:30.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.936+0000 7f809a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f809c0ccad0 0x7f809c15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.936+0000 7f8099e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.937+0000 7f809a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f809c0ccad0 0x7f809c15b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50364/0 (socket says 172.21.15.6:50364) 2024-03-22T12:57:30.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.937+0000 7f809a800640 1 -- 172.21.15.6:0/1882730425 learned_addr learned my addr 172.21.15.6:0/1882730425 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:30.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.940+0000 7f809a800640 1 -- 172.21.15.6:0/1882730425 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 msgr2=0x7f809c06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:30.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.940+0000 7f809a800640 1 --2- 172.21.15.6:0/1882730425 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c06d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.940+0000 7f809a800640 1 -- 172.21.15.6:0/1882730425 >> v1:172.21.15.92:6789/0 conn(0x7f809c15b0e0 legacy=0x7f809c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:30.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.940+0000 7f809a800640 1 -- 172.21.15.6:0/1882730425 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c06cd60 con 0x7f809c0ccad0 2024-03-22T12:57:30.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.945+0000 7f809a800640 1 --2- 172.21.15.6:0/1882730425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c15b980 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8094009e50 tx=0x7f809402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2826107a95701f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:30.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.949+0000 7f8098a00640 1 -- 172.21.15.6:0/1882730425 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809402ca70 con 0x7f809c0ccad0 2024-03-22T12:57:30.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.949+0000 7f8098a00640 1 -- 172.21.15.6:0/1882730425 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809402cc10 con 0x7f809c0ccad0 2024-03-22T12:57:30.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.949+0000 7f8098a00640 1 -- 172.21.15.6:0/1882730425 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80940343f0 con 0x7f809c0ccad0 2024-03-22T12:57:30.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 -- 172.21.15.6:0/1882730425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 msgr2=0x7f809c15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 --2- 172.21.15.6:0/1882730425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c15b980 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8094009e50 tx=0x7f809402bed0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 -- 172.21.15.6:0/1882730425 shutdown_connections 2024-03-22T12:57:30.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 --2- 172.21.15.6:0/1882730425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c15b980 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 --2- 172.21.15.6:0/1882730425 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 -- 172.21.15.6:0/1882730425 >> 172.21.15.6:0/1882730425 conn(0x7f809c0b16b0 msgr2=0x7f809c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:30.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 -- 172.21.15.6:0/1882730425 shutdown_connections 2024-03-22T12:57:30.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.950+0000 7f80a0e00640 1 -- 172.21.15.6:0/1882730425 wait complete. 2024-03-22T12:57:30.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.952+0000 7f80a0e00640 1 Processor -- start 2024-03-22T12:57:30.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.953+0000 7f80a0e00640 1 -- start start 2024-03-22T12:57:30.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.953+0000 7f80a0e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c1e6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.953+0000 7f80a0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c0b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.953+0000 7f80a0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809c15b0e0 0x7f809c1f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:30.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.953+0000 7f80a0e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c154c20 con 0x7f809c15b0e0 2024-03-22T12:57:30.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.953+0000 7f80a0e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c06d320 con 0x7f809c0ccad0 2024-03-22T12:57:30.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.953+0000 7f80a0e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0ca900 con 0x7f809c0ca520 2024-03-22T12:57:30.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.954+0000 7f8099e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c1e6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.954+0000 7f8099400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c0b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.954+0000 7f809a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809c15b0e0 0x7f809c1f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:30.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.955+0000 7f8099400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c0b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50368/0 (socket says 172.21.15.6:50368) 2024-03-22T12:57:30.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.955+0000 7f8099400640 1 -- 172.21.15.6:0/4276535193 learned_addr learned my addr 172.21.15.6:0/4276535193 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:30.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.959+0000 7f809a800640 1 -- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 msgr2=0x7f809c1e6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.959+0000 7f809a800640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c1e6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.959+0000 7f809a800640 1 -- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 msgr2=0x7f809c0b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:30.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.959+0000 7f809a800640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c0b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:30.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.959+0000 7f809a800640 1 -- 172.21.15.6:0/4276535193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c1e66c0 con 0x7f809c15b0e0 2024-03-22T12:57:30.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.961+0000 7f809a800640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809c15b0e0 0x7f809c1f3f10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8094032000 tx=0x7f8094004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:30.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.965+0000 7f8092000640 1 -- 172.21.15.6:0/4276535193 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094043020 con 0x7f809c15b0e0 2024-03-22T12:57:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.966+0000 7f8092000640 1 -- 172.21.15.6:0/4276535193 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809403e450 con 0x7f809c15b0e0 2024-03-22T12:57:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.966+0000 7f8092000640 1 -- 172.21.15.6:0/4276535193 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809403e840 con 0x7f809c15b0e0 2024-03-22T12:57:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.966+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809c1e46b0 con 0x7f809c15b0e0 2024-03-22T12:57:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.966+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809c0c85d0 con 0x7f809c15b0e0 2024-03-22T12:57:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:30.967+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809c0756c0 con 0x7f809c15b0e0 2024-03-22T12:57:31.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.049+0000 7f8092000640 1 -- 172.21.15.6:0/4276535193 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8094046020 con 0x7f809c15b0e0 2024-03-22T12:57:31.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.051+0000 7f8092000640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807c06bbd0 0x7f807c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:31.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.052+0000 7f8099e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807c06bbd0 0x7f807c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:31.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.053+0000 7f8099e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807c06bbd0 0x7f807c06e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f808c009b40 tx=0x7f808c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:31.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.291+0000 7f8092000640 1 -- 172.21.15.6:0/4276535193 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f8094079a20 con 0x7f809c15b0e0 2024-03-22T12:57:31.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.292+0000 7f8092000640 1 -- 172.21.15.6:0/4276535193 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f809404d030 con 0x7f809c15b0e0 2024-03-22T12:57:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.544+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f809c0b5b20 con 0x7f809c15b0e0 2024-03-22T12:57:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.568+0000 7f8092000640 1 -- 172.21.15.6:0/4276535193 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6628 (secure 0 0 0) 0x7f809407b980 con 0x7f809c15b0e0 2024-03-22T12:57:31.568 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:31.569 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:57:18.963149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4259,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4259":{"gid":4259,"name":"d","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6837/2825293724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2825293724},{"type":"v1","addr":"172.21.15.92:6837","nonce":2825293724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4259,4261,4268]},"id":1}]} 2024-03-22T12:57:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.571+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807c06bbd0 msgr2=0x7f807c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.571+0000 7f80a0e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807c06bbd0 0x7f807c06e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f808c009b40 tx=0x7f808c009190 comp rx=0 tx=0).stop 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.571+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809c15b0e0 msgr2=0x7f809c1f3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.571+0000 7f80a0e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809c15b0e0 0x7f809c1f3f10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8094032000 tx=0x7f8094004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 shutdown_connections 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809c15b0e0 0x7f809c1f3f10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807c06bbd0 0x7f807c06e090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f809c0ccad0 0x7f809c0b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 --2- 172.21.15.6:0/4276535193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f809c0ca520 0x7f809c1e6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 >> 172.21.15.6:0/4276535193 conn(0x7f809c0b16b0 msgr2=0x7f809c0b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 shutdown_connections 2024-03-22T12:57:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.572+0000 7f80a0e00640 1 -- 172.21.15.6:0/4276535193 wait complete. 2024-03-22T12:57:31.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.777+0000 7fc2a3200640 1 Processor -- start 2024-03-22T12:57:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.777+0000 7fc2a3200640 1 -- start start 2024-03-22T12:57:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.779+0000 7fc2a3200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.779+0000 7fc2a3200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2a40c9000 0x7fc2a4071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.779+0000 7fc2a3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2a415bae0 0x7fc2a415bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.779+0000 7fc2a3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a415c4b0 con 0x7fc2a415bae0 2024-03-22T12:57:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.779+0000 7fc2a3200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a4072d90 con 0x7fc2a40c9000 2024-03-22T12:57:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.779+0000 7fc2a3200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4073070 con 0x7fc2a40c8c20 2024-03-22T12:57:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.780+0000 7fc2a1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.780+0000 7fc2a1400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2a40c9000 0x7fc2a4071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.780+0000 7fc2a2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2a415bae0 0x7fc2a415bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.780+0000 7fc2a1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a406ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46208/0 (socket says 172.21.15.6:46208) 2024-03-22T12:57:31.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.780+0000 7fc2a1e00640 1 -- 172.21.15.6:0/2134562460 learned_addr learned my addr 172.21.15.6:0/2134562460 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.782+0000 7fc2a1e00640 1 -- 172.21.15.6:0/2134562460 >> v2:172.21.15.92:3300/0 conn(0x7fc2a40c9000 msgr2=0x7fc2a4071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.782+0000 7fc2a1e00640 1 --2- 172.21.15.6:0/2134562460 >> v2:172.21.15.92:3300/0 conn(0x7fc2a40c9000 0x7fc2a4071cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.782+0000 7fc2a1e00640 1 -- 172.21.15.6:0/2134562460 >> v2:172.21.15.6:3300/0 conn(0x7fc2a415bae0 msgr2=0x7fc2a415bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.782+0000 7fc2a1e00640 1 --2- 172.21.15.6:0/2134562460 >> v2:172.21.15.6:3300/0 conn(0x7fc2a415bae0 0x7fc2a415bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.782+0000 7fc2a1e00640 1 -- 172.21.15.6:0/2134562460 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a4073350 con 0x7fc2a40c8c20 2024-03-22T12:57:31.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.784+0000 7fc2a1e00640 1 --2- 172.21.15.6:0/2134562460 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a406ffb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc2a40c3480 tx=0x7fc29402b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d693b8939e2a41ba server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:31.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.789+0000 7fc2a0a00640 1 -- 172.21.15.6:0/2134562460 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294035020 con 0x7fc2a40c8c20 2024-03-22T12:57:31.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.789+0000 7fc2a0a00640 1 -- 172.21.15.6:0/2134562460 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29402d040 con 0x7fc2a40c8c20 2024-03-22T12:57:31.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.789+0000 7fc2a0a00640 1 -- 172.21.15.6:0/2134562460 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294031a80 con 0x7fc2a40c8c20 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 -- 172.21.15.6:0/2134562460 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 msgr2=0x7fc2a406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 --2- 172.21.15.6:0/2134562460 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a406ffb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc2a40c3480 tx=0x7fc29402b900 comp rx=0 tx=0).stop 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 -- 172.21.15.6:0/2134562460 shutdown_connections 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 --2- 172.21.15.6:0/2134562460 >> v2:172.21.15.6:3300/0 conn(0x7fc2a415bae0 0x7fc2a415bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 --2- 172.21.15.6:0/2134562460 >> v2:172.21.15.92:3300/0 conn(0x7fc2a40c9000 0x7fc2a4071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 --2- 172.21.15.6:0/2134562460 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a406ffb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 -- 172.21.15.6:0/2134562460 >> 172.21.15.6:0/2134562460 conn(0x7fc2a406df40 msgr2=0x7fc2a406f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 -- 172.21.15.6:0/2134562460 shutdown_connections 2024-03-22T12:57:31.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.790+0000 7fc2a3200640 1 -- 172.21.15.6:0/2134562460 wait complete. 2024-03-22T12:57:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.792+0000 7fc2a3200640 1 Processor -- start 2024-03-22T12:57:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.792+0000 7fc2a3200640 1 -- start start 2024-03-22T12:57:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.793+0000 7fc2a3200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a40cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.793+0000 7fc2a3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a40c9000 0x7fc2a40cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.793+0000 7fc2a3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a415bae0 0x7fc2a40c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.793+0000 7fc2a3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4072230 con 0x7fc2a415bae0 2024-03-22T12:57:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.793+0000 7fc2a3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a40729d0 con 0x7fc2a40c9000 2024-03-22T12:57:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.793+0000 7fc2a3200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4073630 con 0x7fc2a40c8c20 2024-03-22T12:57:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.794+0000 7fc2a1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a40c9000 0x7fc2a40cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.794+0000 7fc2a1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a40cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.794+0000 7fc2a2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a415bae0 0x7fc2a40c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.795+0000 7fc2a2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a415bae0 0x7fc2a40c47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41472/0 (socket says 172.21.15.6:41472) 2024-03-22T12:57:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.795+0000 7fc2a2800640 1 -- 172.21.15.6:0/3913814901 learned_addr learned my addr 172.21.15.6:0/3913814901 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:31.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.799+0000 7fc2a2800640 1 -- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 msgr2=0x7fc2a40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:31.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.799+0000 7fc2a2800640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a40cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.799+0000 7fc2a2800640 1 -- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a40c9000 msgr2=0x7fc2a40cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.799+0000 7fc2a2800640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a40c9000 0x7fc2a40cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.799+0000 7fc2a2800640 1 -- 172.21.15.6:0/3913814901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a40cc230 con 0x7fc2a415bae0 2024-03-22T12:57:31.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.801+0000 7fc2a2800640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a415bae0 0x7fc2a40c47a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009d30 tx=0x7fc29c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.806+0000 7fc29a000640 1 -- 172.21.15.6:0/3913814901 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c018020 con 0x7fc2a415bae0 2024-03-22T12:57:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.806+0000 7fc29a000640 1 -- 172.21.15.6:0/3913814901 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c014480 con 0x7fc2a415bae0 2024-03-22T12:57:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.806+0000 7fc29a000640 1 -- 172.21.15.6:0/3913814901 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c014870 con 0x7fc2a415bae0 2024-03-22T12:57:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.806+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a4144750 con 0x7fc2a415bae0 2024-03-22T12:57:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.806+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a4144d10 con 0x7fc2a415bae0 2024-03-22T12:57:31.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:31.808+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a4071f20 con 0x7fc2a415bae0 2024-03-22T12:57:32.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.123+0000 7fc29a000640 1 -- 172.21.15.6:0/3913814901 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc29c005d10 con 0x7fc2a415bae0 2024-03-22T12:57:32.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.124+0000 7fc29a000640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc28406bc20 0x7fc28406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.125+0000 7fc2a1e00640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc28406bc20 0x7fc28406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.126+0000 7fc2a1e00640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc28406bc20 0x7fc28406e0e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc294036000 tx=0x7fc29403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:32.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.154+0000 7fc29a000640 1 -- 172.21.15.6:0/3913814901 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fc29c0581d0 con 0x7fc2a415bae0 2024-03-22T12:57:32.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.156+0000 7fc29a000640 1 -- 172.21.15.6:0/3913814901 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc2a4071f20 con 0x7fc2a415bae0 2024-03-22T12:57:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.495+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2a4075650 con 0x7fc2a415bae0 2024-03-22T12:57:32.539 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T12:57:32.526+0000 f5f9640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-22T12:57:32.539 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T12:57:32.526+0000 f5f9640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-22T12:57:32.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.557+0000 7fc29a000640 1 -- 172.21.15.6:0/3913814901 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7fc29c022030 con 0x7fc2a415bae0 2024-03-22T12:57:32.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:32.557 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc28406bc20 msgr2=0x7fc28406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc28406bc20 0x7fc28406e0e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc294036000 tx=0x7fc29403c040 comp rx=0 tx=0).stop 2024-03-22T12:57:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a415bae0 msgr2=0x7fc2a40c47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a415bae0 0x7fc2a40c47a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009d30 tx=0x7fc29c00c490 comp rx=0 tx=0).stop 2024-03-22T12:57:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 shutdown_connections 2024-03-22T12:57:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a415bae0 0x7fc2a40c47a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc28406bc20 0x7fc28406e0e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a40c9000 0x7fc2a40cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 --2- 172.21.15.6:0/3913814901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a40c8c20 0x7fc2a40cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 >> 172.21.15.6:0/3913814901 conn(0x7fc2a406df40 msgr2=0x7fc2a40c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 shutdown_connections 2024-03-22T12:57:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.560+0000 7fc2a3200640 1 -- 172.21.15.6:0/3913814901 wait complete. 2024-03-22T12:57:32.582 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T12:56:14.196164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T12:56:26.857509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T12:56:28.394044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T12:56:34.623081+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T12:57:32.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T12:57:32.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.765+0000 7f8e6a600640 1 Processor -- start 2024-03-22T12:57:32.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.765+0000 7f8e6a600640 1 -- start start 2024-03-22T12:57:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.767+0000 7f8e6a600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640ceae0 0x7f8e640ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.767+0000 7f8e6a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8e640caa80 0x7f8e640cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.767+0000 7f8e6a600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e640ccf20 con 0x7f8e640cf660 2024-03-22T12:57:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.767+0000 7f8e6a600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e640cfa40 con 0x7f8e640caa80 2024-03-22T12:57:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.767+0000 7f8e6a600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6415b470 con 0x7f8e640ceae0 2024-03-22T12:57:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.768+0000 7f8e63400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8e640cf660 0x7f8e640cd9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:38682/0 (socket says 172.21.15.6:38682) 2024-03-22T12:57:32.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.768+0000 7f8e62a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640ceae0 0x7f8e640ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:32.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.768+0000 7f8e63400640 1 -- 172.21.15.6:0/2898258878 learned_addr learned my addr 172.21.15.6:0/2898258878 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:32.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.771+0000 7f8e62000640 1 -- 172.21.15.6:0/2898258878 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846575254 0 0) 0x7f8e640ccf20 con 0x7f8e640cf660 2024-03-22T12:57:32.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.771+0000 7f8e62000640 1 -- 172.21.15.6:0/2898258878 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e48003520 con 0x7f8e640cf660 2024-03-22T12:57:32.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.772+0000 7f8e62a00640 1 -- 172.21.15.6:0/2898258878 >> v2:172.21.15.6:3300/0 conn(0x7f8e640caa80 msgr2=0x7f8e640cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:32.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.772+0000 7f8e62a00640 1 --2- 172.21.15.6:0/2898258878 >> v2:172.21.15.6:3300/0 conn(0x7f8e640caa80 0x7f8e640cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.772+0000 7f8e62a00640 1 -- 172.21.15.6:0/2898258878 >> v1:172.21.15.92:6789/0 conn(0x7f8e640cf660 legacy=0x7f8e640cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:32.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.772+0000 7f8e62a00640 1 -- 172.21.15.6:0/2898258878 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6415a6b0 con 0x7f8e640ceae0 2024-03-22T12:57:32.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.775+0000 7f8e62a00640 1 --2- 172.21.15.6:0/2898258878 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640ceae0 0x7f8e640ca540 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0097a0 tx=0x7f8e4c02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=193aea86710af9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:32.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.779+0000 7f8e62000640 1 -- 172.21.15.6:0/2898258878 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c034070 con 0x7f8e640ceae0 2024-03-22T12:57:32.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.779+0000 7f8e62000640 1 -- 172.21.15.6:0/2898258878 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c030800 con 0x7f8e640ceae0 2024-03-22T12:57:32.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.779+0000 7f8e62000640 1 -- 172.21.15.6:0/2898258878 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c030bd0 con 0x7f8e640ceae0 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 -- 172.21.15.6:0/2898258878 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640ceae0 msgr2=0x7f8e640ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 --2- 172.21.15.6:0/2898258878 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640ceae0 0x7f8e640ca540 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0097a0 tx=0x7f8e4c02b940 comp rx=0 tx=0).stop 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 -- 172.21.15.6:0/2898258878 shutdown_connections 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 --2- 172.21.15.6:0/2898258878 >> v2:172.21.15.6:3300/0 conn(0x7f8e640caa80 0x7f8e640cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 --2- 172.21.15.6:0/2898258878 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640ceae0 0x7f8e640ca540 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 -- 172.21.15.6:0/2898258878 >> 172.21.15.6:0/2898258878 conn(0x7f8e640b16d0 msgr2=0x7f8e640b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 -- 172.21.15.6:0/2898258878 shutdown_connections 2024-03-22T12:57:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.781+0000 7f8e6a600640 1 -- 172.21.15.6:0/2898258878 wait complete. 2024-03-22T12:57:32.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.783+0000 7f8e6a600640 1 Processor -- start 2024-03-22T12:57:32.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.783+0000 7f8e6a600640 1 -- start start 2024-03-22T12:57:32.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.784+0000 7f8e6a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e640caa80 0x7f8e640b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:32.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.784+0000 7f8e6a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e640ceae0 0x7f8e6414b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:32.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.784+0000 7f8e6a600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640cf660 0x7f8e64151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:32.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.784+0000 7f8e6a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6415ac70 con 0x7f8e640caa80 2024-03-22T12:57:32.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.784+0000 7f8e6a600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6415afe0 con 0x7f8e640ceae0 2024-03-22T12:57:32.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.784+0000 7f8e6a600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640ce0d0 con 0x7f8e640cf660 2024-03-22T12:57:32.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.785+0000 7f8e63e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640cf660 0x7f8e64151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:32.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.787+0000 7f8e62a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e640ceae0 0x7f8e6414b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:32.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.787+0000 7f8e63e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640cf660 0x7f8e64151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46226/0 (socket says 172.21.15.6:46226) 2024-03-22T12:57:32.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.787+0000 7f8e63e00640 1 -- 172.21.15.6:0/3779039243 learned_addr learned my addr 172.21.15.6:0/3779039243 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:32.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.789+0000 7f8e63400640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e640caa80 0x7f8e640b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:32.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.791+0000 7f8e62a00640 1 -- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640cf660 msgr2=0x7f8e64151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.791+0000 7f8e62a00640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640cf660 0x7f8e64151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.791+0000 7f8e62a00640 1 -- 172.21.15.6:0/3779039243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e640caa80 msgr2=0x7f8e640b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.791+0000 7f8e62a00640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e640caa80 0x7f8e640b3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:32.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.791+0000 7f8e62a00640 1 -- 172.21.15.6:0/3779039243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6414ebb0 con 0x7f8e640ceae0 2024-03-22T12:57:32.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.794+0000 7f8e62a00640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e640ceae0 0x7f8e6414b770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c033000 tx=0x7f8e4c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:32.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.798+0000 7f8e5b400640 1 -- 172.21.15.6:0/3779039243 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c034040 con 0x7f8e640ceae0 2024-03-22T12:57:32.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.798+0000 7f8e5b400640 1 -- 172.21.15.6:0/3779039243 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c030600 con 0x7f8e640ceae0 2024-03-22T12:57:32.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.798+0000 7f8e5b400640 1 -- 172.21.15.6:0/3779039243 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0423f0 con 0x7f8e640ceae0 2024-03-22T12:57:32.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.798+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6414ee90 con 0x7f8e640ceae0 2024-03-22T12:57:32.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.798+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e641f3f80 con 0x7f8e640ceae0 2024-03-22T12:57:32.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.799+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e640ccf20 con 0x7f8e640ceae0 2024-03-22T12:57:32.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.822+0000 7f8e5b400640 1 -- 172.21.15.6:0/3779039243 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8e4c037070 con 0x7f8e640ceae0 2024-03-22T12:57:32.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.823+0000 7f8e5b400640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e4006bc20 0x7f8e4006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:32.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.823+0000 7f8e63400640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e4006bc20 0x7f8e4006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:32.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:32.824+0000 7f8e63400640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e4006bc20 0x7f8e4006e0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c005c00 tx=0x7f8e5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:33.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.076+0000 7f8e5b400640 1 -- 172.21.15.6:0/3779039243 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f8e4c0307b0 con 0x7f8e640ceae0 2024-03-22T12:57:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.096+0000 7f8e5b400640 1 -- 172.21.15.6:0/3779039243 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8e640ccf20 con 0x7f8e640ceae0 2024-03-22T12:57:33.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.307+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e640757d0 con 0x7f8e640ceae0 2024-03-22T12:57:33.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.402+0000 7f8e5b400640 1 -- 172.21.15.6:0/3779039243 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e4c07b950 con 0x7f8e640ceae0 2024-03-22T12:57:33.402 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.404+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e4006bc20 msgr2=0x7f8e4006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.404+0000 7f8e6a600640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e4006bc20 0x7f8e4006e0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c005c00 tx=0x7f8e5c009040 comp rx=0 tx=0).stop 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.404+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e640ceae0 msgr2=0x7f8e6414b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.404+0000 7f8e6a600640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e640ceae0 0x7f8e6414b770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c033000 tx=0x7f8e4c0046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 shutdown_connections 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e640cf660 0x7f8e64151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e4006bc20 0x7f8e4006e0e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e640ceae0 0x7f8e6414b770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 --2- 172.21.15.6:0/3779039243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e640caa80 0x7f8e640b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 >> 172.21.15.6:0/3779039243 conn(0x7f8e640b16d0 msgr2=0x7f8e64158e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 shutdown_connections 2024-03-22T12:57:33.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.405+0000 7f8e6a600640 1 -- 172.21.15.6:0/3779039243 wait complete. 2024-03-22T12:57:33.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T12:57:33.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.608+0000 7f5031600640 1 Processor -- start 2024-03-22T12:57:33.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.608+0000 7f5031600640 1 -- start start 2024-03-22T12:57:33.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.609+0000 7f5031600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0cf660 0x7f502c0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:33.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.609+0000 7f5031600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502c0ccf20 con 0x7f502c0ceae0 2024-03-22T12:57:33.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.609+0000 7f5031600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502c15a6b0 con 0x7f502c0cdf00 2024-03-22T12:57:33.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.610+0000 7f5031600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0cfa40 con 0x7f502c0cf660 2024-03-22T12:57:33.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.610+0000 7f502b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f502c0ceae0 0x7f502c0cc3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55158/0 (socket says 172.21.15.6:55158) 2024-03-22T12:57:33.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.610+0000 7f502b400640 1 -- 172.21.15.6:0/2907499631 learned_addr learned my addr 172.21.15.6:0/2907499631 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:33.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.610+0000 7f502aa00640 1 --2- 172.21.15.6:0/2907499631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0cf660 0x7f502c0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:33.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.613+0000 7f502aa00640 1 -- 172.21.15.6:0/2907499631 >> v1:172.21.15.92:6789/0 conn(0x7f502c0cdf00 legacy=0x7f502c0ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:33.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.613+0000 7f502aa00640 1 -- 172.21.15.6:0/2907499631 >> v1:172.21.15.6:6789/0 conn(0x7f502c0ceae0 legacy=0x7f502c0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:33.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.613+0000 7f502aa00640 1 -- 172.21.15.6:0/2907499631 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c15a990 con 0x7f502c0cf660 2024-03-22T12:57:33.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.616+0000 7f502aa00640 1 --2- 172.21.15.6:0/2907499631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0cf660 0x7f502c0cd9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f501c0097d0 tx=0x7f501c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92b0af1150aaf192 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:33.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.620+0000 7f5029600640 1 -- 172.21.15.6:0/2907499631 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c034070 con 0x7f502c0cf660 2024-03-22T12:57:33.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.621+0000 7f5029600640 1 -- 172.21.15.6:0/2907499631 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c030880 con 0x7f502c0cf660 2024-03-22T12:57:33.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.621+0000 7f5029600640 1 -- 172.21.15.6:0/2907499631 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c030c50 con 0x7f502c0cf660 2024-03-22T12:57:33.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.621+0000 7f5031600640 1 -- 172.21.15.6:0/2907499631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0cf660 msgr2=0x7f502c0cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:33.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.621+0000 7f5031600640 1 --2- 172.21.15.6:0/2907499631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0cf660 0x7f502c0cd9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f501c0097d0 tx=0x7f501c02b9c0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.622+0000 7f5031600640 1 -- 172.21.15.6:0/2907499631 shutdown_connections 2024-03-22T12:57:33.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.622+0000 7f5031600640 1 --2- 172.21.15.6:0/2907499631 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0cf660 0x7f502c0cd9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.622+0000 7f5031600640 1 -- 172.21.15.6:0/2907499631 >> 172.21.15.6:0/2907499631 conn(0x7f502c0b16d0 msgr2=0x7f502c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:33.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.622+0000 7f5031600640 1 -- 172.21.15.6:0/2907499631 shutdown_connections 2024-03-22T12:57:33.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.622+0000 7f5031600640 1 -- 172.21.15.6:0/2907499631 wait complete. 2024-03-22T12:57:33.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.624+0000 7f5031600640 1 Processor -- start 2024-03-22T12:57:33.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.625+0000 7f5031600640 1 -- start start 2024-03-22T12:57:33.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.626+0000 7f5031600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdf00 0x7f502c1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:33.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.626+0000 7f5031600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0ceae0 0x7f502c0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:33.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.626+0000 7f5031600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f502c0cf660 0x7f502c14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:33.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.626+0000 7f5031600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c15afe0 con 0x7f502c0cdf00 2024-03-22T12:57:33.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.626+0000 7f5031600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c15b470 con 0x7f502c0cf660 2024-03-22T12:57:33.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.626+0000 7f5031600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0cac50 con 0x7f502c0ceae0 2024-03-22T12:57:33.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.627+0000 7f502b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f502c0cf660 0x7f502c14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:33.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.627+0000 7f502a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0ceae0 0x7f502c0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:33.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.627+0000 7f502aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdf00 0x7f502c1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:33.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.627+0000 7f502b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f502c0cf660 0x7f502c14d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:45180/0 (socket says 172.21.15.6:45180) 2024-03-22T12:57:33.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.627+0000 7f502b400640 1 -- 172.21.15.6:0/1997295706 learned_addr learned my addr 172.21.15.6:0/1997295706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.631+0000 7f502b400640 1 -- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0ceae0 msgr2=0x7f502c0c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.631+0000 7f502b400640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0ceae0 0x7f502c0c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.631+0000 7f502b400640 1 -- 172.21.15.6:0/1997295706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdf00 msgr2=0x7f502c1575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.631+0000 7f502b400640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdf00 0x7f502c1575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:33.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.631+0000 7f502b400640 1 -- 172.21.15.6:0/1997295706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c154400 con 0x7f502c0cf660 2024-03-22T12:57:33.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.657+0000 7f502b400640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f502c0cf660 0x7f502c14d540 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f502400ca80 tx=0x7f502400cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.669+0000 7f5022a00640 1 -- 172.21.15.6:0/1997295706 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024017070 con 0x7f502c0cf660 2024-03-22T12:57:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.669+0000 7f5022a00640 1 -- 172.21.15.6:0/1997295706 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502400abd0 con 0x7f502c0cf660 2024-03-22T12:57:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.669+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c1f3670 con 0x7f502c0cf660 2024-03-22T12:57:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.669+0000 7f5022a00640 1 -- 172.21.15.6:0/1997295706 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024012400 con 0x7f502c0cf660 2024-03-22T12:57:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.669+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502c1f3930 con 0x7f502c0cf660 2024-03-22T12:57:33.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.671+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502c0757d0 con 0x7f502c0cf660 2024-03-22T12:57:33.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.696+0000 7f5022a00640 1 -- 172.21.15.6:0/1997295706 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f502401a070 con 0x7f502c0cf660 2024-03-22T12:57:33.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.697+0000 7f5022a00640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f500c06bc20 0x7f500c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:33.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.698+0000 7f5022a00640 1 -- 172.21.15.6:0/1997295706 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f5024057140 con 0x7f502c0cf660 2024-03-22T12:57:33.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.698+0000 7f502aa00640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f500c06bc20 0x7f500c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:33.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.699+0000 7f502aa00640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f500c06bc20 0x7f500c06e0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f501c036000 tx=0x7f501c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:33.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:33.968+0000 7f5022a00640 1 -- 172.21.15.6:0/1997295706 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f502c154400 con 0x7f502c0cf660 2024-03-22T12:57:34.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.197+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f502c06dda0 con 0x7f502c0cf660 2024-03-22T12:57:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.202+0000 7f5022a00640 1 -- 172.21.15.6:0/1997295706 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50240590a0 con 0x7f502c0cf660 2024-03-22T12:57:34.202 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:57:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f500c06bc20 msgr2=0x7f500c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f500c06bc20 0x7f500c06e0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f501c036000 tx=0x7f501c03b040 comp rx=0 tx=0).stop 2024-03-22T12:57:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f502c0cf660 msgr2=0x7f502c14d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f502c0cf660 0x7f502c14d540 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f502400ca80 tx=0x7f502400cf40 comp rx=0 tx=0).stop 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 shutdown_connections 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f500c06bc20 0x7f500c06e0e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f502c0cf660 0x7f502c14d540 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f502c0ceae0 0x7f502c0c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 --2- 172.21.15.6:0/1997295706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdf00 0x7f502c1575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.205+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 >> 172.21.15.6:0/1997295706 conn(0x7f502c0b16d0 msgr2=0x7f502c06c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.206+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 shutdown_connections 2024-03-22T12:57:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.206+0000 7f5031600640 1 -- 172.21.15.6:0/1997295706 wait complete. 2024-03-22T12:57:34.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T12:57:34.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.442+0000 7ff2b8a00640 1 Processor -- start 2024-03-22T12:57:34.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.442+0000 7ff2b8a00640 1 -- start start 2024-03-22T12:57:34.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.443+0000 7ff2b8a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff2b4154410 0x7ff2b40ca640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:34.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.443+0000 7ff2b8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415add0 0x7ff2b40c8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:34.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.443+0000 7ff2b8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b41547f0 con 0x7ff2b415d900 2024-03-22T12:57:34.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.443+0000 7ff2b8a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b415a310 con 0x7ff2b4154410 2024-03-22T12:57:34.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.443+0000 7ff2b8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b40cac80 con 0x7ff2b415add0 2024-03-22T12:57:34.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.444+0000 7ff2b2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415add0 0x7ff2b40c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:34.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.445+0000 7ff2b2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415add0 0x7ff2b40c8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45922/0 (socket says 172.21.15.6:45922) 2024-03-22T12:57:34.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.445+0000 7ff2b2800640 1 -- 172.21.15.6:0/1915164121 learned_addr learned my addr 172.21.15.6:0/1915164121 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:34.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.448+0000 7ff2b2800640 1 -- 172.21.15.6:0/1915164121 >> v2:172.21.15.92:3300/0 conn(0x7ff2b4154410 msgr2=0x7ff2b40ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:34.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.448+0000 7ff2b2800640 1 --2- 172.21.15.6:0/1915164121 >> v2:172.21.15.92:3300/0 conn(0x7ff2b4154410 0x7ff2b40ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.448+0000 7ff2b2800640 1 -- 172.21.15.6:0/1915164121 >> v1:172.21.15.6:6789/0 conn(0x7ff2b415d900 legacy=0x7ff2b40c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:34.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.448+0000 7ff2b2800640 1 -- 172.21.15.6:0/1915164121 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b40caf60 con 0x7ff2b415add0 2024-03-22T12:57:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.451+0000 7ff2b2800640 1 --2- 172.21.15.6:0/1915164121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415add0 0x7ff2b40c8960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4154bd0 tx=0x7ff2ac02bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1135c681fe203ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:34.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.457+0000 7ff2b0a00640 1 -- 172.21.15.6:0/1915164121 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac02cae0 con 0x7ff2b415add0 2024-03-22T12:57:34.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.457+0000 7ff2b0a00640 1 -- 172.21.15.6:0/1915164121 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ac02cc80 con 0x7ff2b415add0 2024-03-22T12:57:34.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.457+0000 7ff2b0a00640 1 -- 172.21.15.6:0/1915164121 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac034610 con 0x7ff2b415add0 2024-03-22T12:57:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1915164121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415add0 msgr2=0x7ff2b40c8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1915164121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415add0 0x7ff2b40c8960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4154bd0 tx=0x7ff2ac02bf40 comp rx=0 tx=0).stop 2024-03-22T12:57:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1915164121 shutdown_connections 2024-03-22T12:57:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1915164121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415add0 0x7ff2b40c8960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1915164121 >> v2:172.21.15.92:3300/0 conn(0x7ff2b4154410 0x7ff2b40ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1915164121 >> 172.21.15.6:0/1915164121 conn(0x7ff2b40714a0 msgr2=0x7ff2b4071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1915164121 shutdown_connections 2024-03-22T12:57:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.458+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1915164121 wait complete. 2024-03-22T12:57:34.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.460+0000 7ff2b8a00640 1 Processor -- start 2024-03-22T12:57:34.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.460+0000 7ff2b8a00640 1 -- start start 2024-03-22T12:57:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.461+0000 7ff2b8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2b4154410 0x7ff2b41529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.461+0000 7ff2b8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2b415add0 0x7ff2b40cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.461+0000 7ff2b8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415d900 0x7ff2b415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.461+0000 7ff2b8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b415a8b0 con 0x7ff2b415add0 2024-03-22T12:57:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.461+0000 7ff2b8a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b415a5d0 con 0x7ff2b4154410 2024-03-22T12:57:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.461+0000 7ff2b8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b415b200 con 0x7ff2b415d900 2024-03-22T12:57:34.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.462+0000 7ff2b2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415d900 0x7ff2b415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:34.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.463+0000 7ff2b2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415d900 0x7ff2b415bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45936/0 (socket says 172.21.15.6:45936) 2024-03-22T12:57:34.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.463+0000 7ff2b2800640 1 -- 172.21.15.6:0/1670594045 learned_addr learned my addr 172.21.15.6:0/1670594045 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:34.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.468+0000 7ff2b2800640 1 -- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2b4154410 msgr2=0x7ff2b41529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:34.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.468+0000 7ff2b2800640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2b4154410 0x7ff2b41529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.468+0000 7ff2b2800640 1 -- 172.21.15.6:0/1670594045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2b415add0 msgr2=0x7ff2b40cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:34.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.468+0000 7ff2b2800640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2b415add0 0x7ff2b40cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:34.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.468+0000 7ff2b2800640 1 -- 172.21.15.6:0/1670594045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b41f3c70 con 0x7ff2b415d900 2024-03-22T12:57:34.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.471+0000 7ff2b2800640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415d900 0x7ff2b415bb90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac036040 tx=0x7ff2ac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:34.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.477+0000 7ff2aa000640 1 -- 172.21.15.6:0/1670594045 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac043020 con 0x7ff2b415d900 2024-03-22T12:57:34.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.477+0000 7ff2aa000640 1 -- 172.21.15.6:0/1670594045 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ac03e450 con 0x7ff2b415d900 2024-03-22T12:57:34.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.477+0000 7ff2aa000640 1 -- 172.21.15.6:0/1670594045 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac03e820 con 0x7ff2b415d900 2024-03-22T12:57:34.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.477+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b41f3f50 con 0x7ff2b415d900 2024-03-22T12:57:34.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.477+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b41f4510 con 0x7ff2b415d900 2024-03-22T12:57:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.478+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b41f46e0 con 0x7ff2b415d900 2024-03-22T12:57:34.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.514+0000 7ff2aa000640 1 -- 172.21.15.6:0/1670594045 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff2ac046020 con 0x7ff2b415d900 2024-03-22T12:57:34.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.516+0000 7ff2aa000640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff29406bc20 0x7ff29406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:34.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.517+0000 7ff2b1e00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff29406bc20 0x7ff29406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.517+0000 7ff2b1e00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff29406bc20 0x7ff29406e0e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009710 tx=0x7ff2a4006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:34.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.520+0000 7ff2aa000640 1 -- 172.21.15.6:0/1670594045 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7ff2ac07a070 con 0x7ff2b415d900 2024-03-22T12:57:34.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:34.834+0000 7ff2aa000640 1 -- 172.21.15.6:0/1670594045 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff2ac04d030 con 0x7ff2b415d900 2024-03-22T12:57:35.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.040+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2b40757d0 con 0x7ff2b415d900 2024-03-22T12:57:35.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.049+0000 7ff2aa000640 1 -- 172.21.15.6:0/1670594045 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff2b41547f0 con 0x7ff2b415d900 2024-03-22T12:57:35.049 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.052+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff29406bc20 msgr2=0x7ff29406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.052+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff29406bc20 0x7ff29406e0e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009710 tx=0x7ff2a4006c90 comp rx=0 tx=0).stop 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.052+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415d900 msgr2=0x7ff2b415bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.052+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415d900 0x7ff2b415bb90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac036040 tx=0x7ff2ac004a40 comp rx=0 tx=0).stop 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 shutdown_connections 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2b415d900 0x7ff2b415bb90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2b415add0 0x7ff2b40cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff29406bc20 0x7ff29406e0e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 --2- 172.21.15.6:0/1670594045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2b4154410 0x7ff2b41529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 >> 172.21.15.6:0/1670594045 conn(0x7ff2b40714a0 msgr2=0x7ff2b40744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 shutdown_connections 2024-03-22T12:57:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.053+0000 7ff2b8a00640 1 -- 172.21.15.6:0/1670594045 wait complete. 2024-03-22T12:57:35.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T12:57:35.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.266+0000 7fd25e400640 1 Processor -- start 2024-03-22T12:57:35.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.267+0000 7fd25e400640 1 -- start start 2024-03-22T12:57:35.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.268+0000 7fd25e400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd2580c0ea0 0x7fd258071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:35.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.268+0000 7fd25e400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd258158c20 0x7fd2580c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:35.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.268+0000 7fd25e400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd258159070 con 0x7fd258152280 2024-03-22T12:57:35.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.268+0000 7fd25e400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd258158790 con 0x7fd258158c20 2024-03-22T12:57:35.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.268+0000 7fd25e400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2580c1450 con 0x7fd2580c0ea0 2024-03-22T12:57:35.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.269+0000 7fd256a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd2580c0ea0 0x7fd258071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:35.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.269+0000 7fd257e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd258158c20 0x7fd2580c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:35.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.270+0000 7fd256a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd2580c0ea0 0x7fd258071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:45210/0 (socket says 172.21.15.6:45210) 2024-03-22T12:57:35.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.270+0000 7fd256a00640 1 -- 172.21.15.6:0/3872879704 learned_addr learned my addr 172.21.15.6:0/3872879704 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:35.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.270+0000 7fd257400640 1 --1- 172.21.15.6:0/3872879704 >> v1:172.21.15.6:6789/0 conn(0x7fd258152280 0x7fd25806b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55172/0 (socket says 172.21.15.6:55172) 2024-03-22T12:57:35.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.272+0000 7fd256a00640 1 -- 172.21.15.6:0/3872879704 >> v2:172.21.15.6:3300/0 conn(0x7fd258158c20 msgr2=0x7fd2580c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:35.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.272+0000 7fd256a00640 1 --2- 172.21.15.6:0/3872879704 >> v2:172.21.15.6:3300/0 conn(0x7fd258158c20 0x7fd2580c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.272+0000 7fd256a00640 1 -- 172.21.15.6:0/3872879704 >> v1:172.21.15.6:6789/0 conn(0x7fd258152280 legacy=0x7fd25806b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.272+0000 7fd256a00640 1 -- 172.21.15.6:0/3872879704 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580c1730 con 0x7fd2580c0ea0 2024-03-22T12:57:35.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.274+0000 7fd256a00640 1 --2- 172.21.15.6:0/3872879704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2580c0ea0 0x7fd258071920 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2580c27a0 tx=0x7fd24402b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f38fedb10e61ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:35.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.276+0000 7fd256000640 1 -- 172.21.15.6:0/3872879704 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244035020 con 0x7fd2580c0ea0 2024-03-22T12:57:35.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.278+0000 7fd256000640 1 -- 172.21.15.6:0/3872879704 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24402d040 con 0x7fd2580c0ea0 2024-03-22T12:57:35.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.278+0000 7fd256000640 1 -- 172.21.15.6:0/3872879704 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244031a80 con 0x7fd2580c0ea0 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 -- 172.21.15.6:0/3872879704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2580c0ea0 msgr2=0x7fd258071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 --2- 172.21.15.6:0/3872879704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2580c0ea0 0x7fd258071920 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2580c27a0 tx=0x7fd24402b900 comp rx=0 tx=0).stop 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 -- 172.21.15.6:0/3872879704 shutdown_connections 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 --2- 172.21.15.6:0/3872879704 >> v2:172.21.15.6:3300/0 conn(0x7fd258158c20 0x7fd2580c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 --2- 172.21.15.6:0/3872879704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2580c0ea0 0x7fd258071920 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 -- 172.21.15.6:0/3872879704 >> 172.21.15.6:0/3872879704 conn(0x7fd25806e310 msgr2=0x7fd25806f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 -- 172.21.15.6:0/3872879704 shutdown_connections 2024-03-22T12:57:35.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.279+0000 7fd25e400640 1 -- 172.21.15.6:0/3872879704 wait complete. 2024-03-22T12:57:35.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.281+0000 7fd25e400640 1 Processor -- start 2024-03-22T12:57:35.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.282+0000 7fd25e400640 1 -- start start 2024-03-22T12:57:35.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.283+0000 7fd25e400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2580c0ea0 0x7fd25814cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.283+0000 7fd25e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd258152280 0x7fd2580725c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.283+0000 7fd25e400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd258158c20 0x7fd2580c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.283+0000 7fd25e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2581580d0 con 0x7fd258152280 2024-03-22T12:57:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.283+0000 7fd25e400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd258158420 con 0x7fd258158c20 2024-03-22T12:57:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.283+0000 7fd25e400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd258152660 con 0x7fd2580c0ea0 2024-03-22T12:57:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.283+0000 7fd256a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd258152280 0x7fd2580725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.284+0000 7fd257e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd258158c20 0x7fd2580c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.284+0000 7fd257400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2580c0ea0 0x7fd25814cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.284+0000 7fd256a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd258152280 0x7fd2580725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42372/0 (socket says 172.21.15.6:42372) 2024-03-22T12:57:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.284+0000 7fd256a00640 1 -- 172.21.15.6:0/2588713925 learned_addr learned my addr 172.21.15.6:0/2588713925 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.287+0000 7fd257e00640 1 -- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2580c0ea0 msgr2=0x7fd25814cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.287+0000 7fd257e00640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2580c0ea0 0x7fd25814cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.287+0000 7fd257e00640 1 -- 172.21.15.6:0/2588713925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd258152280 msgr2=0x7fd2580725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.287+0000 7fd257e00640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd258152280 0x7fd2580725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.287+0000 7fd257e00640 1 -- 172.21.15.6:0/2588713925 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580c5060 con 0x7fd258158c20 2024-03-22T12:57:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.290+0000 7fd257e00640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd258158c20 0x7fd2580c3040 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd25000ec00 tx=0x7fd25000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.294+0000 7fd24f400640 1 -- 172.21.15.6:0/2588713925 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250018020 con 0x7fd258158c20 2024-03-22T12:57:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.294+0000 7fd24f400640 1 -- 172.21.15.6:0/2588713925 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25000cd40 con 0x7fd258158c20 2024-03-22T12:57:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.294+0000 7fd24f400640 1 -- 172.21.15.6:0/2588713925 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2500146c0 con 0x7fd258158c20 2024-03-22T12:57:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.294+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25814c720 con 0x7fd258158c20 2024-03-22T12:57:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.294+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2580c4d80 con 0x7fd258158c20 2024-03-22T12:57:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.295+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25814f810 con 0x7fd258158c20 2024-03-22T12:57:35.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.328+0000 7fd24f400640 1 -- 172.21.15.6:0/2588713925 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd250005d10 con 0x7fd258158c20 2024-03-22T12:57:35.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.329+0000 7fd24f400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd23806bc20 0x7fd23806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:35.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.329+0000 7fd257400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd23806bc20 0x7fd23806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:35.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.330+0000 7fd257400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd23806bc20 0x7fd23806e0e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd248007fd0 tx=0x7fd248007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:35.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.332+0000 7fd24f400640 1 -- 172.21.15.6:0/2588713925 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fd2500580a0 con 0x7fd258158c20 2024-03-22T12:57:35.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.578+0000 7fd24f400640 1 -- 172.21.15.6:0/2588713925 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd25814f810 con 0x7fd258158c20 2024-03-22T12:57:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.823+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd258073460 con 0x7fd258158c20 2024-03-22T12:57:35.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.829+0000 7fd24f400640 1 -- 172.21.15.6:0/2588713925 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v30) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd258073460 con 0x7fd258158c20 2024-03-22T12:57:35.829 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-22T12:57:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd23806bc20 msgr2=0x7fd23806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd23806bc20 0x7fd23806e0e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd248007fd0 tx=0x7fd248007e80 comp rx=0 tx=0).stop 2024-03-22T12:57:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd258158c20 msgr2=0x7fd2580c3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd258158c20 0x7fd2580c3040 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd25000ec00 tx=0x7fd25000c490 comp rx=0 tx=0).stop 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 shutdown_connections 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd23806bc20 0x7fd23806e0e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd258158c20 0x7fd2580c3040 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd258152280 0x7fd2580725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 --2- 172.21.15.6:0/2588713925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2580c0ea0 0x7fd25814cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.832+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 >> 172.21.15.6:0/2588713925 conn(0x7fd25806e310 msgr2=0x7fd258157830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.833+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 shutdown_connections 2024-03-22T12:57:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:35.833+0000 7fd25e400640 1 -- 172.21.15.6:0/2588713925 wait complete. 2024-03-22T12:57:35.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:57:36.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.036+0000 7f6f88c00640 1 Processor -- start 2024-03-22T12:57:36.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.036+0000 7f6f88c00640 1 -- start start 2024-03-22T12:57:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.038+0000 7f6f88c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6f840ccad0 0x7f6f8406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.038+0000 7f6f88c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f8415b0e0 0x7f6f8415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.038+0000 7f6f88c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8406c7a0 con 0x7f6f840ca520 2024-03-22T12:57:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.038+0000 7f6f88c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f840cd000 con 0x7f6f8415b0e0 2024-03-22T12:57:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.038+0000 7f6f88c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8406ca80 con 0x7f6f840ccad0 2024-03-22T12:57:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.039+0000 7f6f79400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6f840ccad0 0x7f6f8406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.039+0000 7f6f81e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6f840ca520 0x7f6f840cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55178/0 (socket says 172.21.15.6:55178) 2024-03-22T12:57:36.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.039+0000 7f6f81e00640 1 -- 172.21.15.6:0/2829587530 learned_addr learned my addr 172.21.15.6:0/2829587530 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:36.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.040+0000 7f6f82800640 1 --2- 172.21.15.6:0/2829587530 >> v2:172.21.15.6:3300/0 conn(0x7f6f8415b0e0 0x7f6f8415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.043+0000 7f6f81400640 1 -- 172.21.15.6:0/2829587530 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489083053 0 0) 0x7f6f8406c7a0 con 0x7f6f840ca520 2024-03-22T12:57:36.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.043+0000 7f6f81400640 1 -- 172.21.15.6:0/2829587530 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f68003520 con 0x7f6f840ca520 2024-03-22T12:57:36.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.043+0000 7f6f79400640 1 -- 172.21.15.6:0/2829587530 >> v2:172.21.15.6:3300/0 conn(0x7f6f8415b0e0 msgr2=0x7f6f8415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:36.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.043+0000 7f6f79400640 1 --2- 172.21.15.6:0/2829587530 >> v2:172.21.15.6:3300/0 conn(0x7f6f8415b0e0 0x7f6f8415b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.043+0000 7f6f79400640 1 -- 172.21.15.6:0/2829587530 >> v1:172.21.15.6:6789/0 conn(0x7f6f840ca520 legacy=0x7f6f840cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.043+0000 7f6f79400640 1 -- 172.21.15.6:0/2829587530 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8406cd60 con 0x7f6f840ccad0 2024-03-22T12:57:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.046+0000 7f6f79400640 1 --2- 172.21.15.6:0/2829587530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ccad0 0x7f6f8406d840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6f840b2490 tx=0x7f6f6c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92901123966c80a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.050+0000 7f6f81400640 1 -- 172.21.15.6:0/2829587530 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c035020 con 0x7f6f840ccad0 2024-03-22T12:57:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.050+0000 7f6f81400640 1 -- 172.21.15.6:0/2829587530 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c031830 con 0x7f6f840ccad0 2024-03-22T12:57:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.050+0000 7f6f81400640 1 -- 172.21.15.6:0/2829587530 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c031c00 con 0x7f6f840ccad0 2024-03-22T12:57:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.051+0000 7f6f88c00640 1 -- 172.21.15.6:0/2829587530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ccad0 msgr2=0x7f6f8406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.051+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2829587530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ccad0 0x7f6f8406d840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6f840b2490 tx=0x7f6f6c02b8d0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.051+0000 7f6f88c00640 1 -- 172.21.15.6:0/2829587530 shutdown_connections 2024-03-22T12:57:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.051+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2829587530 >> v2:172.21.15.6:3300/0 conn(0x7f6f8415b0e0 0x7f6f8415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.051+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2829587530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ccad0 0x7f6f8406d840 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.051+0000 7f6f88c00640 1 -- 172.21.15.6:0/2829587530 >> 172.21.15.6:0/2829587530 conn(0x7f6f840b16b0 msgr2=0x7f6f840b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.051+0000 7f6f88c00640 1 -- 172.21.15.6:0/2829587530 shutdown_connections 2024-03-22T12:57:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.052+0000 7f6f88c00640 1 -- 172.21.15.6:0/2829587530 wait complete. 2024-03-22T12:57:36.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.053+0000 7f6f88c00640 1 Processor -- start 2024-03-22T12:57:36.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.053+0000 7f6f88c00640 1 -- start start 2024-03-22T12:57:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.054+0000 7f6f88c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ca520 0x7f6f841e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.054+0000 7f6f88c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f840ccad0 0x7f6f841e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.054+0000 7f6f88c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6f8415b0e0 0x7f6f8414ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.054+0000 7f6f88c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84154c20 con 0x7f6f840ccad0 2024-03-22T12:57:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.054+0000 7f6f88c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8406d320 con 0x7f6f840ca520 2024-03-22T12:57:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.054+0000 7f6f88c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f840ca900 con 0x7f6f8415b0e0 2024-03-22T12:57:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.055+0000 7f6f79400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f840ccad0 0x7f6f841e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.055+0000 7f6f81e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ca520 0x7f6f841e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.055+0000 7f6f82800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6f8415b0e0 0x7f6f8414ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.055+0000 7f6f79400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f840ccad0 0x7f6f841e6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42388/0 (socket says 172.21.15.6:42388) 2024-03-22T12:57:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.055+0000 7f6f79400640 1 -- 172.21.15.6:0/2089586862 learned_addr learned my addr 172.21.15.6:0/2089586862 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.059+0000 7f6f79400640 1 -- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6f8415b0e0 msgr2=0x7f6f8414ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.059+0000 7f6f79400640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6f8415b0e0 0x7f6f8414ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.059+0000 7f6f79400640 1 -- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ca520 msgr2=0x7f6f841e6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.059+0000 7f6f79400640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ca520 0x7f6f841e6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.059+0000 7f6f79400640 1 -- 172.21.15.6:0/2089586862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f84153d60 con 0x7f6f840ccad0 2024-03-22T12:57:36.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.062+0000 7f6f79400640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f840ccad0 0x7f6f841e6fe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c009890 tx=0x7f6f6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.065+0000 7f6f7aa00640 1 -- 172.21.15.6:0/2089586862 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c035040 con 0x7f6f840ccad0 2024-03-22T12:57:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.065+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f84154000 con 0x7f6f840ccad0 2024-03-22T12:57:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.065+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f841545a0 con 0x7f6f840ccad0 2024-03-22T12:57:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.066+0000 7f6f7aa00640 1 -- 172.21.15.6:0/2089586862 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c031780 con 0x7f6f840ccad0 2024-03-22T12:57:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.066+0000 7f6f7aa00640 1 -- 172.21.15.6:0/2089586862 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c03bb00 con 0x7f6f840ccad0 2024-03-22T12:57:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.067+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c005290 con 0x7f6f840ccad0 2024-03-22T12:57:36.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.089+0000 7f6f7aa00640 1 -- 172.21.15.6:0/2089586862 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6f6c038020 con 0x7f6f840ccad0 2024-03-22T12:57:36.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.090+0000 7f6f7aa00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6f6006bbd0 0x7f6f6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.091+0000 7f6f81e00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6f6006bbd0 0x7f6f6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.091+0000 7f6f81e00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6f6006bbd0 0x7f6f6006e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f74005880 tx=0x7f6f74009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:36.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.094+0000 7f6f7aa00640 1 -- 172.21.15.6:0/2089586862 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f6f6c079480 con 0x7f6f840ccad0 2024-03-22T12:57:36.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.378+0000 7f6f7aa00640 1 -- 172.21.15.6:0/2089586862 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6f6c054080 con 0x7f6f840ccad0 2024-03-22T12:57:36.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.622+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f5c005580 con 0x7f6f840ccad0 2024-03-22T12:57:36.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.714+0000 7f6f7aa00640 1 -- 172.21.15.6:0/2089586862 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6628 (secure 0 0 0) 0x7f6f6c07b3e0 con 0x7f6f840ccad0 2024-03-22T12:57:36.715 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:36.715 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:57:18.963149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4259,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4259":{"gid":4259,"name":"d","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6837/2825293724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2825293724},{"type":"v1","addr":"172.21.15.92:6837","nonce":2825293724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4259,4261,4268]},"id":1}]} 2024-03-22T12:57:36.715 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-03-22T12:57:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6f6006bbd0 msgr2=0x7f6f6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6f6006bbd0 0x7f6f6006e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f74005880 tx=0x7f6f74009040 comp rx=0 tx=0).stop 2024-03-22T12:57:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f840ccad0 msgr2=0x7f6f841e6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f840ccad0 0x7f6f841e6fe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c009890 tx=0x7f6f6c0046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 shutdown_connections 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6f8415b0e0 0x7f6f8414ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f840ccad0 0x7f6f841e6fe0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6f6006bbd0 0x7f6f6006e090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 --2- 172.21.15.6:0/2089586862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f840ca520 0x7f6f841e6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.718+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 >> 172.21.15.6:0/2089586862 conn(0x7f6f840b16b0 msgr2=0x7f6f840ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.719+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 shutdown_connections 2024-03-22T12:57:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.719+0000 7f6f88c00640 1 -- 172.21.15.6:0/2089586862 wait complete. 2024-03-22T12:57:36.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T12:57:36.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.939+0000 7fbf07e00640 1 Processor -- start 2024-03-22T12:57:36.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.939+0000 7fbf07e00640 1 -- start start 2024-03-22T12:57:36.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.941+0000 7fbf07e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbf000c7ee0 0x7fbf0015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.941+0000 7fbf07e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c68c0 0x7fbf0006b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.941+0000 7fbf07e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000c4740 con 0x7fbf00154240 2024-03-22T12:57:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.941+0000 7fbf07e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0015e130 con 0x7fbf000c7ee0 2024-03-22T12:57:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.941+0000 7fbf07e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00154620 con 0x7fbf000c68c0 2024-03-22T12:57:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.942+0000 7fbf04a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbf000c7ee0 0x7fbf0015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.942+0000 7fbf05e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c68c0 0x7fbf0006b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.942+0000 7fbf05400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbf00154240 0x7fbf000c4030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55182/0 (socket says 172.21.15.6:55182) 2024-03-22T12:57:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.942+0000 7fbf05400640 1 -- 172.21.15.6:0/1049971844 learned_addr learned my addr 172.21.15.6:0/1049971844 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.944+0000 7fbf05e00640 1 -- 172.21.15.6:0/1049971844 >> v2:172.21.15.92:3300/0 conn(0x7fbf000c7ee0 msgr2=0x7fbf0015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.944+0000 7fbf05e00640 1 --2- 172.21.15.6:0/1049971844 >> v2:172.21.15.92:3300/0 conn(0x7fbf000c7ee0 0x7fbf0015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.944+0000 7fbf05e00640 1 -- 172.21.15.6:0/1049971844 >> v1:172.21.15.6:6789/0 conn(0x7fbf00154240 legacy=0x7fbf000c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.944+0000 7fbf05e00640 1 -- 172.21.15.6:0/1049971844 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000c6e00 con 0x7fbf000c68c0 2024-03-22T12:57:36.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.947+0000 7fbf05e00640 1 --2- 172.21.15.6:0/1049971844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c68c0 0x7fbf0006b080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2fcd7fc3f9fbb57 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.952+0000 7fbeffe00640 1 -- 172.21.15.6:0/1049971844 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8039070 con 0x7fbf000c68c0 2024-03-22T12:57:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.952+0000 7fbeffe00640 1 -- 172.21.15.6:0/1049971844 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef802cce0 con 0x7fbf000c68c0 2024-03-22T12:57:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.952+0000 7fbeffe00640 1 -- 172.21.15.6:0/1049971844 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8034660 con 0x7fbf000c68c0 2024-03-22T12:57:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.952+0000 7fbf07e00640 1 -- 172.21.15.6:0/1049971844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c68c0 msgr2=0x7fbf0006b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.952+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1049971844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c68c0 0x7fbf0006b080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef802bf30 comp rx=0 tx=0).stop 2024-03-22T12:57:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.953+0000 7fbf07e00640 1 -- 172.21.15.6:0/1049971844 shutdown_connections 2024-03-22T12:57:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.953+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1049971844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c68c0 0x7fbf0006b080 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.953+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1049971844 >> v2:172.21.15.92:3300/0 conn(0x7fbf000c7ee0 0x7fbf0015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.953+0000 7fbf07e00640 1 -- 172.21.15.6:0/1049971844 >> 172.21.15.6:0/1049971844 conn(0x7fbf000cb930 msgr2=0x7fbf000cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.953+0000 7fbf07e00640 1 -- 172.21.15.6:0/1049971844 shutdown_connections 2024-03-22T12:57:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.953+0000 7fbf07e00640 1 -- 172.21.15.6:0/1049971844 wait complete. 2024-03-22T12:57:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.954+0000 7fbf07e00640 1 Processor -- start 2024-03-22T12:57:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.955+0000 7fbf07e00640 1 -- start start 2024-03-22T12:57:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.955+0000 7fbf07e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf000c68c0 0x7fbf0015c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.955+0000 7fbf07e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c7ee0 0x7fbf001f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.955+0000 7fbf07e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf00154240 0x7fbf0014a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.955+0000 7fbf07e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0015dae0 con 0x7fbf00154240 2024-03-22T12:57:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.955+0000 7fbf07e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0015d800 con 0x7fbf000c68c0 2024-03-22T12:57:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.955+0000 7fbf07e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0015ddc0 con 0x7fbf000c7ee0 2024-03-22T12:57:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.956+0000 7fbf05e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf00154240 0x7fbf0014a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.956+0000 7fbf04a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c7ee0 0x7fbf001f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.956+0000 7fbf05400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf000c68c0 0x7fbf0015c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.957+0000 7fbf05400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf000c68c0 0x7fbf0015c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:45246/0 (socket says 172.21.15.6:45246) 2024-03-22T12:57:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.957+0000 7fbf05400640 1 -- 172.21.15.6:0/1938874198 learned_addr learned my addr 172.21.15.6:0/1938874198 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.960+0000 7fbf05400640 1 -- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c7ee0 msgr2=0x7fbf001f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.960+0000 7fbf05400640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c7ee0 0x7fbf001f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.960+0000 7fbf05400640 1 -- 172.21.15.6:0/1938874198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf00154240 msgr2=0x7fbf0014a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.960+0000 7fbf05400640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf00154240 0x7fbf0014a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.960+0000 7fbf05400640 1 -- 172.21.15.6:0/1938874198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf001e3620 con 0x7fbf000c68c0 2024-03-22T12:57:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.963+0000 7fbf05400640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf000c68c0 0x7fbf0015c0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c3c0 tx=0x7fbef400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:36.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.967+0000 7fbefd600640 1 -- 172.21.15.6:0/1938874198 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4017070 con 0x7fbf000c68c0 2024-03-22T12:57:36.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.967+0000 7fbefd600640 1 -- 172.21.15.6:0/1938874198 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef4005ce0 con 0x7fbf000c68c0 2024-03-22T12:57:36.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.967+0000 7fbefd600640 1 -- 172.21.15.6:0/1938874198 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef40072f0 con 0x7fbf000c68c0 2024-03-22T12:57:36.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.967+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0014b9f0 con 0x7fbf000c68c0 2024-03-22T12:57:36.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.967+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf001e5fa0 con 0x7fbf000c68c0 2024-03-22T12:57:36.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:36.968+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf000756c0 con 0x7fbf000c68c0 2024-03-22T12:57:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.227+0000 7fbefd600640 1 -- 172.21.15.6:0/1938874198 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbef401a070 con 0x7fbf000c68c0 2024-03-22T12:57:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.229+0000 7fbefd600640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbee006bc20 0x7fbee006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:37.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.229+0000 7fbf04a00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbee006bc20 0x7fbee006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.231+0000 7fbf04a00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbee006bc20 0x7fbee006e0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbf001edc70 tx=0x7fbeec009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:37.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.232+0000 7fbefd600640 1 -- 172.21.15.6:0/1938874198 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7fbef4056a50 con 0x7fbf000c68c0 2024-03-22T12:57:37.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.244+0000 7fbefd600640 1 -- 172.21.15.6:0/1938874198 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbef4021070 con 0x7fbf000c68c0 2024-03-22T12:57:37.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.457+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf001f3670 con 0x7fbf000c68c0 2024-03-22T12:57:37.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.501+0000 7fbefd600640 1 -- 172.21.15.6:0/1938874198 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18901 (secure 0 0 0) 0x7fbef40589b0 con 0x7fbf000c68c0 2024-03-22T12:57:37.501 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:37.501 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:57:04.100776+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbee006bc20 msgr2=0x7fbee006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbee006bc20 0x7fbee006e0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbf001edc70 tx=0x7fbeec009190 comp rx=0 tx=0).stop 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf000c68c0 msgr2=0x7fbf0015c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf000c68c0 0x7fbf0015c0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c3c0 tx=0x7fbef400c880 comp rx=0 tx=0).stop 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 shutdown_connections 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf00154240 0x7fbf0014a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf000c7ee0 0x7fbf001f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbee006bc20 0x7fbee006e0e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 --2- 172.21.15.6:0/1938874198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf000c68c0 0x7fbf0015c0c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 >> 172.21.15.6:0/1938874198 conn(0x7fbf000cb930 msgr2=0x7fbf0015b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 shutdown_connections 2024-03-22T12:57:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.505+0000 7fbf07e00640 1 -- 172.21.15.6:0/1938874198 wait complete. 2024-03-22T12:57:37.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:57:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.715+0000 7f3975200640 1 Processor -- start 2024-03-22T12:57:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.715+0000 7f3975200640 1 -- start start 2024-03-22T12:57:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.716+0000 7f3975200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f397006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.717+0000 7f3975200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f397015ae30 0x7f39700cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.717+0000 7f3975200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397015b260 con 0x7f39700ce8a0 2024-03-22T12:57:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.717+0000 7f3975200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39700cee00 con 0x7f397015ae30 2024-03-22T12:57:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.717+0000 7f3975200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39700cf0c0 con 0x7f39700cc380 2024-03-22T12:57:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.717+0000 7f396ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f39700ce8a0 0x7f39700cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:42826/0 (socket says 172.21.15.6:42826) 2024-03-22T12:57:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.717+0000 7f396ea00640 1 -- 172.21.15.6:0/3255754593 learned_addr learned my addr 172.21.15.6:0/3255754593 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:57:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.718+0000 7f396f400640 1 --2- 172.21.15.6:0/3255754593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f397006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.721+0000 7f396f400640 1 -- 172.21.15.6:0/3255754593 >> v2:172.21.15.92:3300/0 conn(0x7f397015ae30 msgr2=0x7f39700cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.721+0000 7f396f400640 1 --2- 172.21.15.6:0/3255754593 >> v2:172.21.15.92:3300/0 conn(0x7f397015ae30 0x7f39700cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.721+0000 7f396f400640 1 -- 172.21.15.6:0/3255754593 >> v1:172.21.15.92:6789/0 conn(0x7f39700ce8a0 legacy=0x7f39700cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.721+0000 7f396f400640 1 -- 172.21.15.6:0/3255754593 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700cf3a0 con 0x7f39700cc380 2024-03-22T12:57:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.724+0000 7f396f400640 1 --2- 172.21.15.6:0/3255754593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f397006c230 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f39600097a0 tx=0x7f396002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c2cd2441490410b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.729+0000 7f396e000640 1 -- 172.21.15.6:0/3255754593 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960035020 con 0x7f39700cc380 2024-03-22T12:57:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.729+0000 7f396e000640 1 -- 172.21.15.6:0/3255754593 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3960031830 con 0x7f39700cc380 2024-03-22T12:57:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.729+0000 7f396e000640 1 -- 172.21.15.6:0/3255754593 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960031c00 con 0x7f39700cc380 2024-03-22T12:57:37.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 -- 172.21.15.6:0/3255754593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 msgr2=0x7f397006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:37.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 --2- 172.21.15.6:0/3255754593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f397006c230 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f39600097a0 tx=0x7f396002b8d0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 -- 172.21.15.6:0/3255754593 shutdown_connections 2024-03-22T12:57:37.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 --2- 172.21.15.6:0/3255754593 >> v2:172.21.15.92:3300/0 conn(0x7f397015ae30 0x7f39700cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 --2- 172.21.15.6:0/3255754593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f397006c230 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 -- 172.21.15.6:0/3255754593 >> 172.21.15.6:0/3255754593 conn(0x7f39700b1480 msgr2=0x7f39700b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:37.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 -- 172.21.15.6:0/3255754593 shutdown_connections 2024-03-22T12:57:37.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.730+0000 7f3975200640 1 -- 172.21.15.6:0/3255754593 wait complete. 2024-03-22T12:57:37.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.732+0000 7f3975200640 1 Processor -- start 2024-03-22T12:57:37.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.732+0000 7f3975200640 1 -- start start 2024-03-22T12:57:37.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.733+0000 7f3975200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f39701f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:37.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.733+0000 7f3975200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39700ce8a0 0x7f3970155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:37.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.733+0000 7f3975200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f397015ae30 0x7f3970156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:37.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.733+0000 7f3975200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397015a910 con 0x7f397015ae30 2024-03-22T12:57:37.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.733+0000 7f3975200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397015a630 con 0x7f39700ce8a0 2024-03-22T12:57:37.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.733+0000 7f3975200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39701549d0 con 0x7f39700cc380 2024-03-22T12:57:37.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.734+0000 7f396ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39700ce8a0 0x7f3970155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:37.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.734+0000 7f396f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f39701f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:37.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.734+0000 7f396ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39700ce8a0 0x7f3970155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:45270/0 (socket says 172.21.15.6:45270) 2024-03-22T12:57:37.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.734+0000 7f396ea00640 1 -- 172.21.15.6:0/1883169219 learned_addr learned my addr 172.21.15.6:0/1883169219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.738+0000 7f396fe00640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f397015ae30 0x7f3970156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.738+0000 7f396f400640 1 -- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39700ce8a0 msgr2=0x7f3970155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.738+0000 7f396f400640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39700ce8a0 0x7f3970155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.738+0000 7f396f400640 1 -- 172.21.15.6:0/1883169219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f397015ae30 msgr2=0x7f3970156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.738+0000 7f396f400640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f397015ae30 0x7f3970156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.738+0000 7f396f400640 1 -- 172.21.15.6:0/1883169219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397014a720 con 0x7f39700cc380 2024-03-22T12:57:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.741+0000 7f396f400640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f39701f3e90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3960033000 tx=0x7f39600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.744+0000 7f3967400640 1 -- 172.21.15.6:0/1883169219 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960040070 con 0x7f39700cc380 2024-03-22T12:57:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.745+0000 7f3967400640 1 -- 172.21.15.6:0/1883169219 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39600399c0 con 0x7f39700cc380 2024-03-22T12:57:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.745+0000 7f3967400640 1 -- 172.21.15.6:0/1883169219 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960035040 con 0x7f39700cc380 2024-03-22T12:57:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.745+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3970146ae0 con 0x7f39700cc380 2024-03-22T12:57:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.745+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3970146da0 con 0x7f39700cc380 2024-03-22T12:57:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.746+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3970075650 con 0x7f39700cc380 2024-03-22T12:57:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.781+0000 7f3967400640 1 -- 172.21.15.6:0/1883169219 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f396004b060 con 0x7f39700cc380 2024-03-22T12:57:37.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.782+0000 7f3967400640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f395006bbd0 0x7f395006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:37.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.783+0000 7f396ea00640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f395006bbd0 0x7f395006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:37.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:37.784+0000 7f396ea00640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f395006bbd0 0x7f395006e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3970145a50 tx=0x7f395c007ee0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.022+0000 7f3967400640 1 -- 172.21.15.6:0/1883169219 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f3960078c50 con 0x7f39700cc380 2024-03-22T12:57:38.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.043+0000 7f3967400640 1 -- 172.21.15.6:0/1883169219 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f396007b2b0 con 0x7f39700cc380 2024-03-22T12:57:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.304+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f397014b4d0 con 0x7f39700cc380 2024-03-22T12:57:38.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.319+0000 7f3967400640 1 -- 172.21.15.6:0/1883169219 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6628 (secure 0 0 0) 0x7f396007abb0 con 0x7f39700cc380 2024-03-22T12:57:38.319 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:57:38.319 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:57:18.963149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4259,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4259":{"gid":4259,"name":"d","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6837/2825293724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":2825293724},{"type":"v1","addr":"172.21.15.92:6837","nonce":2825293724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4259,4261,4268]},"id":1}]} 2024-03-22T12:57:38.320 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-03-22T12:57:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f395006bbd0 msgr2=0x7f395006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f395006bbd0 0x7f395006e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3970145a50 tx=0x7f395c007ee0 comp rx=0 tx=0).stop 2024-03-22T12:57:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 msgr2=0x7f39701f3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f39701f3e90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3960033000 tx=0x7f39600046c0 comp rx=0 tx=0).stop 2024-03-22T12:57:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 shutdown_connections 2024-03-22T12:57:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f397015ae30 0x7f3970156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f395006bbd0 0x7f395006e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39700ce8a0 0x7f3970155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 --2- 172.21.15.6:0/1883169219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39700cc380 0x7f39701f3e90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 >> 172.21.15.6:0/1883169219 conn(0x7f39700b1480 msgr2=0x7f3970158f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 shutdown_connections 2024-03-22T12:57:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:38.322+0000 7f3975200640 1 -- 172.21.15.6:0/1883169219 wait complete. 2024-03-22T12:57:38.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:57:18.963149+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4259, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4259': {'gid': 4259, 'name': 'd', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6837/2825293724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6836', 'nonce': 2825293724}, {'type': 'v1', 'addr': '172.21.15.92:6837', 'nonce': 2825293724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4259, 4261, 4268]} 2024-03-22T12:57:38.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-22T12:57:38.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-22T12:57:39.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.145+0000 7f0cf9800640 1 Processor -- start 2024-03-22T12:57:39.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.145+0000 7f0cf9800640 1 -- start start 2024-03-22T12:57:39.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.147+0000 7f0cf9800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0cf41548b0 0x7f0cf415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.147+0000 7f0cf9800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0cf4067550 0x7f0cf40cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.147+0000 7f0cf9800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf40c5470 con 0x7f0cf40cc700 2024-03-22T12:57:39.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.147+0000 7f0cf9800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf40c4ef0 con 0x7f0cf41548b0 2024-03-22T12:57:39.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.147+0000 7f0cf9800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf415b640 con 0x7f0cf4067550 2024-03-22T12:57:39.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.148+0000 7f0cf2000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0cf4067550 0x7f0cf40cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.149+0000 7f0cf2000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0cf4067550 0x7f0cf40cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:45274/0 (socket says 172.21.15.6:45274) 2024-03-22T12:57:39.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.149+0000 7f0cf2000640 1 -- 172.21.15.6:0/3105700361 learned_addr learned my addr 172.21.15.6:0/3105700361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:39.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.149+0000 7f0cf2a00640 1 --2- 172.21.15.6:0/3105700361 >> v2:172.21.15.6:3300/0 conn(0x7f0cf41548b0 0x7f0cf415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.151+0000 7f0cf2000640 1 -- 172.21.15.6:0/3105700361 >> v2:172.21.15.6:3300/0 conn(0x7f0cf41548b0 msgr2=0x7f0cf415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:39.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.151+0000 7f0cf2000640 1 --2- 172.21.15.6:0/3105700361 >> v2:172.21.15.6:3300/0 conn(0x7f0cf41548b0 0x7f0cf415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.151+0000 7f0cf2000640 1 -- 172.21.15.6:0/3105700361 >> v1:172.21.15.6:6789/0 conn(0x7f0cf40cc700 legacy=0x7f0cf415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:39.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.151+0000 7f0cf2000640 1 -- 172.21.15.6:0/3105700361 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf40c5190 con 0x7f0cf4067550 2024-03-22T12:57:39.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.153+0000 7f0cf2000640 1 --2- 172.21.15.6:0/3105700361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf4067550 0x7f0cf40cb340 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0cf406e0f0 tx=0x7f0ce002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db6fb8d099e1d0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:39.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.157+0000 7f0cf1600640 1 -- 172.21.15.6:0/3105700361 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce002e070 con 0x7f0cf4067550 2024-03-22T12:57:39.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.157+0000 7f0cf1600640 1 -- 172.21.15.6:0/3105700361 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0037070 con 0x7f0cf4067550 2024-03-22T12:57:39.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.157+0000 7f0cf1600640 1 -- 172.21.15.6:0/3105700361 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0031a00 con 0x7f0cf4067550 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 -- 172.21.15.6:0/3105700361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf4067550 msgr2=0x7f0cf40cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 --2- 172.21.15.6:0/3105700361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf4067550 0x7f0cf40cb340 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0cf406e0f0 tx=0x7f0ce002b880 comp rx=0 tx=0).stop 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 -- 172.21.15.6:0/3105700361 shutdown_connections 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 --2- 172.21.15.6:0/3105700361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf4067550 0x7f0cf40cb340 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 --2- 172.21.15.6:0/3105700361 >> v2:172.21.15.6:3300/0 conn(0x7f0cf41548b0 0x7f0cf415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 -- 172.21.15.6:0/3105700361 >> 172.21.15.6:0/3105700361 conn(0x7f0cf406cbc0 msgr2=0x7f0cf4066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 -- 172.21.15.6:0/3105700361 shutdown_connections 2024-03-22T12:57:39.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.158+0000 7f0cf9800640 1 -- 172.21.15.6:0/3105700361 wait complete. 2024-03-22T12:57:39.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.160+0000 7f0cf9800640 1 Processor -- start 2024-03-22T12:57:39.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.160+0000 7f0cf9800640 1 -- start start 2024-03-22T12:57:39.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.161+0000 7f0cf9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cf4067550 0x7f0cf41f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.161+0000 7f0cf9800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cf40cc700 0x7f0cf41563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.161+0000 7f0cf9800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf41548b0 0x7f0cf40c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.161+0000 7f0cf9800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40c5880 con 0x7f0cf4067550 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.161+0000 7f0cf9800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40cab40 con 0x7f0cf41548b0 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.161+0000 7f0cf9800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4154c90 con 0x7f0cf40cc700 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.162+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cf4067550 0x7f0cf41f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.162+0000 7f0cf3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf41548b0 0x7f0cf40c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.162+0000 7f0cf2000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cf40cc700 0x7f0cf41563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.163+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cf4067550 0x7f0cf41f0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42418/0 (socket says 172.21.15.6:42418) 2024-03-22T12:57:39.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.163+0000 7f0cf2a00640 1 -- 172.21.15.6:0/1760919324 learned_addr learned my addr 172.21.15.6:0/1760919324 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:39.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.166+0000 7f0cf2000640 1 -- 172.21.15.6:0/1760919324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf41548b0 msgr2=0x7f0cf40c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:39.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.166+0000 7f0cf2000640 1 --2- 172.21.15.6:0/1760919324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cf41548b0 0x7f0cf40c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.166+0000 7f0cf2000640 1 -- 172.21.15.6:0/1760919324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cf4067550 msgr2=0x7f0cf41f0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:39.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.166+0000 7f0cf2000640 1 --2- 172.21.15.6:0/1760919324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cf4067550 0x7f0cf41f0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.166+0000 7f0cf2000640 1 -- 172.21.15.6:0/1760919324 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf4150760 con 0x7f0cf40cc700 2024-03-22T12:57:39.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.166+0000 7f0cf2a00640 1 --2- 172.21.15.6:0/1760919324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cf4067550 0x7f0cf41f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:57:39.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.169+0000 7f0cf2000640 1 --2- 172.21.15.6:0/1760919324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cf40cc700 0x7f0cf41563c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0033000 tx=0x7f0ce00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:39.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.173+0000 7f0ceaa00640 1 -- 172.21.15.6:0/1760919324 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0032880 con 0x7f0cf40cc700 2024-03-22T12:57:39.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.173+0000 7f0ceaa00640 1 -- 172.21.15.6:0/1760919324 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0032a20 con 0x7f0cf40cc700 2024-03-22T12:57:39.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.173+0000 7f0ceaa00640 1 -- 172.21.15.6:0/1760919324 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0032e10 con 0x7f0cf40cc700 2024-03-22T12:57:39.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.173+0000 7f0cf9800640 1 -- 172.21.15.6:0/1760919324 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf40cbed0 con 0x7f0cf40cc700 2024-03-22T12:57:39.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.173+0000 7f0cf9800640 1 -- 172.21.15.6:0/1760919324 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf4150ce0 con 0x7f0cf40cc700 2024-03-22T12:57:39.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.176+0000 7f0cfc59b740 1 Processor -- start 2024-03-22T12:57:39.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.177+0000 7f0cfc59b740 1 -- start start 2024-03-22T12:57:39.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.177+0000 7f0cfc59b740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e8028d4650 0x55e8028d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.177+0000 7f0cfc59b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028d4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.178+0000 7f0cfc59b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e8028d30b0 0x55e802913fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.178+0000 7f0cfc59b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e802838c90 con 0x55e80291e4d0 2024-03-22T12:57:39.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.178+0000 7f0cfc59b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8027e1980 con 0x55e8028d30b0 2024-03-22T12:57:39.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.178+0000 7f0cfc59b740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e802834d30 con 0x55e8028d4650 2024-03-22T12:57:39.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.178+0000 7f0cf2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e8028d30b0 0x55e802913fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.179+0000 7f0cf2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e8028d30b0 0x55e802913fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:45298/0 (socket says 172.21.15.6:45298) 2024-03-22T12:57:39.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.179+0000 7f0cf2000640 1 -- 172.21.15.6:0/905570463 learned_addr learned my addr 172.21.15.6:0/905570463 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:39.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.180+0000 7f0cf2a00640 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028d4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.182+0000 7f0cf2000640 1 -- 172.21.15.6:0/905570463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e8028d4650 msgr2=0x55e8028d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:39.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.182+0000 7f0cf2000640 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e8028d4650 0x55e8028d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.182+0000 7f0cf2000640 1 -- 172.21.15.6:0/905570463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 msgr2=0x55e8028d4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:39.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.182+0000 7f0cf2000640 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028d4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.182+0000 7f0cf2000640 1 -- 172.21.15.6:0/905570463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8027f9510 con 0x55e8028d30b0 2024-03-22T12:57:39.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.187+0000 7f0cf2000640 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e8028d30b0 0x55e802913fd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ce002be90 tx=0x7f0ce003e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cca050992f98f99c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:39.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.194+0000 7f0cdf400640 1 -- 172.21.15.6:0/905570463 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce004e700 con 0x55e8028d30b0 2024-03-22T12:57:39.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.194+0000 7f0cdf400640 1 -- 172.21.15.6:0/905570463 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0037040 con 0x55e8028d30b0 2024-03-22T12:57:39.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.194+0000 7f0cdf400640 1 -- 172.21.15.6:0/905570463 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce004eaa0 con 0x55e8028d30b0 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 -- 172.21.15.6:0/905570463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e8028d30b0 msgr2=0x55e802913fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e8028d30b0 0x55e802913fd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ce002be90 tx=0x7f0ce003e000 comp rx=0 tx=0).stop 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 -- 172.21.15.6:0/905570463 shutdown_connections 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e8028d30b0 0x55e802913fd0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ce002be90 tx=0x7f0ce003e000 comp rx=0 tx=0).stop 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028d4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 --2- 172.21.15.6:0/905570463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e8028d4650 0x55e8028d4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 -- 172.21.15.6:0/905570463 >> 172.21.15.6:0/905570463 conn(0x55e8028d0dc0 msgr2=0x55e8028d3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:57:39.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.195+0000 7f0cfc59b740 1 -- 172.21.15.6:0/905570463 shutdown_connections 2024-03-22T12:57:39.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.197+0000 7f0ceaa00640 1 -- 172.21.15.6:0/1760919324 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0ce004e500 con 0x7f0cf40cc700 2024-03-22T12:57:39.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.197+0000 7f0cfc59b740 1 -- 172.21.15.6:0/905570463 wait complete. 2024-03-22T12:57:39.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.197+0000 7f0cfc59b740 1 -- start start 2024-03-22T12:57:39.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.198+0000 7f0cfc59b740 1 Processor -- start 2024-03-22T12:57:39.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.198+0000 7f0ceaa00640 1 --2- 172.21.15.6:0/1760919324 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0cd406bb90 0x7f0cd406e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.198+0000 7f0cf3400640 1 --2- 172.21.15.6:0/1760919324 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0cd406bb90 0x7f0cd406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cfc59b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028f9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cfc59b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e802919020 0x55e8028ee680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cfc59b740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e802915420 0x55e802915880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cfc59b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e802838c90 con 0x55e80291e4d0 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cfc59b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8027f4dd0 con 0x55e802919020 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cfc59b740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e8027e1980 con 0x55e802915420 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cfc59b740 10 client.0 _ll_register_callbacks cb 0x55e80254cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0cf3400640 1 --2- 172.21.15.6:0/1760919324 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0cd406bb90 0x7f0cd406e050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0014e0 tx=0x7f0cec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.199+0000 7f0ceaa00640 1 -- 172.21.15.6:0/1760919324 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f0ce007a5c0 con 0x7f0cf40cc700 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.200+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028f9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.200+0000 7f0cf2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e802919020 0x55e8028ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.200+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028f9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42426/0 (socket says 172.21.15.6:42426) 2024-03-22T12:57:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.200+0000 7f0cf2a00640 1 -- 172.21.15.6:0/2477684754 learned_addr learned my addr 172.21.15.6:0/2477684754 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:57:39.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.204+0000 7f0cf2a00640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e802915420 msgr2=0x55e802915880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:39.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.204+0000 7f0cf2a00640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x55e802915420 0x55e802915880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.204+0000 7f0cf2a00640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e802919020 msgr2=0x55e8028ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:57:39.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.204+0000 7f0cf2a00640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e802919020 0x55e8028ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:57:39.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.204+0000 7f0cf2a00640 1 -- 172.21.15.6:0/2477684754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e80254d2d0 con 0x55e80291e4d0 2024-03-22T12:57:39.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.206+0000 7f0cf2a00640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e80291e4d0 0x55e8028f9e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4014ec0 tx=0x7f0ce4012700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a927df884d3bf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:57:39.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.206+0000 7f0cd3e00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-22T12:57:39.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.213+0000 7f0cd3e00640 1 -- 172.21.15.6:0/2477684754 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4004030 con 0x55e80291e4d0 2024-03-22T12:57:39.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.213+0000 7f0cd3e00640 1 -- 172.21.15.6:0/2477684754 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(30..30 src has 1..30) v4 ==== 6793+0+0 (secure 0 0 0) 0x7f0ce40041d0 con 0x55e80291e4d0 2024-03-22T12:57:39.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.213+0000 7f0cfc59b740 1 -- 172.21.15.6:0/2477684754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ce0037040 con 0x55e80291e4d0 2024-03-22T12:57:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.291+0000 7f0cd3e00640 1 -- 172.21.15.6:0/2477684754 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ce40043d0 con 0x55e80291e4d0 2024-03-22T12:57:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.292+0000 7f0cfc59b740 10 client.4587 fetch_fsmap learned FSMap version 18 2024-03-22T12:57:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.292+0000 7f0cfc59b740 1 -- 172.21.15.6:0/2477684754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x55e8026176b0 con 0x55e80291e4d0 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cd3e00640 1 -- 172.21.15.6:0/2477684754 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0037040 con 0x55e80291e4d0 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cd3e00640 1 -- 172.21.15.6:0/2477684754 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 18) v1 ==== 4552+0+0 (secure 0 0 0) 0x7f0ce402a080 con 0x55e80291e4d0 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cfc59b740 10 client.4587 fetch_fsmap finished waiting for FSMap version 18 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cfc59b740 10 client.4587 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cfc59b740 20 client.4587 populate_metadata read hostname 'smithi006' 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cfc59b740 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cfc59b740 4 client.4587 mds_command: new command op to 4252 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T12:57:39.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:39.307+0000 7f0cfc59b740 1 -- 172.21.15.6:0/2477684754 --> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e8027da520 con 0x55e802928da0 2024-03-22T12:57:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:49.314+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:57:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:49.314+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T12:57:59.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:59.520+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:57:59.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:57:59.520+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T12:57:59.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:57:59.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:58:09.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:09.928+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:58:09.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:09.929+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T12:58:20.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:20.739+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:58:20.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:20.739+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T12:58:29.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:58:29.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:58:32.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:32.341+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:58:32.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:32.341+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-22T12:58:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:45.547+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:58:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:58:45.547+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-22T12:58:59.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:58:59.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:59:01.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:01.965+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:59:01.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:01.965+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-22T12:59:24.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:24.771+0000 7f0cf2000640 1 -- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 msgr2=0x55e802934330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.92:6834/3494871082, fault. 2024-03-22T12:59:24.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:24.771+0000 7f0cf2000640 1 --2- 172.21.15.6:0/2477684754 >> [v2:172.21.15.92:6834/3494871082,v1:172.21.15.92:6835/3494871082] conn(0x55e802928da0 0x55e802934330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-22T12:59:30.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:59:30.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T12:59:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-22T12:59:40.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.258+0000 7f33c6800640 1 Processor -- start 2024-03-22T12:59:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.259+0000 7f33c6800640 1 -- start start 2024-03-22T12:59:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.260+0000 7f33c6800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c006f8c0 0x7f33c00716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.260+0000 7f33c6800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f33c0071c10 0x7f33c00c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.260+0000 7f33c6800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c01590b0 con 0x7f33c0152400 2024-03-22T12:59:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.260+0000 7f33c6800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00724d0 con 0x7f33c0071c10 2024-03-22T12:59:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.260+0000 7f33c6800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00721f0 con 0x7f33c006f8c0 2024-03-22T12:59:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.261+0000 7f33bf400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f33c0152400 0x7f33c015b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50522/0 (socket says 172.21.15.6:50522) 2024-03-22T12:59:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.261+0000 7f33bf400640 1 -- 172.21.15.6:0/980534439 learned_addr learned my addr 172.21.15.6:0/980534439 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:59:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.261+0000 7f33bfe00640 1 --2- 172.21.15.6:0/980534439 >> v2:172.21.15.92:3300/0 conn(0x7f33c0071c10 0x7f33c00c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.261+0000 7f33bea00640 1 --2- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c006f8c0 0x7f33c00716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.264+0000 7f33be000640 1 -- 172.21.15.6:0/980534439 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700569666 0 0) 0x7f33c01590b0 con 0x7f33c0152400 2024-03-22T12:59:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.264+0000 7f33be000640 1 -- 172.21.15.6:0/980534439 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a4003520 con 0x7f33c0152400 2024-03-22T12:59:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.265+0000 7f33bfe00640 1 -- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c006f8c0 msgr2=0x7f33c00716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.265+0000 7f33bfe00640 1 --2- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c006f8c0 0x7f33c00716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.265+0000 7f33bfe00640 1 -- 172.21.15.6:0/980534439 >> v1:172.21.15.6:6789/0 conn(0x7f33c0152400 legacy=0x7f33c015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.265+0000 7f33bfe00640 1 -- 172.21.15.6:0/980534439 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c01582f0 con 0x7f33c0071c10 2024-03-22T12:59:40.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.268+0000 7f33bfe00640 1 --2- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0071c10 0x7f33c00c5040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009d90 tx=0x7f33b802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8e8881b50dfe340 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.272+0000 7f33be000640 1 -- 172.21.15.6:0/980534439 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8039070 con 0x7f33c0071c10 2024-03-22T12:59:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.272+0000 7f33be000640 1 -- 172.21.15.6:0/980534439 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b802cce0 con 0x7f33c0071c10 2024-03-22T12:59:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.272+0000 7f33be000640 1 -- 172.21.15.6:0/980534439 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8034660 con 0x7f33c0071c10 2024-03-22T12:59:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 -- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0071c10 msgr2=0x7f33c00c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 --2- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0071c10 0x7f33c00c5040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009d90 tx=0x7f33b802bf30 comp rx=0 tx=0).stop 2024-03-22T12:59:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 -- 172.21.15.6:0/980534439 shutdown_connections 2024-03-22T12:59:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 --2- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0071c10 0x7f33c00c5040 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 --2- 172.21.15.6:0/980534439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c006f8c0 0x7f33c00716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 -- 172.21.15.6:0/980534439 >> 172.21.15.6:0/980534439 conn(0x7f33c00cb930 msgr2=0x7f33c00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 -- 172.21.15.6:0/980534439 shutdown_connections 2024-03-22T12:59:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.273+0000 7f33c6800640 1 -- 172.21.15.6:0/980534439 wait complete. 2024-03-22T12:59:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.276+0000 7f33c6800640 1 Processor -- start 2024-03-22T12:59:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.276+0000 7f33c6800640 1 -- start start 2024-03-22T12:59:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.277+0000 7f33c6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c006f8c0 0x7f33c014c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.277+0000 7f33c6800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c0071c10 0x7f33c0156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.277+0000 7f33c6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0152400 0x7f33c0157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.278+0000 7f33c6800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c0158b90 con 0x7f33c006f8c0 2024-03-22T12:59:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.278+0000 7f33c6800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01588b0 con 0x7f33c0152400 2024-03-22T12:59:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.278+0000 7f33c6800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01527e0 con 0x7f33c0071c10 2024-03-22T12:59:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.278+0000 7f33bea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c0071c10 0x7f33c0156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.278+0000 7f33bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c006f8c0 0x7f33c014c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.278+0000 7f33bfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0152400 0x7f33c0157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.279+0000 7f33bfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0152400 0x7f33c0157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50872/0 (socket says 172.21.15.6:50872) 2024-03-22T12:59:40.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.279+0000 7f33bfe00640 1 -- 172.21.15.6:0/2551181980 learned_addr learned my addr 172.21.15.6:0/2551181980 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:40.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.284+0000 7f33bfe00640 1 -- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c0071c10 msgr2=0x7f33c0156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:40.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.284+0000 7f33bfe00640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c0071c10 0x7f33c0156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.284+0000 7f33bfe00640 1 -- 172.21.15.6:0/2551181980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c006f8c0 msgr2=0x7f33c014c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:40.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.284+0000 7f33bfe00640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c006f8c0 0x7f33c014c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.284+0000 7f33bfe00640 1 -- 172.21.15.6:0/2551181980 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0155f70 con 0x7f33c0152400 2024-03-22T12:59:40.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.287+0000 7f33bfe00640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0152400 0x7f33c0157200 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009eb0 tx=0x7f33b8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:40.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.292+0000 7f33b7400640 1 -- 172.21.15.6:0/2551181980 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8044020 con 0x7f33c0152400 2024-03-22T12:59:40.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.292+0000 7f33b7400640 1 -- 172.21.15.6:0/2551181980 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b8035e50 con 0x7f33c0152400 2024-03-22T12:59:40.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.292+0000 7f33b7400640 1 -- 172.21.15.6:0/2551181980 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8039040 con 0x7f33c0152400 2024-03-22T12:59:40.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.292+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c014d7c0 con 0x7f33c0152400 2024-03-22T12:59:40.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.292+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c01436c0 con 0x7f33c0152400 2024-03-22T12:59:40.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.293+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c00c9340 con 0x7f33c0152400 2024-03-22T12:59:40.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.325+0000 7f33b7400640 1 -- 172.21.15.6:0/2551181980 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f33b8047020 con 0x7f33c0152400 2024-03-22T12:59:40.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.326+0000 7f33b7400640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f33a006bc20 0x7f33a006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:40.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.327+0000 7f33bf400640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f33a006bc20 0x7f33a006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:40.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.328+0000 7f33bf400640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f33a006bc20 0x7f33a006e0e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f33b0005880 tx=0x7f33b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:40.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.329+0000 7f33b7400640 1 -- 172.21.15.6:0/2551181980 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f33b8079c40 con 0x7f33c0152400 2024-03-22T12:59:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.601+0000 7f33b7400640 1 -- 172.21.15.6:0/2551181980 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f33c00c9340 con 0x7f33c0152400 2024-03-22T12:59:40.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.848+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33c00756c0 con 0x7f33c0152400 2024-03-22T12:59:40.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.862+0000 7f33b7400640 1 -- 172.21.15.6:0/2551181980 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f33b807ce20 con 0x7f33c0152400 2024-03-22T12:59:40.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:40.865 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:40.866 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:40.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f33a006bc20 msgr2=0x7f33a006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:40.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f33a006bc20 0x7f33a006e0e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f33b0005880 tx=0x7f33b0009040 comp rx=0 tx=0).stop 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0152400 msgr2=0x7f33c0157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0152400 0x7f33c0157200 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009eb0 tx=0x7f33b8004a40 comp rx=0 tx=0).stop 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 shutdown_connections 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f33a006bc20 0x7f33a006e0e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33c0152400 0x7f33c0157200 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33c0071c10 0x7f33c0156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 --2- 172.21.15.6:0/2551181980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33c006f8c0 0x7f33c014c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 >> 172.21.15.6:0/2551181980 conn(0x7f33c00cb930 msgr2=0x7f33c0150920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 shutdown_connections 2024-03-22T12:59:40.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:40.865+0000 7f33c6800640 1 -- 172.21.15.6:0/2551181980 wait complete. 2024-03-22T12:59:41.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:41.486 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:42.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.687+0000 7f9ada600640 1 Processor -- start 2024-03-22T12:59:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.687+0000 7f9ada600640 1 -- start start 2024-03-22T12:59:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.688+0000 7f9ada600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.688+0000 7f9ada600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad415b260 con 0x7f9ad415ae30 2024-03-22T12:59:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.688+0000 7f9ada600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad40cee00 con 0x7f9ad40cc380 2024-03-22T12:59:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.688+0000 7f9ada600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40cf0c0 con 0x7f9ad40ce8a0 2024-03-22T12:59:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.689+0000 7f9ad3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.689+0000 7f9ad9c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9ad415ae30 0x7f9ad4164b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50534/0 (socket says 172.21.15.6:50534) 2024-03-22T12:59:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.689+0000 7f9ad9c00640 1 -- 172.21.15.6:0/382374820 learned_addr learned my addr 172.21.15.6:0/382374820 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:59:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.692+0000 7f9ad3e00640 1 -- 172.21.15.6:0/382374820 >> v1:172.21.15.92:6789/0 conn(0x7f9ad40cc380 legacy=0x7f9ad40cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.692+0000 7f9ad3e00640 1 -- 172.21.15.6:0/382374820 >> v1:172.21.15.6:6789/0 conn(0x7f9ad415ae30 legacy=0x7f9ad4164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.692+0000 7f9ad3e00640 1 -- 172.21.15.6:0/382374820 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40cf3a0 con 0x7f9ad40ce8a0 2024-03-22T12:59:42.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.694+0000 7f9ad3e00640 1 --2- 172.21.15.6:0/382374820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad406c230 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00097d0 tx=0x7f9ac002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb85de37d56c4571 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.698+0000 7f9ad3400640 1 -- 172.21.15.6:0/382374820 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0035020 con 0x7f9ad40ce8a0 2024-03-22T12:59:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.699+0000 7f9ad3400640 1 -- 172.21.15.6:0/382374820 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac0031890 con 0x7f9ad40ce8a0 2024-03-22T12:59:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.699+0000 7f9ad3400640 1 -- 172.21.15.6:0/382374820 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0031c60 con 0x7f9ad40ce8a0 2024-03-22T12:59:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.699+0000 7f9ada600640 1 -- 172.21.15.6:0/382374820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 msgr2=0x7f9ad406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.699+0000 7f9ada600640 1 --2- 172.21.15.6:0/382374820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad406c230 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00097d0 tx=0x7f9ac002b930 comp rx=0 tx=0).stop 2024-03-22T12:59:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.699+0000 7f9ada600640 1 -- 172.21.15.6:0/382374820 shutdown_connections 2024-03-22T12:59:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.699+0000 7f9ada600640 1 --2- 172.21.15.6:0/382374820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad406c230 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.699+0000 7f9ada600640 1 -- 172.21.15.6:0/382374820 >> 172.21.15.6:0/382374820 conn(0x7f9ad40b1480 msgr2=0x7f9ad40b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.700+0000 7f9ada600640 1 -- 172.21.15.6:0/382374820 shutdown_connections 2024-03-22T12:59:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.700+0000 7f9ada600640 1 -- 172.21.15.6:0/382374820 wait complete. 2024-03-22T12:59:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.701+0000 7f9ada600640 1 Processor -- start 2024-03-22T12:59:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.702+0000 7f9ada600640 1 -- start start 2024-03-22T12:59:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.702+0000 7f9ada600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ad40cc380 0x7f9ad40b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.702+0000 7f9ada600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad415d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.703+0000 7f9ada600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ad415ae30 0x7f9ad415d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.703+0000 7f9ada600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad415a910 con 0x7f9ad40cc380 2024-03-22T12:59:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.703+0000 7f9ada600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad415a630 con 0x7f9ad415ae30 2024-03-22T12:59:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.703+0000 7f9ada600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad41549d0 con 0x7f9ad40ce8a0 2024-03-22T12:59:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.703+0000 7f9ad3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad415d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.703+0000 7f9ad9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ad40cc380 0x7f9ad40b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.703+0000 7f9ad9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ad415ae30 0x7f9ad415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.704+0000 7f9ad3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad415d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51586/0 (socket says 172.21.15.6:51586) 2024-03-22T12:59:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.704+0000 7f9ad3e00640 1 -- 172.21.15.6:0/2533393117 learned_addr learned my addr 172.21.15.6:0/2533393117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:42.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.708+0000 7f9ad9200640 1 -- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 msgr2=0x7f9ad415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:42.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.708+0000 7f9ad9200640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad415d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:42.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.708+0000 7f9ad9200640 1 -- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ad415ae30 msgr2=0x7f9ad415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:42.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.708+0000 7f9ad9200640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ad415ae30 0x7f9ad415d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:42.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.708+0000 7f9ad9200640 1 -- 172.21.15.6:0/2533393117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad415b5c0 con 0x7f9ad40cc380 2024-03-22T12:59:42.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.711+0000 7f9ad9200640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ad40cc380 0x7f9ad40b2c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c390 tx=0x7f9ac800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.719+0000 7f9ad0c00640 1 -- 172.21.15.6:0/2533393117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8017070 con 0x7f9ad40cc380 2024-03-22T12:59:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.719+0000 7f9ad0c00640 1 -- 172.21.15.6:0/2533393117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac80052d0 con 0x7f9ad40cc380 2024-03-22T12:59:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.719+0000 7f9ad0c00640 1 -- 172.21.15.6:0/2533393117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8003830 con 0x7f9ad40cc380 2024-03-22T12:59:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.719+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad415b880 con 0x7f9ad40cc380 2024-03-22T12:59:42.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.720+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad40c1cf0 con 0x7f9ad40cc380 2024-03-22T12:59:42.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.721+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad4075650 con 0x7f9ad40cc380 2024-03-22T12:59:42.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.754+0000 7f9ad0c00640 1 -- 172.21.15.6:0/2533393117 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ac801a070 con 0x7f9ad40cc380 2024-03-22T12:59:42.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.756+0000 7f9ad0c00640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ab406bc20 0x7f9ab406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:42.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.756+0000 7f9ad3e00640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ab406bc20 0x7f9ab406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:42.757+0000 7f9ad3e00640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ab406bc20 0x7f9ab406e0e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9ac002b970 tx=0x7f9ac0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:43.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.081+0000 7f9ad0c00640 1 -- 172.21.15.6:0/2533393117 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9ac80566a0 con 0x7f9ad40cc380 2024-03-22T12:59:43.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.081+0000 7f9ad0c00640 1 -- 172.21.15.6:0/2533393117 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9ac8021020 con 0x7f9ad40cc380 2024-03-22T12:59:43.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.351+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ad40cae50 con 0x7f9ad40cc380 2024-03-22T12:59:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.379+0000 7f9ad0c00640 1 -- 172.21.15.6:0/2533393117 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f9ac8058600 con 0x7f9ad40cc380 2024-03-22T12:59:43.379 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:43.380 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:43.380 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ab406bc20 msgr2=0x7f9ab406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ab406bc20 0x7f9ab406e0e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9ac002b970 tx=0x7f9ac0008030 comp rx=0 tx=0).stop 2024-03-22T12:59:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ad40cc380 msgr2=0x7f9ad40b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ad40cc380 0x7f9ad40b2c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c390 tx=0x7f9ac800c850 comp rx=0 tx=0).stop 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 shutdown_connections 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ab406bc20 0x7f9ab406e0e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ad415ae30 0x7f9ad415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9ad40ce8a0 0x7f9ad415d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 --2- 172.21.15.6:0/2533393117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ad40cc380 0x7f9ad40b2c00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 >> 172.21.15.6:0/2533393117 conn(0x7f9ad40b1480 msgr2=0x7f9ad40c2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 shutdown_connections 2024-03-22T12:59:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:43.383+0000 7f9ada600640 1 -- 172.21.15.6:0/2533393117 wait complete. 2024-03-22T12:59:44.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:44.004 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:45.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:45.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.204+0000 7fcfa6800640 1 Processor -- start 2024-03-22T12:59:45.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.204+0000 7fcfa6800640 1 -- start start 2024-03-22T12:59:45.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.205+0000 7fcfa6800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcfa00cc380 0x7fcfa006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:45.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.205+0000 7fcfa6800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00ce8a0 0x7fcfa015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:45.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.206+0000 7fcfa6800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa00cee00 con 0x7fcfa00cc760 2024-03-22T12:59:45.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.206+0000 7fcfa6800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa00cf0c0 con 0x7fcfa00cc380 2024-03-22T12:59:45.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.206+0000 7fcfa6800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa00cf3a0 con 0x7fcfa00ce8a0 2024-03-22T12:59:45.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.206+0000 7fcfa5400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcfa00cc380 0x7fcfa006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:45.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.206+0000 7fcfa4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00ce8a0 0x7fcfa015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:45.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.207+0000 7fcfa5400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcfa00cc380 0x7fcfa006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:57444/0 (socket says 172.21.15.6:57444) 2024-03-22T12:59:45.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.207+0000 7fcfa5400640 1 -- 172.21.15.6:0/756063146 learned_addr learned my addr 172.21.15.6:0/756063146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.210+0000 7fcfa4a00640 1 -- 172.21.15.6:0/756063146 >> v2:172.21.15.92:3300/0 conn(0x7fcfa00cc380 msgr2=0x7fcfa006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.210+0000 7fcfa4a00640 1 --2- 172.21.15.6:0/756063146 >> v2:172.21.15.92:3300/0 conn(0x7fcfa00cc380 0x7fcfa006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.210+0000 7fcfa4a00640 1 -- 172.21.15.6:0/756063146 >> v1:172.21.15.92:6789/0 conn(0x7fcfa00cc760 legacy=0x7fcfa00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:45.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.210+0000 7fcfa4a00640 1 -- 172.21.15.6:0/756063146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa00cf680 con 0x7fcfa00ce8a0 2024-03-22T12:59:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.213+0000 7fcfa4a00640 1 --2- 172.21.15.6:0/756063146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00ce8a0 0x7fcfa015ae30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0097a0 tx=0x7fcf8c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d62fdb6c67215f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.218+0000 7fcf9fe00640 1 -- 172.21.15.6:0/756063146 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c035020 con 0x7fcfa00ce8a0 2024-03-22T12:59:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.218+0000 7fcf9fe00640 1 -- 172.21.15.6:0/756063146 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c031830 con 0x7fcfa00ce8a0 2024-03-22T12:59:45.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.218+0000 7fcf9fe00640 1 -- 172.21.15.6:0/756063146 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c031c00 con 0x7fcfa00ce8a0 2024-03-22T12:59:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 -- 172.21.15.6:0/756063146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00ce8a0 msgr2=0x7fcfa015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 --2- 172.21.15.6:0/756063146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00ce8a0 0x7fcfa015ae30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0097a0 tx=0x7fcf8c02b8d0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 -- 172.21.15.6:0/756063146 shutdown_connections 2024-03-22T12:59:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 --2- 172.21.15.6:0/756063146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00ce8a0 0x7fcfa015ae30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 --2- 172.21.15.6:0/756063146 >> v2:172.21.15.92:3300/0 conn(0x7fcfa00cc380 0x7fcfa006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 -- 172.21.15.6:0/756063146 >> 172.21.15.6:0/756063146 conn(0x7fcfa00b1480 msgr2=0x7fcfa00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 -- 172.21.15.6:0/756063146 shutdown_connections 2024-03-22T12:59:45.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.219+0000 7fcfa6800640 1 -- 172.21.15.6:0/756063146 wait complete. 2024-03-22T12:59:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.221+0000 7fcfa6800640 1 Processor -- start 2024-03-22T12:59:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.222+0000 7fcfa6800640 1 -- start start 2024-03-22T12:59:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.223+0000 7fcfa6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcfa00cc380 0x7fcfa00c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.223+0000 7fcfa6800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00cc760 0x7fcfa01f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.223+0000 7fcfa6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfa00ce8a0 0x7fcfa01e3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.223+0000 7fcfa6800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa01549d0 con 0x7fcfa00cc380 2024-03-22T12:59:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.223+0000 7fcfa6800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa015a910 con 0x7fcfa00ce8a0 2024-03-22T12:59:45.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.223+0000 7fcfa6800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa00ccb90 con 0x7fcfa00cc760 2024-03-22T12:59:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.224+0000 7fcfa4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00cc760 0x7fcfa01f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.224+0000 7fcfa5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcfa00cc380 0x7fcfa00c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.224+0000 7fcfa5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfa00ce8a0 0x7fcfa01e3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.224+0000 7fcfa4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00cc760 0x7fcfa01f3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55378/0 (socket says 172.21.15.6:55378) 2024-03-22T12:59:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.224+0000 7fcfa4a00640 1 -- 172.21.15.6:0/1072643427 learned_addr learned my addr 172.21.15.6:0/1072643427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.227+0000 7fcfa4a00640 1 -- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfa00ce8a0 msgr2=0x7fcfa01e3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.228+0000 7fcfa4a00640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfa00ce8a0 0x7fcfa01e3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.228+0000 7fcfa4a00640 1 -- 172.21.15.6:0/1072643427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcfa00cc380 msgr2=0x7fcfa00c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:45.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.228+0000 7fcfa4a00640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcfa00cc380 0x7fcfa00c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.228+0000 7fcfa4a00640 1 -- 172.21.15.6:0/1072643427 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa00c7d20 con 0x7fcfa00cc760 2024-03-22T12:59:45.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.230+0000 7fcfa4a00640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00cc760 0x7fcfa01f3ca0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009890 tx=0x7fcf8c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.234+0000 7fcf9d600640 1 -- 172.21.15.6:0/1072643427 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c035040 con 0x7fcfa00cc760 2024-03-22T12:59:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.234+0000 7fcf9d600640 1 -- 172.21.15.6:0/1072643427 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c03b920 con 0x7fcfa00cc760 2024-03-22T12:59:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.234+0000 7fcf9d600640 1 -- 172.21.15.6:0/1072643427 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c03bd10 con 0x7fcfa00cc760 2024-03-22T12:59:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.234+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa0159040 con 0x7fcfa00cc760 2024-03-22T12:59:45.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.235+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa00c8270 con 0x7fcfa00cc760 2024-03-22T12:59:45.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.236+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa0074f60 con 0x7fcfa00cc760 2024-03-22T12:59:45.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.497+0000 7fcf9d600640 1 -- 172.21.15.6:0/1072643427 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcf8c038020 con 0x7fcfa00cc760 2024-03-22T12:59:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.498+0000 7fcf9d600640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf8006bc20 0x7fcf8006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.499+0000 7fcfa5400640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf8006bc20 0x7fcf8006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.499+0000 7fcf9d600640 1 -- 172.21.15.6:0/1072643427 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fcf8c079320 con 0x7fcfa00cc760 2024-03-22T12:59:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.499+0000 7fcfa5400640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf8006bc20 0x7fcf8006e0e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009b30 tx=0x7fcf94009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:45.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.511+0000 7fcf9d600640 1 -- 172.21.15.6:0/1072643427 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcfa0074f60 con 0x7fcfa00cc760 2024-03-22T12:59:45.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.763+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfa0075650 con 0x7fcfa00cc760 2024-03-22T12:59:45.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.776+0000 7fcf9d600640 1 -- 172.21.15.6:0/1072643427 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fcf8c031800 con 0x7fcfa00cc760 2024-03-22T12:59:45.776 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:45.776 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:45.776 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:45.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.778+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf8006bc20 msgr2=0x7fcf8006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.778+0000 7fcfa6800640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf8006bc20 0x7fcf8006e0e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009b30 tx=0x7fcf94009240 comp rx=0 tx=0).stop 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.778+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00cc760 msgr2=0x7fcfa01f3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.778+0000 7fcfa6800640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00cc760 0x7fcfa01f3ca0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009890 tx=0x7fcf8c0046c0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.778+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 shutdown_connections 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.779+0000 7fcfa6800640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf8006bc20 0x7fcf8006e0e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.779+0000 7fcfa6800640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfa00ce8a0 0x7fcfa01e3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.779+0000 7fcfa6800640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfa00cc760 0x7fcfa01f3ca0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.779+0000 7fcfa6800640 1 --2- 172.21.15.6:0/1072643427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcfa00cc380 0x7fcfa00c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.779+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 >> 172.21.15.6:0/1072643427 conn(0x7fcfa00b1480 msgr2=0x7fcfa00cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.779+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 shutdown_connections 2024-03-22T12:59:45.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:45.779+0000 7fcfa6800640 1 -- 172.21.15.6:0/1072643427 wait complete. 2024-03-22T12:59:46.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:46.399 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:47.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.593+0000 7f38d8a00640 1 Processor -- start 2024-03-22T12:59:47.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.594+0000 7f38d8a00640 1 -- start start 2024-03-22T12:59:47.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.595+0000 7f38d8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38d40cc3b0 0x7f38d40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:47.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.595+0000 7f38d8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40ccea0 0x7f38d415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.595+0000 7f38d8a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d40ce230 con 0x7f38d40cf770 2024-03-22T12:59:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.595+0000 7f38d8a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d40ce510 con 0x7f38d40cc3b0 2024-03-22T12:59:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.595+0000 7f38d8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40ce7f0 con 0x7f38d40ccea0 2024-03-22T12:59:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.596+0000 7f38d1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38d40cc3b0 0x7f38d40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.596+0000 7f38d2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40ccea0 0x7f38d415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:47.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.596+0000 7f38d1400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f38d40cf770 0x7f38d40cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:36434/0 (socket says 172.21.15.6:36434) 2024-03-22T12:59:47.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.596+0000 7f38d1400640 1 -- 172.21.15.6:0/349494778 learned_addr learned my addr 172.21.15.6:0/349494778 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:59:47.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.599+0000 7f38d2800640 1 -- 172.21.15.6:0/349494778 >> v2:172.21.15.6:3300/0 conn(0x7f38d40cc3b0 msgr2=0x7f38d40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.599+0000 7f38d2800640 1 --2- 172.21.15.6:0/349494778 >> v2:172.21.15.6:3300/0 conn(0x7f38d40cc3b0 0x7f38d40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.599+0000 7f38d2800640 1 -- 172.21.15.6:0/349494778 >> v1:172.21.15.92:6789/0 conn(0x7f38d40cf770 legacy=0x7f38d40cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:47.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.599+0000 7f38d2800640 1 -- 172.21.15.6:0/349494778 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d40cead0 con 0x7f38d40ccea0 2024-03-22T12:59:47.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.603+0000 7f38d2800640 1 --2- 172.21.15.6:0/349494778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40ccea0 0x7f38d415ab40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0097a0 tx=0x7f38cc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8654499ff646308 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:47.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.608+0000 7f38d0a00640 1 -- 172.21.15.6:0/349494778 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc031780 con 0x7f38d40ccea0 2024-03-22T12:59:47.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.608+0000 7f38d0a00640 1 -- 172.21.15.6:0/349494778 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc037070 con 0x7f38d40ccea0 2024-03-22T12:59:47.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.608+0000 7f38d0a00640 1 -- 172.21.15.6:0/349494778 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc031b40 con 0x7f38d40ccea0 2024-03-22T12:59:47.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.608+0000 7f38d8a00640 1 -- 172.21.15.6:0/349494778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40ccea0 msgr2=0x7f38d415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:47.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.608+0000 7f38d8a00640 1 --2- 172.21.15.6:0/349494778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40ccea0 0x7f38d415ab40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0097a0 tx=0x7f38cc02b820 comp rx=0 tx=0).stop 2024-03-22T12:59:47.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.609+0000 7f38d8a00640 1 -- 172.21.15.6:0/349494778 shutdown_connections 2024-03-22T12:59:47.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.609+0000 7f38d8a00640 1 --2- 172.21.15.6:0/349494778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40ccea0 0x7f38d415ab40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:47.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.609+0000 7f38d8a00640 1 --2- 172.21.15.6:0/349494778 >> v2:172.21.15.6:3300/0 conn(0x7f38d40cc3b0 0x7f38d40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:47.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.609+0000 7f38d8a00640 1 -- 172.21.15.6:0/349494778 >> 172.21.15.6:0/349494778 conn(0x7f38d40b16d0 msgr2=0x7f38d40b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:47.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.609+0000 7f38d8a00640 1 -- 172.21.15.6:0/349494778 shutdown_connections 2024-03-22T12:59:47.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.609+0000 7f38d8a00640 1 -- 172.21.15.6:0/349494778 wait complete. 2024-03-22T12:59:47.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.610+0000 7f38d8a00640 1 Processor -- start 2024-03-22T12:59:47.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.611+0000 7f38d8a00640 1 -- start start 2024-03-22T12:59:47.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.611+0000 7f38d8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38d40cc3b0 0x7f38d40c4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:47.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.611+0000 7f38d8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d40ccea0 0x7f38d40b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:47.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40cf770 0x7f38d40c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:47.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d415a5d0 con 0x7f38d40cc3b0 2024-03-22T12:59:47.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d8a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d406dc10 con 0x7f38d40ccea0 2024-03-22T12:59:47.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4154bb0 con 0x7f38d40cf770 2024-03-22T12:59:47.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38d40cc3b0 0x7f38d40c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:47.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40cf770 0x7f38d40c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:47.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d40ccea0 0x7f38d40b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:47.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.612+0000 7f38d2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40cf770 0x7f38d40c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55396/0 (socket says 172.21.15.6:55396) 2024-03-22T12:59:47.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.613+0000 7f38d2800640 1 -- 172.21.15.6:0/3870924483 learned_addr learned my addr 172.21.15.6:0/3870924483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:47.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.616+0000 7f38d2800640 1 -- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d40ccea0 msgr2=0x7f38d40b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:47.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.616+0000 7f38d2800640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d40ccea0 0x7f38d40b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:47.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.616+0000 7f38d2800640 1 -- 172.21.15.6:0/3870924483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38d40cc3b0 msgr2=0x7f38d40c4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:47.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.616+0000 7f38d2800640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38d40cc3b0 0x7f38d40c4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:47.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.616+0000 7f38d2800640 1 -- 172.21.15.6:0/3870924483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d41f3e60 con 0x7f38d40cf770 2024-03-22T12:59:47.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.619+0000 7f38d2800640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40cf770 0x7f38d40c8d50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f38cc034040 tx=0x7f38cc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:47.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.623+0000 7f38ca000640 1 -- 172.21.15.6:0/3870924483 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc032610 con 0x7f38d40cf770 2024-03-22T12:59:47.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.623+0000 7f38ca000640 1 -- 172.21.15.6:0/3870924483 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc0327b0 con 0x7f38d40cf770 2024-03-22T12:59:47.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.623+0000 7f38ca000640 1 -- 172.21.15.6:0/3870924483 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc032ba0 con 0x7f38d40cf770 2024-03-22T12:59:47.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.623+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d40c7650 con 0x7f38d40cf770 2024-03-22T12:59:47.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.623+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d40c7c10 con 0x7f38d40cf770 2024-03-22T12:59:47.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.625+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d415b550 con 0x7f38d40cf770 2024-03-22T12:59:47.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.649+0000 7f38ca000640 1 -- 172.21.15.6:0/3870924483 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f38cc04c060 con 0x7f38d40cf770 2024-03-22T12:59:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.650+0000 7f38ca000640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f38b006bc20 0x7f38b006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.651+0000 7f38d1e00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f38b006bc20 0x7f38b006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:47.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.651+0000 7f38d1e00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f38b006bc20 0x7f38b006e0e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f38c40113e0 tx=0x7f38c400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.653+0000 7f38ca000640 1 -- 172.21.15.6:0/3870924483 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f38cc078c70 con 0x7f38d40cf770 2024-03-22T12:59:47.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:47.914+0000 7f38ca000640 1 -- 172.21.15.6:0/3870924483 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38d415b550 con 0x7f38d40cf770 2024-03-22T12:59:48.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.167+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38d40756c0 con 0x7f38d40cf770 2024-03-22T12:59:48.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.180+0000 7f38ca000640 1 -- 172.21.15.6:0/3870924483 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f38cc040070 con 0x7f38d40cf770 2024-03-22T12:59:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:48.180 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:48.180 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f38b006bc20 msgr2=0x7f38b006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f38b006bc20 0x7f38b006e0e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f38c40113e0 tx=0x7f38c400b040 comp rx=0 tx=0).stop 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40cf770 msgr2=0x7f38d40c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40cf770 0x7f38d40c8d50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f38cc034040 tx=0x7f38cc0046c0 comp rx=0 tx=0).stop 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 shutdown_connections 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38d40cf770 0x7f38d40c8d50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f38b006bc20 0x7f38b006e0e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d40ccea0 0x7f38d40b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 --2- 172.21.15.6:0/3870924483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38d40cc3b0 0x7f38d40c4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 >> 172.21.15.6:0/3870924483 conn(0x7f38d40b16d0 msgr2=0x7f38d4149ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 shutdown_connections 2024-03-22T12:59:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:48.183+0000 7f38d8a00640 1 -- 172.21.15.6:0/3870924483 wait complete. 2024-03-22T12:59:48.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:48.805 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:49.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:50.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.006+0000 7fda68a00640 1 Processor -- start 2024-03-22T12:59:50.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.006+0000 7fda68a00640 1 -- start start 2024-03-22T12:59:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.007+0000 7fda68a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640ce8a0 0x7fda6406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.007+0000 7fda68a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fda6415ae30 0x7fda640cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.008+0000 7fda68a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6415b260 con 0x7fda640cc380 2024-03-22T12:59:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.008+0000 7fda68a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda640cee00 con 0x7fda6415ae30 2024-03-22T12:59:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.008+0000 7fda68a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640cf0c0 con 0x7fda640ce8a0 2024-03-22T12:59:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.008+0000 7fda63400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fda640cc380 0x7fda640cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55326/0 (socket says 172.21.15.6:55326) 2024-03-22T12:59:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.008+0000 7fda63400640 1 -- 172.21.15.6:0/1441434016 learned_addr learned my addr 172.21.15.6:0/1441434016 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:59:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.008+0000 7fda63e00640 1 --2- 172.21.15.6:0/1441434016 >> v2:172.21.15.92:3300/0 conn(0x7fda6415ae30 0x7fda640cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.008+0000 7fda5aa00640 1 --2- 172.21.15.6:0/1441434016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640ce8a0 0x7fda6406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.011+0000 7fda5aa00640 1 -- 172.21.15.6:0/1441434016 >> v2:172.21.15.92:3300/0 conn(0x7fda6415ae30 msgr2=0x7fda640cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.011+0000 7fda5aa00640 1 --2- 172.21.15.6:0/1441434016 >> v2:172.21.15.92:3300/0 conn(0x7fda6415ae30 0x7fda640cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.011+0000 7fda5aa00640 1 -- 172.21.15.6:0/1441434016 >> v1:172.21.15.6:6789/0 conn(0x7fda640cc380 legacy=0x7fda640cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.011+0000 7fda5aa00640 1 -- 172.21.15.6:0/1441434016 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640cf3a0 con 0x7fda640ce8a0 2024-03-22T12:59:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.014+0000 7fda5aa00640 1 --2- 172.21.15.6:0/1441434016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640ce8a0 0x7fda6406c230 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0097a0 tx=0x7fda4c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7796f2fdcffbfa56 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:50.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.018+0000 7fda62a00640 1 -- 172.21.15.6:0/1441434016 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c035020 con 0x7fda640ce8a0 2024-03-22T12:59:50.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.018+0000 7fda62a00640 1 -- 172.21.15.6:0/1441434016 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c031810 con 0x7fda640ce8a0 2024-03-22T12:59:50.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.018+0000 7fda62a00640 1 -- 172.21.15.6:0/1441434016 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c031be0 con 0x7fda640ce8a0 2024-03-22T12:59:50.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.019+0000 7fda68a00640 1 -- 172.21.15.6:0/1441434016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640ce8a0 msgr2=0x7fda6406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:50.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.019+0000 7fda68a00640 1 --2- 172.21.15.6:0/1441434016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640ce8a0 0x7fda6406c230 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0097a0 tx=0x7fda4c02b8b0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.019+0000 7fda68a00640 1 -- 172.21.15.6:0/1441434016 shutdown_connections 2024-03-22T12:59:50.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.019+0000 7fda68a00640 1 --2- 172.21.15.6:0/1441434016 >> v2:172.21.15.92:3300/0 conn(0x7fda6415ae30 0x7fda640cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.019+0000 7fda68a00640 1 --2- 172.21.15.6:0/1441434016 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640ce8a0 0x7fda6406c230 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.020+0000 7fda68a00640 1 -- 172.21.15.6:0/1441434016 >> 172.21.15.6:0/1441434016 conn(0x7fda640b1480 msgr2=0x7fda640b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.020+0000 7fda68a00640 1 -- 172.21.15.6:0/1441434016 shutdown_connections 2024-03-22T12:59:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.020+0000 7fda68a00640 1 -- 172.21.15.6:0/1441434016 wait complete. 2024-03-22T12:59:50.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.022+0000 7fda68a00640 1 Processor -- start 2024-03-22T12:59:50.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.022+0000 7fda68a00640 1 -- start start 2024-03-22T12:59:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.024+0000 7fda68a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640cc380 0x7fda641532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.024+0000 7fda68a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda640ce8a0 0x7fda641e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.024+0000 7fda68a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda6415ae30 0x7fda641570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.024+0000 7fda68a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6415a910 con 0x7fda6415ae30 2024-03-22T12:59:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.024+0000 7fda68a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6415a630 con 0x7fda640ce8a0 2024-03-22T12:59:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.024+0000 7fda68a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda641549d0 con 0x7fda640cc380 2024-03-22T12:59:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.025+0000 7fda63400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640cc380 0x7fda641532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.025+0000 7fda5aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda640ce8a0 0x7fda641e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.025+0000 7fda63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda6415ae30 0x7fda641570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.025+0000 7fda63400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640cc380 0x7fda641532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55400/0 (socket says 172.21.15.6:55400) 2024-03-22T12:59:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.025+0000 7fda63400640 1 -- 172.21.15.6:0/1226964867 learned_addr learned my addr 172.21.15.6:0/1226964867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.028+0000 7fda5aa00640 1 -- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640cc380 msgr2=0x7fda641532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.028+0000 7fda5aa00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640cc380 0x7fda641532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.028+0000 7fda5aa00640 1 -- 172.21.15.6:0/1226964867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda6415ae30 msgr2=0x7fda641570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.028+0000 7fda5aa00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda6415ae30 0x7fda641570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.029+0000 7fda5aa00640 1 -- 172.21.15.6:0/1226964867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda641e3650 con 0x7fda640ce8a0 2024-03-22T12:59:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.031+0000 7fda5aa00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda640ce8a0 0x7fda641e4a80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda4c033000 tx=0x7fda4c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.033+0000 7fda5be00640 1 -- 172.21.15.6:0/1226964867 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c041070 con 0x7fda640ce8a0 2024-03-22T12:59:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.033+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda641e3930 con 0x7fda640ce8a0 2024-03-22T12:59:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.033+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda641568e0 con 0x7fda640ce8a0 2024-03-22T12:59:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.034+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda641e3df0 con 0x7fda640ce8a0 2024-03-22T12:59:50.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.241+0000 7fda5be00640 1 -- 172.21.15.6:0/1226964867 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c031810 con 0x7fda640ce8a0 2024-03-22T12:59:50.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.241+0000 7fda5be00640 1 -- 172.21.15.6:0/1226964867 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c035040 con 0x7fda640ce8a0 2024-03-22T12:59:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.246+0000 7fda5be00640 1 -- 172.21.15.6:0/1226964867 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fda4c04c060 con 0x7fda640ce8a0 2024-03-22T12:59:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.248+0000 7fda5be00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda4006bbd0 0x7fda4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.248+0000 7fda63400640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda4006bbd0 0x7fda4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.249+0000 7fda63400640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda4006bbd0 0x7fda4006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda54001e90 tx=0x7fda54001a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.329+0000 7fda5be00640 1 -- 172.21.15.6:0/1226964867 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fda4c0788f0 con 0x7fda640ce8a0 2024-03-22T12:59:50.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.341+0000 7fda5be00640 1 -- 172.21.15.6:0/1226964867 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fda4c07a850 con 0x7fda640ce8a0 2024-03-22T12:59:50.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.627+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda64075650 con 0x7fda640ce8a0 2024-03-22T12:59:50.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.651+0000 7fda5be00640 1 -- 172.21.15.6:0/1226964867 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fda4c07e8f0 con 0x7fda640ce8a0 2024-03-22T12:59:50.651 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:50.651 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:50.651 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.654+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda4006bbd0 msgr2=0x7fda4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.654+0000 7fda68a00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda4006bbd0 0x7fda4006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda54001e90 tx=0x7fda54001a20 comp rx=0 tx=0).stop 2024-03-22T12:59:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.654+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda640ce8a0 msgr2=0x7fda641e4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.654+0000 7fda68a00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda640ce8a0 0x7fda641e4a80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda4c033000 tx=0x7fda4c0046c0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 shutdown_connections 2024-03-22T12:59:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda6415ae30 0x7fda641570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fda4006bbd0 0x7fda4006e090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda640ce8a0 0x7fda641e4a80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 --2- 172.21.15.6:0/1226964867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda640cc380 0x7fda641532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:50.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 >> 172.21.15.6:0/1226964867 conn(0x7fda640b1480 msgr2=0x7fda64157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:50.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 shutdown_connections 2024-03-22T12:59:50.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:50.655+0000 7fda68a00640 1 -- 172.21.15.6:0/1226964867 wait complete. 2024-03-22T12:59:51.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:51.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:52.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.472+0000 7fb9bb200640 1 Processor -- start 2024-03-22T12:59:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.472+0000 7fb9bb200640 1 -- start start 2024-03-22T12:59:52.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.474+0000 7fb9bb200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb9bc154540 0x7fb9bc15e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:52.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.474+0000 7fb9bb200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc0c91e0 0x7fb9bc0c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:52.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.474+0000 7fb9bb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9bc15e7f0 con 0x7fb9bc06ede0 2024-03-22T12:59:52.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.474+0000 7fb9bb200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9bc0c79e0 con 0x7fb9bc154540 2024-03-22T12:59:52.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.474+0000 7fb9bb200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc15b2d0 con 0x7fb9bc0c91e0 2024-03-22T12:59:52.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.475+0000 7fb9ba800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc0c91e0 0x7fb9bc0c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:52.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.475+0000 7fb9b9e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb9bc154540 0x7fb9bc15e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:52.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.475+0000 7fb9ba800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc0c91e0 0x7fb9bc0c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55412/0 (socket says 172.21.15.6:55412) 2024-03-22T12:59:52.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.475+0000 7fb9ba800640 1 -- 172.21.15.6:0/2476569311 learned_addr learned my addr 172.21.15.6:0/2476569311 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:52.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.478+0000 7fb9ba800640 1 -- 172.21.15.6:0/2476569311 >> v2:172.21.15.92:3300/0 conn(0x7fb9bc154540 msgr2=0x7fb9bc15e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:52.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.478+0000 7fb9ba800640 1 --2- 172.21.15.6:0/2476569311 >> v2:172.21.15.92:3300/0 conn(0x7fb9bc154540 0x7fb9bc15e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:52.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.478+0000 7fb9ba800640 1 -- 172.21.15.6:0/2476569311 >> v1:172.21.15.6:6789/0 conn(0x7fb9bc06ede0 legacy=0x7fb9bc15abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:52.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.478+0000 7fb9ba800640 1 -- 172.21.15.6:0/2476569311 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc070890 con 0x7fb9bc0c91e0 2024-03-22T12:59:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.529+0000 7fb9ba800640 1 --2- 172.21.15.6:0/2476569311 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc0c91e0 0x7fb9bc0c9670 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009e50 tx=0x7fb9b402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db73e3d33d18873b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.535+0000 7fb9b8a00640 1 -- 172.21.15.6:0/2476569311 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b402cb00 con 0x7fb9bc0c91e0 2024-03-22T12:59:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.535+0000 7fb9b8a00640 1 -- 172.21.15.6:0/2476569311 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b402cca0 con 0x7fb9bc0c91e0 2024-03-22T12:59:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.535+0000 7fb9b8a00640 1 -- 172.21.15.6:0/2476569311 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4034610 con 0x7fb9bc0c91e0 2024-03-22T12:59:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 -- 172.21.15.6:0/2476569311 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc0c91e0 msgr2=0x7fb9bc0c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:52.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 --2- 172.21.15.6:0/2476569311 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc0c91e0 0x7fb9bc0c9670 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009e50 tx=0x7fb9b402bf60 comp rx=0 tx=0).stop 2024-03-22T12:59:52.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 -- 172.21.15.6:0/2476569311 shutdown_connections 2024-03-22T12:59:52.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 --2- 172.21.15.6:0/2476569311 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc0c91e0 0x7fb9bc0c9670 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:52.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 --2- 172.21.15.6:0/2476569311 >> v2:172.21.15.92:3300/0 conn(0x7fb9bc154540 0x7fb9bc15e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:52.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 -- 172.21.15.6:0/2476569311 >> 172.21.15.6:0/2476569311 conn(0x7fb9bc06c190 msgr2=0x7fb9bc06c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:52.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 -- 172.21.15.6:0/2476569311 shutdown_connections 2024-03-22T12:59:52.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.536+0000 7fb9bb200640 1 -- 172.21.15.6:0/2476569311 wait complete. 2024-03-22T12:59:52.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.539+0000 7fb9bb200640 1 Processor -- start 2024-03-22T12:59:52.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.539+0000 7fb9bb200640 1 -- start start 2024-03-22T12:59:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.540+0000 7fb9bb200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc06ede0 0x7fb9bc149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.540+0000 7fb9bb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9bc0c91e0 0x7fb9bc1e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.540+0000 7fb9bb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9bc154540 0x7fb9bc144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.540+0000 7fb9bb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc15a730 con 0x7fb9bc0c91e0 2024-03-22T12:59:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.540+0000 7fb9bb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc0c9bb0 con 0x7fb9bc154540 2024-03-22T12:59:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.540+0000 7fb9bb200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc15a3c0 con 0x7fb9bc06ede0 2024-03-22T12:59:52.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.541+0000 7fb9b9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc06ede0 0x7fb9bc149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:52.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.541+0000 7fb9b9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc06ede0 0x7fb9bc149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55428/0 (socket says 172.21.15.6:55428) 2024-03-22T12:59:52.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.541+0000 7fb9b9e00640 1 -- 172.21.15.6:0/1809054435 learned_addr learned my addr 172.21.15.6:0/1809054435 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:52.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.541+0000 7fb9b9400640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9bc0c91e0 0x7fb9bc1e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:52.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.545+0000 7fb9b9e00640 1 -- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9bc154540 msgr2=0x7fb9bc144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:52.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.545+0000 7fb9b9e00640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9bc154540 0x7fb9bc144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:52.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.545+0000 7fb9b9e00640 1 -- 172.21.15.6:0/1809054435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9bc0c91e0 msgr2=0x7fb9bc1e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:52.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.545+0000 7fb9b9e00640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9bc0c91e0 0x7fb9bc1e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:52.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.545+0000 7fb9b9e00640 1 -- 172.21.15.6:0/1809054435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc1e42e0 con 0x7fb9bc06ede0 2024-03-22T12:59:52.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.547+0000 7fb9b9e00640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc06ede0 0x7fb9bc149a20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00e870 tx=0x7fb9ac00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.550+0000 7fb9b2000640 1 -- 172.21.15.6:0/1809054435 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac017070 con 0x7fb9bc06ede0 2024-03-22T12:59:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.550+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9bc1f0c40 con 0x7fb9bc06ede0 2024-03-22T12:59:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.550+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9bc1efe80 con 0x7fb9bc06ede0 2024-03-22T12:59:52.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.551+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9bc06e900 con 0x7fb9bc06ede0 2024-03-22T12:59:52.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.769+0000 7fb9b2000640 1 -- 172.21.15.6:0/1809054435 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ac005990 con 0x7fb9bc06ede0 2024-03-22T12:59:52.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.810+0000 7fb9b2000640 1 -- 172.21.15.6:0/1809054435 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac021060 con 0x7fb9bc06ede0 2024-03-22T12:59:52.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.837+0000 7fb9b2000640 1 -- 172.21.15.6:0/1809054435 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb9ac01a070 con 0x7fb9bc06ede0 2024-03-22T12:59:52.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.839+0000 7fb9b2000640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb99c06bbd0 0x7fb99c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:52.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.839+0000 7fb9b2000640 1 -- 172.21.15.6:0/1809054435 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb9ac08b5a0 con 0x7fb9bc06ede0 2024-03-22T12:59:52.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.840+0000 7fb9b9400640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb99c06bbd0 0x7fb99c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:52.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.841+0000 7fb9b9400640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb99c06bbd0 0x7fb99c06e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc151230 tx=0x7fb9a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:52.857+0000 7fb9b2000640 1 -- 172.21.15.6:0/1809054435 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb9bc06e900 con 0x7fb9bc06ede0 2024-03-22T12:59:53.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.111+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9bc075650 con 0x7fb9bc06ede0 2024-03-22T12:59:53.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.127+0000 7fb9b2000640 1 -- 172.21.15.6:0/1809054435 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fb9ac059520 con 0x7fb9bc06ede0 2024-03-22T12:59:53.127 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:53.127 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:53.127 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb99c06bbd0 msgr2=0x7fb99c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb99c06bbd0 0x7fb99c06e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc151230 tx=0x7fb9a8009040 comp rx=0 tx=0).stop 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc06ede0 msgr2=0x7fb9bc149a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc06ede0 0x7fb9bc149a20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00e870 tx=0x7fb9ac00ed30 comp rx=0 tx=0).stop 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 shutdown_connections 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb99c06bbd0 0x7fb99c06e090 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9bc154540 0x7fb9bc144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:53.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9bc0c91e0 0x7fb9bc1e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:53.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 --2- 172.21.15.6:0/1809054435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9bc06ede0 0x7fb9bc149a20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:53.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 >> 172.21.15.6:0/1809054435 conn(0x7fb9bc06c190 msgr2=0x7fb9bc15abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:53.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 shutdown_connections 2024-03-22T12:59:53.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:53.129+0000 7fb9bb200640 1 -- 172.21.15.6:0/1809054435 wait complete. 2024-03-22T12:59:53.750 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:53.750 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:54.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.943+0000 7efef0600640 1 Processor -- start 2024-03-22T12:59:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.944+0000 7efef0600640 1 -- start start 2024-03-22T12:59:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.945+0000 7efef0600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efee8154240 0x7efee815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.945+0000 7efef0600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80c4740 con 0x7efee80c68c0 2024-03-22T12:59:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.945+0000 7efef0600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee8154620 con 0x7efee80c7ee0 2024-03-22T12:59:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.945+0000 7efef0600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee815e130 con 0x7efee8154240 2024-03-22T12:59:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.946+0000 7efeee600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efee80c68c0 0x7efee806b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59684/0 (socket says 172.21.15.6:59684) 2024-03-22T12:59:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.946+0000 7efeee600640 1 -- 172.21.15.6:0/3084920175 learned_addr learned my addr 172.21.15.6:0/3084920175 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T12:59:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.946+0000 7efeedc00640 1 --2- 172.21.15.6:0/3084920175 >> v2:172.21.15.92:3300/0 conn(0x7efee8154240 0x7efee815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.949+0000 7efee7e00640 1 -- 172.21.15.6:0/3084920175 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303107820 0 0) 0x7efee80c4740 con 0x7efee80c68c0 2024-03-22T12:59:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.949+0000 7efee7e00640 1 -- 172.21.15.6:0/3084920175 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0003520 con 0x7efee80c68c0 2024-03-22T12:59:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.949+0000 7efeedc00640 1 -- 172.21.15.6:0/3084920175 >> v1:172.21.15.92:6789/0 conn(0x7efee80c7ee0 legacy=0x7efee80c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.949+0000 7efeedc00640 1 -- 172.21.15.6:0/3084920175 >> v1:172.21.15.6:6789/0 conn(0x7efee80c68c0 legacy=0x7efee806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.949+0000 7efeedc00640 1 -- 172.21.15.6:0/3084920175 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80c6e00 con 0x7efee8154240 2024-03-22T12:59:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.954+0000 7efeedc00640 1 --2- 172.21.15.6:0/3084920175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee815ac90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efedc0097a0 tx=0x7efedc02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85dd3a9ae6f25503 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:54.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.958+0000 7efee7e00640 1 -- 172.21.15.6:0/3084920175 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc02f810 con 0x7efee8154240 2024-03-22T12:59:54.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.958+0000 7efee7e00640 1 -- 172.21.15.6:0/3084920175 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc02f9b0 con 0x7efee8154240 2024-03-22T12:59:54.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.958+0000 7efee7e00640 1 -- 172.21.15.6:0/3084920175 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc02fd80 con 0x7efee8154240 2024-03-22T12:59:54.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.959+0000 7efef0600640 1 -- 172.21.15.6:0/3084920175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 msgr2=0x7efee815ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:54.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.959+0000 7efef0600640 1 --2- 172.21.15.6:0/3084920175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee815ac90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efedc0097a0 tx=0x7efedc02b9f0 comp rx=0 tx=0).stop 2024-03-22T12:59:54.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.959+0000 7efef0600640 1 -- 172.21.15.6:0/3084920175 shutdown_connections 2024-03-22T12:59:54.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.959+0000 7efef0600640 1 --2- 172.21.15.6:0/3084920175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee815ac90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.959+0000 7efef0600640 1 -- 172.21.15.6:0/3084920175 >> 172.21.15.6:0/3084920175 conn(0x7efee80cb930 msgr2=0x7efee80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.960+0000 7efef0600640 1 -- 172.21.15.6:0/3084920175 shutdown_connections 2024-03-22T12:59:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.960+0000 7efef0600640 1 -- 172.21.15.6:0/3084920175 wait complete. 2024-03-22T12:59:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.962+0000 7efef0600640 1 Processor -- start 2024-03-22T12:59:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.962+0000 7efef0600640 1 -- start start 2024-03-22T12:59:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.963+0000 7efef0600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efee80c68c0 0x7efee81519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.963+0000 7efef0600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efee80c7ee0 0x7efee80727f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.963+0000 7efef0600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee8071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.963+0000 7efef0600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee815dae0 con 0x7efee80c68c0 2024-03-22T12:59:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.963+0000 7efef0600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee815d800 con 0x7efee8154240 2024-03-22T12:59:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.963+0000 7efef0600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee815ddc0 con 0x7efee80c7ee0 2024-03-22T12:59:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.964+0000 7efeedc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efee80c68c0 0x7efee81519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.964+0000 7efeee600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee8071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.964+0000 7efeed200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efee80c7ee0 0x7efee80727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.964+0000 7efeed200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efee80c7ee0 0x7efee80727f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35022/0 (socket says 172.21.15.6:35022) 2024-03-22T12:59:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.964+0000 7efeed200640 1 -- 172.21.15.6:0/1047104511 learned_addr learned my addr 172.21.15.6:0/1047104511 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:54.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.969+0000 7efeed200640 1 -- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 msgr2=0x7efee8071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:54.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.969+0000 7efeed200640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee8071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:54.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.969+0000 7efeed200640 1 -- 172.21.15.6:0/1047104511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efee80c68c0 msgr2=0x7efee81519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:54.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.969+0000 7efeed200640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efee80c68c0 0x7efee81519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:54.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.969+0000 7efeed200640 1 -- 172.21.15.6:0/1047104511 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80cb6b0 con 0x7efee80c7ee0 2024-03-22T12:59:54.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.969+0000 7efeee600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee8071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T12:59:54.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.973+0000 7efeed200640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efee80c7ee0 0x7efee80727f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efed400c460 tx=0x7efed400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:54.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.977+0000 7efee5600640 1 -- 172.21.15.6:0/1047104511 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed40052d0 con 0x7efee80c7ee0 2024-03-22T12:59:54.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.977+0000 7efee5600640 1 -- 172.21.15.6:0/1047104511 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed4005470 con 0x7efee80c7ee0 2024-03-22T12:59:54.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.977+0000 7efee5600640 1 -- 172.21.15.6:0/1047104511 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4012870 con 0x7efee80c7ee0 2024-03-22T12:59:54.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.977+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee80c97d0 con 0x7efee80c7ee0 2024-03-22T12:59:54.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.977+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee80c9d90 con 0x7efee80c7ee0 2024-03-22T12:59:54.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.978+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee80756c0 con 0x7efee80c7ee0 2024-03-22T12:59:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.998+0000 7efee5600640 1 -- 172.21.15.6:0/1047104511 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efed4012a10 con 0x7efee80c7ee0 2024-03-22T12:59:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:54.999+0000 7efee5600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efec806bc20 0x7efec806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.000+0000 7efeedc00640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efec806bc20 0x7efec806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.000+0000 7efeedc00640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efec806bc20 0x7efec806e0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efedc036000 tx=0x7efedc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.002+0000 7efee5600640 1 -- 172.21.15.6:0/1047104511 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7efed40574d0 con 0x7efee80c7ee0 2024-03-22T12:59:55.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.263+0000 7efee5600640 1 -- 172.21.15.6:0/1047104511 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efed4059430 con 0x7efee80c7ee0 2024-03-22T12:59:55.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.547+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efee80c7ab0 con 0x7efee80c7ee0 2024-03-22T12:59:55.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.562+0000 7efee5600640 1 -- 172.21.15.6:0/1047104511 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7efed405d4d0 con 0x7efee80c7ee0 2024-03-22T12:59:55.562 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:55.562 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:55.562 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:55.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efec806bc20 msgr2=0x7efec806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:55.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efec806bc20 0x7efec806e0e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efedc036000 tx=0x7efedc008030 comp rx=0 tx=0).stop 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efee80c7ee0 msgr2=0x7efee80727f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efee80c7ee0 0x7efee80727f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efed400c460 tx=0x7efed400c920 comp rx=0 tx=0).stop 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 shutdown_connections 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efec806bc20 0x7efec806e0e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efee8154240 0x7efee8071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efee80c7ee0 0x7efee80727f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 --2- 172.21.15.6:0/1047104511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efee80c68c0 0x7efee81519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.564+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 >> 172.21.15.6:0/1047104511 conn(0x7efee80cb930 msgr2=0x7efee80c52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.565+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 shutdown_connections 2024-03-22T12:59:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:55.565+0000 7efef0600640 1 -- 172.21.15.6:0/1047104511 wait complete. 2024-03-22T12:59:56.186 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:56.186 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:57.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:57.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.376+0000 7f034d000640 1 Processor -- start 2024-03-22T12:59:57.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.377+0000 7f034d000640 1 -- start start 2024-03-22T12:59:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.378+0000 7f034d000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f03480ca520 0x7f034806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.378+0000 7f034d000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03480ccad0 0x7f034815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.378+0000 7f034d000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034806c7a0 con 0x7f034815b0e0 2024-03-22T12:59:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.378+0000 7f034d000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03480cd000 con 0x7f03480ca520 2024-03-22T12:59:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.378+0000 7f034d000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034806ca80 con 0x7f03480ccad0 2024-03-22T12:59:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.379+0000 7f0345e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f03480ca520 0x7f034806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.379+0000 7f0346800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03480ccad0 0x7f034815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.379+0000 7f0346800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03480ccad0 0x7f034815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35038/0 (socket says 172.21.15.6:35038) 2024-03-22T12:59:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.379+0000 7f0346800640 1 -- 172.21.15.6:0/3172666765 learned_addr learned my addr 172.21.15.6:0/3172666765 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:57.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.383+0000 7f0346800640 1 -- 172.21.15.6:0/3172666765 >> v2:172.21.15.92:3300/0 conn(0x7f03480ca520 msgr2=0x7f034806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:57.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.383+0000 7f0346800640 1 --2- 172.21.15.6:0/3172666765 >> v2:172.21.15.92:3300/0 conn(0x7f03480ca520 0x7f034806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:57.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.383+0000 7f0346800640 1 -- 172.21.15.6:0/3172666765 >> v1:172.21.15.92:6789/0 conn(0x7f034815b0e0 legacy=0x7f03480cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:57.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.383+0000 7f0346800640 1 -- 172.21.15.6:0/3172666765 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034806cd60 con 0x7f03480ccad0 2024-03-22T12:59:57.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.386+0000 7f0346800640 1 --2- 172.21.15.6:0/3172666765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03480ccad0 0x7f034815b980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0340009e50 tx=0x7f034002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc3c56e74a271d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:57.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.390+0000 7f0344a00640 1 -- 172.21.15.6:0/3172666765 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034002cb00 con 0x7f03480ccad0 2024-03-22T12:59:57.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.390+0000 7f0344a00640 1 -- 172.21.15.6:0/3172666765 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034002cca0 con 0x7f03480ccad0 2024-03-22T12:59:57.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.390+0000 7f0344a00640 1 -- 172.21.15.6:0/3172666765 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340034610 con 0x7f03480ccad0 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 -- 172.21.15.6:0/3172666765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03480ccad0 msgr2=0x7f034815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 --2- 172.21.15.6:0/3172666765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03480ccad0 0x7f034815b980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0340009e50 tx=0x7f034002bf60 comp rx=0 tx=0).stop 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 -- 172.21.15.6:0/3172666765 shutdown_connections 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 --2- 172.21.15.6:0/3172666765 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f03480ccad0 0x7f034815b980 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 --2- 172.21.15.6:0/3172666765 >> v2:172.21.15.92:3300/0 conn(0x7f03480ca520 0x7f034806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 -- 172.21.15.6:0/3172666765 >> 172.21.15.6:0/3172666765 conn(0x7f03480b16b0 msgr2=0x7f03480b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 -- 172.21.15.6:0/3172666765 shutdown_connections 2024-03-22T12:59:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.391+0000 7f034d000640 1 -- 172.21.15.6:0/3172666765 wait complete. 2024-03-22T12:59:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.393+0000 7f034d000640 1 Processor -- start 2024-03-22T12:59:57.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.393+0000 7f034d000640 1 -- start start 2024-03-22T12:59:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.394+0000 7f034d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03480ca520 0x7f03481f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.394+0000 7f034d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480ccad0 0x7f03480c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.394+0000 7f034d000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f034815b0e0 0x7f03481e5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.394+0000 7f034d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0348154c20 con 0x7f03480ca520 2024-03-22T12:59:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.394+0000 7f034d000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034806d320 con 0x7f03480ccad0 2024-03-22T12:59:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.394+0000 7f034d000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03480ca900 con 0x7f034815b0e0 2024-03-22T12:59:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.395+0000 7f0346800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f034815b0e0 0x7f03481e5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.395+0000 7f0345400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480ccad0 0x7f03480c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.395+0000 7f0345e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03480ca520 0x7f03481f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.395+0000 7f0346800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f034815b0e0 0x7f03481e5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35044/0 (socket says 172.21.15.6:35044) 2024-03-22T12:59:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.395+0000 7f0346800640 1 -- 172.21.15.6:0/1399443882 learned_addr learned my addr 172.21.15.6:0/1399443882 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:57.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.400+0000 7f0346800640 1 -- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480ccad0 msgr2=0x7f03480c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:57.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.400+0000 7f0346800640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480ccad0 0x7f03480c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:57.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.400+0000 7f0346800640 1 -- 172.21.15.6:0/1399443882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03480ca520 msgr2=0x7f03481f0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:57.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.400+0000 7f0346800640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03480ca520 0x7f03481f0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:57.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.400+0000 7f0346800640 1 -- 172.21.15.6:0/1399443882 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03481f1520 con 0x7f034815b0e0 2024-03-22T12:59:57.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.403+0000 7f0346800640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f034815b0e0 0x7f03481e5c40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0340036040 tx=0x7f0340004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:57.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.408+0000 7f033e000640 1 -- 172.21.15.6:0/1399443882 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340043020 con 0x7f034815b0e0 2024-03-22T12:59:57.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.408+0000 7f033e000640 1 -- 172.21.15.6:0/1399443882 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034002ce50 con 0x7f034815b0e0 2024-03-22T12:59:57.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.408+0000 7f033e000640 1 -- 172.21.15.6:0/1399443882 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034003e690 con 0x7f034815b0e0 2024-03-22T12:59:57.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.408+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03481e4e50 con 0x7f034815b0e0 2024-03-22T12:59:57.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.408+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03481f07e0 con 0x7f034815b0e0 2024-03-22T12:59:57.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.409+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03481f05f0 con 0x7f034815b0e0 2024-03-22T12:59:57.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.711+0000 7f033e000640 1 -- 172.21.15.6:0/1399443882 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0340046020 con 0x7f034815b0e0 2024-03-22T12:59:57.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.713+0000 7f033e000640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032806bc50 0x7f032806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:57.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.714+0000 7f0345e00640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032806bc50 0x7f032806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:57.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.715+0000 7f0345e00640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032806bc50 0x7f032806e110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0338009b40 tx=0x7f0338009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:57.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.716+0000 7f033e000640 1 -- 172.21.15.6:0/1399443882 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0340079880 con 0x7f034815b0e0 2024-03-22T12:59:57.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.730+0000 7f033e000640 1 -- 172.21.15.6:0/1399443882 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f034004d030 con 0x7f034815b0e0 2024-03-22T12:59:57.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:57.984+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03480756c0 con 0x7f034815b0e0 2024-03-22T12:59:58.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.008+0000 7f033e000640 1 -- 172.21.15.6:0/1399443882 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f034007b7e0 con 0x7f034815b0e0 2024-03-22T12:59:58.008 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T12:59:58.008 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T12:59:58.008 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T12:59:58.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.010+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032806bc50 msgr2=0x7f032806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.010+0000 7f034d000640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032806bc50 0x7f032806e110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0338009b40 tx=0x7f0338009190 comp rx=0 tx=0).stop 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.010+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f034815b0e0 msgr2=0x7f03481e5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.010+0000 7f034d000640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f034815b0e0 0x7f03481e5c40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0340036040 tx=0x7f0340004a40 comp rx=0 tx=0).stop 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 shutdown_connections 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f034815b0e0 0x7f03481e5c40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f032806bc50 0x7f032806e110 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480ccad0 0x7f03480c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 --2- 172.21.15.6:0/1399443882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03480ca520 0x7f03481f0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 >> 172.21.15.6:0/1399443882 conn(0x7f03480b16b0 msgr2=0x7f03480cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 shutdown_connections 2024-03-22T12:59:58.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:58.011+0000 7f034d000640 1 -- 172.21.15.6:0/1399443882 wait complete. 2024-03-22T12:59:58.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T12:59:58.631 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T12:59:59.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T12:59:59.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.844+0000 7f6ddfe00640 1 Processor -- start 2024-03-22T12:59:59.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.844+0000 7f6ddfe00640 1 -- start start 2024-03-22T12:59:59.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.845+0000 7f6ddfe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6de00cc380 0x7f6de006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:59.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.846+0000 7f6ddfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:59.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.846+0000 7f6ddfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de015b260 con 0x7f6de00ce8a0 2024-03-22T12:59:59.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.846+0000 7f6ddfe00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de00cee00 con 0x7f6de00cc380 2024-03-22T12:59:59.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.846+0000 7f6ddfe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de00cf0c0 con 0x7f6de015ae30 2024-03-22T12:59:59.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.846+0000 7f6ddf400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:59.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.847+0000 7f6ddf400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de00cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35052/0 (socket says 172.21.15.6:35052) 2024-03-22T12:59:59.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.847+0000 7f6ddf400640 1 -- 172.21.15.6:0/4142751802 learned_addr learned my addr 172.21.15.6:0/4142751802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:59.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.849+0000 7f6ddea00640 1 --2- 172.21.15.6:0/4142751802 >> v2:172.21.15.92:3300/0 conn(0x7f6de00cc380 0x7f6de006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.850+0000 7f6ddf400640 1 -- 172.21.15.6:0/4142751802 >> v2:172.21.15.92:3300/0 conn(0x7f6de00cc380 msgr2=0x7f6de006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T12:59:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.850+0000 7f6ddf400640 1 --2- 172.21.15.6:0/4142751802 >> v2:172.21.15.92:3300/0 conn(0x7f6de00cc380 0x7f6de006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.850+0000 7f6ddf400640 1 -- 172.21.15.6:0/4142751802 >> v1:172.21.15.6:6789/0 conn(0x7f6de00ce8a0 legacy=0x7f6de00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.850+0000 7f6ddf400640 1 -- 172.21.15.6:0/4142751802 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00cf3a0 con 0x7f6de015ae30 2024-03-22T12:59:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.853+0000 7f6ddf400640 1 --2- 172.21.15.6:0/4142751802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de00cc760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009e50 tx=0x7f6dd802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4774650b3788cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.858+0000 7f6ddd600640 1 -- 172.21.15.6:0/4142751802 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd802cb00 con 0x7f6de015ae30 2024-03-22T12:59:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddd600640 1 -- 172.21.15.6:0/4142751802 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd802cca0 con 0x7f6de015ae30 2024-03-22T12:59:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddd600640 1 -- 172.21.15.6:0/4142751802 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8034610 con 0x7f6de015ae30 2024-03-22T12:59:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddfe00640 1 -- 172.21.15.6:0/4142751802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 msgr2=0x7f6de00cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/4142751802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de00cc760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009e50 tx=0x7f6dd802bf60 comp rx=0 tx=0).stop 2024-03-22T12:59:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddfe00640 1 -- 172.21.15.6:0/4142751802 shutdown_connections 2024-03-22T12:59:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/4142751802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de00cc760 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/4142751802 >> v2:172.21.15.92:3300/0 conn(0x7f6de00cc380 0x7f6de006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.859+0000 7f6ddfe00640 1 -- 172.21.15.6:0/4142751802 >> 172.21.15.6:0/4142751802 conn(0x7f6de00b1480 msgr2=0x7f6de00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T12:59:59.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.860+0000 7f6ddfe00640 1 -- 172.21.15.6:0/4142751802 shutdown_connections 2024-03-22T12:59:59.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.860+0000 7f6ddfe00640 1 -- 172.21.15.6:0/4142751802 wait complete. 2024-03-22T12:59:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.861+0000 7f6ddfe00640 1 Processor -- start 2024-03-22T12:59:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.862+0000 7f6ddfe00640 1 -- start start 2024-03-22T12:59:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.862+0000 7f6ddfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6de00cc380 0x7f6de01e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.862+0000 7f6ddfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6de00ce8a0 0x7f6de015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.862+0000 7f6ddfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de015e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T12:59:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.862+0000 7f6ddfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de015a910 con 0x7f6de00ce8a0 2024-03-22T12:59:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.862+0000 7f6ddfe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de015a630 con 0x7f6de00cc380 2024-03-22T12:59:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.862+0000 7f6ddfe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de01549d0 con 0x7f6de015ae30 2024-03-22T12:59:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.863+0000 7f6ddf400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.863+0000 7f6ddea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6de00cc380 0x7f6de01e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.863+0000 7f6dde000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6de00ce8a0 0x7f6de015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T12:59:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.863+0000 7f6ddf400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de015e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35054/0 (socket says 172.21.15.6:35054) 2024-03-22T12:59:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.863+0000 7f6ddf400640 1 -- 172.21.15.6:0/1841356708 learned_addr learned my addr 172.21.15.6:0/1841356708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T12:59:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.867+0000 7f6dde000640 1 -- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 msgr2=0x7f6de015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.867+0000 7f6dde000640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de015e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.867+0000 7f6dde000640 1 -- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6de00cc380 msgr2=0x7f6de01e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T12:59:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.867+0000 7f6dde000640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6de00cc380 0x7f6de01e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T12:59:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.867+0000 7f6dde000640 1 -- 172.21.15.6:0/1841356708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00b4230 con 0x7f6de00ce8a0 2024-03-22T12:59:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.870+0000 7f6dde000640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6de00ce8a0 0x7f6de015dac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00c3c0 tx=0x7f6dcc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T12:59:59.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.872+0000 7f6dd6a00640 1 -- 172.21.15.6:0/1841356708 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc0052d0 con 0x7f6de00ce8a0 2024-03-22T12:59:59.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.872+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de00b2940 con 0x7f6de00ce8a0 2024-03-22T12:59:59.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.873+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de00b3f50 con 0x7f6de00ce8a0 2024-03-22T12:59:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.874+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de0075650 con 0x7f6de00ce8a0 2024-03-22T12:59:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.875+0000 7f6dd6a00640 1 -- 172.21.15.6:0/1841356708 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dcc005470 con 0x7f6de00ce8a0 2024-03-22T12:59:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T12:59:59.875+0000 7f6dd6a00640 1 -- 172.21.15.6:0/1841356708 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc020060 con 0x7f6de00ce8a0 2024-03-22T13:00:00.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:00:00.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:00:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.195+0000 7f6dd6a00640 1 -- 172.21.15.6:0/1841356708 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6dcc007240 con 0x7f6de00ce8a0 2024-03-22T13:00:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.197+0000 7f6dd6a00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6dc006bbd0 0x7f6dc006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.197+0000 7f6ddea00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6dc006bbd0 0x7f6dc006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.198+0000 7f6ddea00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6dc006bbd0 0x7f6dc006e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009aa0 tx=0x7f6dd0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:00.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.220+0000 7f6dd6a00640 1 -- 172.21.15.6:0/1841356708 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6dcc0567a0 con 0x7f6de00ce8a0 2024-03-22T13:00:00.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.220+0000 7f6dd6a00640 1 -- 172.21.15.6:0/1841356708 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6de015b260 con 0x7f6de00ce8a0 2024-03-22T13:00:00.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.557+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de00c9d30 con 0x7f6de00ce8a0 2024-03-22T13:00:00.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.587+0000 7f6dd6a00640 1 -- 172.21.15.6:0/1841356708 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f6dcc058700 con 0x7f6de00ce8a0 2024-03-22T13:00:00.587 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:00.587 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:00.587 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6dc006bbd0 msgr2=0x7f6dc006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6dc006bbd0 0x7f6dc006e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009aa0 tx=0x7f6dd0009190 comp rx=0 tx=0).stop 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6de00ce8a0 msgr2=0x7f6de015dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6de00ce8a0 0x7f6de015dac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00c3c0 tx=0x7f6dcc00c880 comp rx=0 tx=0).stop 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 shutdown_connections 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6de015ae30 0x7f6de015e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6de00ce8a0 0x7f6de015dac0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6dc006bbd0 0x7f6dc006e090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.589+0000 7f6ddfe00640 1 --2- 172.21.15.6:0/1841356708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6de00cc380 0x7f6de01e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.590+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 >> 172.21.15.6:0/1841356708 conn(0x7f6de00b1480 msgr2=0x7f6de01596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.590+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 shutdown_connections 2024-03-22T13:00:00.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:00.590+0000 7f6ddfe00640 1 -- 172.21.15.6:0/1841356708 wait complete. 2024-03-22T13:00:01.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:01.213 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:02.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.409+0000 7f1c40a00640 1 Processor -- start 2024-03-22T13:00:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.410+0000 7f1c40a00640 1 -- start start 2024-03-22T13:00:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.411+0000 7f1c40a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c3c0c7ee0 0x7f1c3c15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.411+0000 7f1c40a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c0c68c0 0x7f1c3c06b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.411+0000 7f1c40a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0c4740 con 0x7f1c3c154240 2024-03-22T13:00:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.411+0000 7f1c40a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3c15e130 con 0x7f1c3c0c7ee0 2024-03-22T13:00:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.411+0000 7f1c40a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c154620 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.412+0000 7f1c3a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c0c68c0 0x7f1c3c06b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.412+0000 7f1c39400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c3c0c7ee0 0x7f1c3c15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:02.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.412+0000 7f1c3a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c0c68c0 0x7f1c3c06b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35068/0 (socket says 172.21.15.6:35068) 2024-03-22T13:00:02.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.412+0000 7f1c3a800640 1 -- 172.21.15.6:0/1091336463 learned_addr learned my addr 172.21.15.6:0/1091336463 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:02.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.416+0000 7f1c3a800640 1 -- 172.21.15.6:0/1091336463 >> v2:172.21.15.6:3300/0 conn(0x7f1c3c0c7ee0 msgr2=0x7f1c3c15ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.416+0000 7f1c3a800640 1 --2- 172.21.15.6:0/1091336463 >> v2:172.21.15.6:3300/0 conn(0x7f1c3c0c7ee0 0x7f1c3c15ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.416+0000 7f1c3a800640 1 -- 172.21.15.6:0/1091336463 >> v1:172.21.15.6:6789/0 conn(0x7f1c3c154240 legacy=0x7f1c3c0c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.416+0000 7f1c3a800640 1 -- 172.21.15.6:0/1091336463 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0c6e00 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.419+0000 7f1c3a800640 1 --2- 172.21.15.6:0/1091336463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c0c68c0 0x7f1c3c06b080 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009e50 tx=0x7f1c3402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9f9040caf109eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:02.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.422+0000 7f1c38a00640 1 -- 172.21.15.6:0/1091336463 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3402cb90 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.424+0000 7f1c38a00640 1 -- 172.21.15.6:0/1091336463 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3402cd30 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.424+0000 7f1c38a00640 1 -- 172.21.15.6:0/1091336463 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c34034610 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.425+0000 7f1c40a00640 1 -- 172.21.15.6:0/1091336463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c0c68c0 msgr2=0x7f1c3c06b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:02.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.425+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1091336463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c0c68c0 0x7f1c3c06b080 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009e50 tx=0x7f1c3402bff0 comp rx=0 tx=0).stop 2024-03-22T13:00:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.425+0000 7f1c40a00640 1 -- 172.21.15.6:0/1091336463 shutdown_connections 2024-03-22T13:00:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.425+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1091336463 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c0c68c0 0x7f1c3c06b080 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.425+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1091336463 >> v2:172.21.15.6:3300/0 conn(0x7f1c3c0c7ee0 0x7f1c3c15ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.425+0000 7f1c40a00640 1 -- 172.21.15.6:0/1091336463 >> 172.21.15.6:0/1091336463 conn(0x7f1c3c0cb930 msgr2=0x7f1c3c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.425+0000 7f1c40a00640 1 -- 172.21.15.6:0/1091336463 shutdown_connections 2024-03-22T13:00:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.426+0000 7f1c40a00640 1 -- 172.21.15.6:0/1091336463 wait complete. 2024-03-22T13:00:02.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.427+0000 7f1c40a00640 1 Processor -- start 2024-03-22T13:00:02.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.428+0000 7f1c40a00640 1 -- start start 2024-03-22T13:00:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c40a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c3c0c68c0 0x7f1c3c15bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c40a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c3c0c7ee0 0x7f1c3c147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c40a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c154240 0x7f1c3c1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c40a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c15dae0 con 0x7f1c3c0c7ee0 2024-03-22T13:00:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c40a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c15d800 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c40a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c15ddc0 con 0x7f1c3c154240 2024-03-22T13:00:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c39e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c3c0c68c0 0x7f1c3c15bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.429+0000 7f1c3a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c154240 0x7f1c3c1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.430+0000 7f1c3a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c154240 0x7f1c3c1efe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35070/0 (socket says 172.21.15.6:35070) 2024-03-22T13:00:02.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.430+0000 7f1c3a800640 1 -- 172.21.15.6:0/1470800393 learned_addr learned my addr 172.21.15.6:0/1470800393 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:02.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.433+0000 7f1c39400640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c3c0c7ee0 0x7f1c3c147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:02.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.435+0000 7f1c39e00640 1 -- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c154240 msgr2=0x7f1c3c1efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:02.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.435+0000 7f1c39e00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c154240 0x7f1c3c1efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.435+0000 7f1c39e00640 1 -- 172.21.15.6:0/1470800393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c3c0c7ee0 msgr2=0x7f1c3c147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.435+0000 7f1c39e00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c3c0c7ee0 0x7f1c3c147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:02.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.435+0000 7f1c39e00640 1 -- 172.21.15.6:0/1470800393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c15c2e0 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.439+0000 7f1c39e00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c3c0c68c0 0x7f1c3c15bba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00c3f0 tx=0x7f1c2c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:02.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.443+0000 7f1c32000640 1 -- 172.21.15.6:0/1470800393 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c0052d0 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.443+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3c15c5c0 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.443+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c14dd60 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.444+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3c0756c0 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.706+0000 7f1c32000640 1 -- 172.21.15.6:0/1470800393 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c005470 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.706+0000 7f1c32000640 1 -- 172.21.15.6:0/1470800393 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c020060 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.709+0000 7f1c32000640 1 -- 172.21.15.6:0/1470800393 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1c2c0072a0 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.711+0000 7f1c32000640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c1c06bbd0 0x7f1c1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:02.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.712+0000 7f1c39400640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c1c06bbd0 0x7f1c1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:02.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.713+0000 7f1c39400640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c1c06bbd0 0x7f1c1c06e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0ca9f0 tx=0x7f1c28009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:02.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.727+0000 7f1c32000640 1 -- 172.21.15.6:0/1470800393 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1c2c056850 con 0x7f1c3c0c68c0 2024-03-22T13:00:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:02.746+0000 7f1c32000640 1 -- 172.21.15.6:0/1470800393 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1c2c0587b0 con 0x7f1c3c0c68c0 2024-03-22T13:00:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.021+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c3c0c7ab0 con 0x7f1c3c0c68c0 2024-03-22T13:00:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.043+0000 7f1c32000640 1 -- 172.21.15.6:0/1470800393 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f1c2c05c850 con 0x7f1c3c0c68c0 2024-03-22T13:00:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:03.043 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c1c06bbd0 msgr2=0x7f1c1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c1c06bbd0 0x7f1c1c06e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0ca9f0 tx=0x7f1c28009190 comp rx=0 tx=0).stop 2024-03-22T13:00:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c3c0c68c0 msgr2=0x7f1c3c15bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c3c0c68c0 0x7f1c3c15bba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00c3f0 tx=0x7f1c2c00c8b0 comp rx=0 tx=0).stop 2024-03-22T13:00:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 shutdown_connections 2024-03-22T13:00:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3c154240 0x7f1c3c1efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c3c0c7ee0 0x7f1c3c147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c1c06bbd0 0x7f1c1c06e090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.045+0000 7f1c40a00640 1 --2- 172.21.15.6:0/1470800393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c3c0c68c0 0x7f1c3c15bba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.046+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 >> 172.21.15.6:0/1470800393 conn(0x7f1c3c0cb930 msgr2=0x7f1c3c1534d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.046+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 shutdown_connections 2024-03-22T13:00:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:03.046+0000 7f1c40a00640 1 -- 172.21.15.6:0/1470800393 wait complete. 2024-03-22T13:00:03.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:03.670 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:04.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.866+0000 7f6becc00640 1 Processor -- start 2024-03-22T13:00:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.866+0000 7f6becc00640 1 -- start start 2024-03-22T13:00:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.868+0000 7f6becc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80ccad0 0x7f6be806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.868+0000 7f6becc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be806c7a0 con 0x7f6be80ca520 2024-03-22T13:00:04.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.868+0000 7f6becc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be806ca80 con 0x7f6be815b0e0 2024-03-22T13:00:04.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.868+0000 7f6becc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80cd000 con 0x7f6be80ccad0 2024-03-22T13:00:04.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.868+0000 7f6be5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6be80ca520 0x7f6be80cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46372/0 (socket says 172.21.15.6:46372) 2024-03-22T13:00:04.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.869+0000 7f6be5e00640 1 -- 172.21.15.6:0/2359853161 learned_addr learned my addr 172.21.15.6:0/2359853161 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.872+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696808993 0 0) 0x7f6be806c7a0 con 0x7f6be80ca520 2024-03-22T13:00:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.872+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc003520 con 0x7f6be80ca520 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198896336 0 0) 0x7f6bcc003520 con 0x7f6be80ca520 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80ccad0 msgr2=0x7f6be806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80ccad0 0x7f6be806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 >> v1:172.21.15.92:6789/0 conn(0x7f6be815b0e0 legacy=0x7f6be8164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be806cd60 con 0x7f6be80ca520 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f6bd8001670 con 0x7f6be80ca520 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 >> v1:172.21.15.6:6789/0 conn(0x7f6be80ca520 legacy=0x7f6be80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6bcc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80ccad0 0x7f6bcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bcc004e60 0x7f6bcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80cd000 con 0x7f6bcc0039e0 2024-03-22T13:00:04.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc005510 con 0x7f6bcc004e60 2024-03-22T13:00:04.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.879+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc00bf00 con 0x7f6be80ccad0 2024-03-22T13:00:04.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.884+0000 7f6be5e00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6bcc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:04.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.888+0000 7f6be5e00640 1 -- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80ccad0 msgr2=0x7f6bcc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:04.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.888+0000 7f6be5e00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80ccad0 0x7f6bcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:04.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.888+0000 7f6be5e00640 1 -- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bcc004e60 msgr2=0x7f6bcc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:04.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.888+0000 7f6be5e00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bcc004e60 0x7f6bcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:04.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.888+0000 7f6be5e00640 1 -- 172.21.15.6:0/2359853161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc003520 con 0x7f6bcc0039e0 2024-03-22T13:00:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.891+0000 7f6be5e00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6bcc003de0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c160 tx=0x7f6bcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e77cb561baf7201 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:04.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.895+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8007c80 con 0x7f6bcc0039e0 2024-03-22T13:00:04.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.896+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8030070 con 0x7f6bcc0039e0 2024-03-22T13:00:04.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.896+0000 7f6be4a00640 1 -- 172.21.15.6:0/2359853161 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8005db0 con 0x7f6bcc0039e0 2024-03-22T13:00:04.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.897+0000 7f6becc00640 1 -- 172.21.15.6:0/2359853161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 msgr2=0x7f6bcc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.897+0000 7f6becc00640 1 --2- 172.21.15.6:0/2359853161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6bcc003de0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c160 tx=0x7f6bcc005830 comp rx=0 tx=0).stop 2024-03-22T13:00:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.897+0000 7f6be6800640 1 -- 172.21.15.6:0/2359853161 reap_dead start 2024-03-22T13:00:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.897+0000 7f6becc00640 1 -- 172.21.15.6:0/2359853161 shutdown_connections 2024-03-22T13:00:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.897+0000 7f6becc00640 1 -- 172.21.15.6:0/2359853161 >> 172.21.15.6:0/2359853161 conn(0x7f6be80b16b0 msgr2=0x7f6be80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.897+0000 7f6becc00640 1 -- 172.21.15.6:0/2359853161 shutdown_connections 2024-03-22T13:00:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.897+0000 7f6becc00640 1 -- 172.21.15.6:0/2359853161 wait complete. 2024-03-22T13:00:04.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.899+0000 7f6becc00640 1 Processor -- start 2024-03-22T13:00:04.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.899+0000 7f6becc00640 1 -- start start 2024-03-22T13:00:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.900+0000 7f6becc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6be81f0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.900+0000 7f6becc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6be80b55f0 0x7f6be815c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.900+0000 7f6becc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80b2b10 0x7f6be80b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.900+0000 7f6becc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8154c20 con 0x7f6bcc0039e0 2024-03-22T13:00:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.900+0000 7f6becc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be806d320 con 0x7f6be80b55f0 2024-03-22T13:00:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.900+0000 7f6becc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be806d840 con 0x7f6be80b2b10 2024-03-22T13:00:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.901+0000 7f6be5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6be81f0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.901+0000 7f6be5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6be81f0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34074/0 (socket says 172.21.15.6:34074) 2024-03-22T13:00:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.901+0000 7f6be5e00640 1 -- 172.21.15.6:0/3733860059 learned_addr learned my addr 172.21.15.6:0/3733860059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.905+0000 7f6be5e00640 1 -- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80b2b10 msgr2=0x7f6be80b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.905+0000 7f6be5e00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80b2b10 0x7f6be80b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.905+0000 7f6be5e00640 1 -- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6be80b55f0 msgr2=0x7f6be815c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.905+0000 7f6be5e00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6be80b55f0 0x7f6be815c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.905+0000 7f6be5e00640 1 -- 172.21.15.6:0/3733860059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be80b34d0 con 0x7f6bcc0039e0 2024-03-22T13:00:04.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.908+0000 7f6be5e00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6be81f0d20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c250 tx=0x7f6bd8001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:04.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.914+0000 7f6bde000640 1 -- 172.21.15.6:0/3733860059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8007c60 con 0x7f6bcc0039e0 2024-03-22T13:00:04.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.914+0000 7f6bde000640 1 -- 172.21.15.6:0/3733860059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd80049f0 con 0x7f6bcc0039e0 2024-03-22T13:00:04.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.914+0000 7f6bde000640 1 -- 172.21.15.6:0/3733860059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8004da0 con 0x7f6bcc0039e0 2024-03-22T13:00:04.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.914+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be80c6840 con 0x7f6bcc0039e0 2024-03-22T13:00:04.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.914+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be81f3d80 con 0x7f6bcc0039e0 2024-03-22T13:00:04.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.915+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be80cb080 con 0x7f6bcc0039e0 2024-03-22T13:00:04.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.949+0000 7f6bde000640 1 -- 172.21.15.6:0/3733860059 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6bd8007070 con 0x7f6bcc0039e0 2024-03-22T13:00:04.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.951+0000 7f6bde000640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6bc806bbd0 0x7f6bc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:04.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.951+0000 7f6be5400640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6bc806bbd0 0x7f6bc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:04.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.952+0000 7f6be5400640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6bc806bbd0 0x7f6bc806e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6be815c190 tx=0x7f6bd4006d10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:04.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:04.955+0000 7f6bde000640 1 -- 172.21.15.6:0/3733860059 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6bd80789c0 con 0x7f6bcc0039e0 2024-03-22T13:00:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.316+0000 7f6bde000640 1 -- 172.21.15.6:0/3733860059 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6be80b34d0 con 0x7f6bcc0039e0 2024-03-22T13:00:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.571+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be80b5b20 con 0x7f6bcc0039e0 2024-03-22T13:00:05.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.595+0000 7f6bde000640 1 -- 172.21.15.6:0/3733860059 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f6bd807b020 con 0x7f6bcc0039e0 2024-03-22T13:00:05.595 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:05.595 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:05.595 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.597+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6bc806bbd0 msgr2=0x7f6bc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.597+0000 7f6becc00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6bc806bbd0 0x7f6bc806e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6be815c190 tx=0x7f6bd4006d10 comp rx=0 tx=0).stop 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.597+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 msgr2=0x7f6be81f0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.597+0000 7f6becc00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6be81f0d20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c250 tx=0x7f6bd8001970 comp rx=0 tx=0).stop 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 shutdown_connections 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6be80b2b10 0x7f6be80b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6bc806bbd0 0x7f6bc806e090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6be80b55f0 0x7f6be815c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 --2- 172.21.15.6:0/3733860059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bcc0039e0 0x7f6be81f0d20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:05.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 >> 172.21.15.6:0/3733860059 conn(0x7f6be80b16b0 msgr2=0x7f6be80cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:05.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 shutdown_connections 2024-03-22T13:00:05.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:05.598+0000 7f6becc00640 1 -- 172.21.15.6:0/3733860059 wait complete. 2024-03-22T13:00:06.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:06.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:07.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:07.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.418+0000 7fac9b000640 1 Processor -- start 2024-03-22T13:00:07.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.419+0000 7fac9b000640 1 -- start start 2024-03-22T13:00:07.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.420+0000 7fac9b000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac9415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:07.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.420+0000 7fac9b000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac941590b0 con 0x7fac9415bc80 2024-03-22T13:00:07.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.420+0000 7fac9b000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940721f0 con 0x7fac9406f8c0 2024-03-22T13:00:07.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.420+0000 7fac9b000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac940724d0 con 0x7fac94152400 2024-03-22T13:00:07.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.421+0000 7fac93e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac9415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:07.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.421+0000 7fac93400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac9415bc80 0x7fac940716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46378/0 (socket says 172.21.15.6:46378) 2024-03-22T13:00:07.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.421+0000 7fac93400640 1 -- 172.21.15.6:0/1226366564 learned_addr learned my addr 172.21.15.6:0/1226366564 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:07.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.423+0000 7fac93e00640 1 -- 172.21.15.6:0/1226366564 >> v1:172.21.15.92:6789/0 conn(0x7fac9406f8c0 legacy=0x7fac940c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:07.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.423+0000 7fac93e00640 1 -- 172.21.15.6:0/1226366564 >> v1:172.21.15.6:6789/0 conn(0x7fac9415bc80 legacy=0x7fac940716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:07.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.423+0000 7fac93e00640 1 -- 172.21.15.6:0/1226366564 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac941582f0 con 0x7fac94152400 2024-03-22T13:00:07.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.426+0000 7fac93e00640 1 --2- 172.21.15.6:0/1226366564 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac9415b740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac94072650 tx=0x7fac8402b9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc9b96896311608 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.431+0000 7fac92a00640 1 -- 172.21.15.6:0/1226366564 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84034070 con 0x7fac94152400 2024-03-22T13:00:07.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.431+0000 7fac92a00640 1 -- 172.21.15.6:0/1226366564 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84030870 con 0x7fac94152400 2024-03-22T13:00:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.431+0000 7fac92a00640 1 -- 172.21.15.6:0/1226366564 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84030c40 con 0x7fac94152400 2024-03-22T13:00:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.431+0000 7fac9b000640 1 -- 172.21.15.6:0/1226366564 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 msgr2=0x7fac9415b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.431+0000 7fac9b000640 1 --2- 172.21.15.6:0/1226366564 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac9415b740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac94072650 tx=0x7fac8402b9b0 comp rx=0 tx=0).stop 2024-03-22T13:00:07.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.432+0000 7fac9b000640 1 -- 172.21.15.6:0/1226366564 shutdown_connections 2024-03-22T13:00:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.432+0000 7fac9b000640 1 --2- 172.21.15.6:0/1226366564 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac9415b740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.432+0000 7fac9b000640 1 -- 172.21.15.6:0/1226366564 >> 172.21.15.6:0/1226366564 conn(0x7fac940cb930 msgr2=0x7fac940cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.432+0000 7fac9b000640 1 -- 172.21.15.6:0/1226366564 shutdown_connections 2024-03-22T13:00:07.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.432+0000 7fac9b000640 1 -- 172.21.15.6:0/1226366564 wait complete. 2024-03-22T13:00:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.434+0000 7fac9b000640 1 Processor -- start 2024-03-22T13:00:07.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.434+0000 7fac9b000640 1 -- start start 2024-03-22T13:00:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac9b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac9406f8c0 0x7fac94157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:07.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac9b000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac94156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac9b000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac9415bc80 0x7fac94153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac9b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94158b90 con 0x7fac9406f8c0 2024-03-22T13:00:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac9b000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac941588b0 con 0x7fac9415bc80 2024-03-22T13:00:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac9b000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac941527e0 con 0x7fac94152400 2024-03-22T13:00:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac93400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac94156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.435+0000 7fac93e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac9406f8c0 0x7fac94157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.436+0000 7fac99000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac9415bc80 0x7fac94153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.436+0000 7fac93400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac94156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:48902/0 (socket says 172.21.15.6:48902) 2024-03-22T13:00:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.436+0000 7fac93400640 1 -- 172.21.15.6:0/2548641990 learned_addr learned my addr 172.21.15.6:0/2548641990 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:07.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.439+0000 7fac93400640 1 -- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac9415bc80 msgr2=0x7fac94153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:07.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.439+0000 7fac93400640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac9415bc80 0x7fac94153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:07.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.439+0000 7fac93400640 1 -- 172.21.15.6:0/2548641990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac9406f8c0 msgr2=0x7fac94157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:07.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.439+0000 7fac93400640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac9406f8c0 0x7fac94157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:07.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.439+0000 7fac93400640 1 -- 172.21.15.6:0/2548641990 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94157e60 con 0x7fac94152400 2024-03-22T13:00:07.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.442+0000 7fac93400640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac94156670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac8000c3c0 tx=0x7fac8000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:07.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.446+0000 7fac8be00640 1 -- 172.21.15.6:0/2548641990 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac80017070 con 0x7fac94152400 2024-03-22T13:00:07.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.446+0000 7fac8be00640 1 -- 172.21.15.6:0/2548641990 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac80005ce0 con 0x7fac94152400 2024-03-22T13:00:07.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.446+0000 7fac8be00640 1 -- 172.21.15.6:0/2548641990 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac800072d0 con 0x7fac94152400 2024-03-22T13:00:07.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.446+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac941580d0 con 0x7fac94152400 2024-03-22T13:00:07.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.446+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac941f3de0 con 0x7fac94152400 2024-03-22T13:00:07.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.447+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac940756c0 con 0x7fac94152400 2024-03-22T13:00:07.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.707+0000 7fac8be00640 1 -- 172.21.15.6:0/2548641990 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac8001a070 con 0x7fac94152400 2024-03-22T13:00:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.709+0000 7fac8be00640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac7406bc50 0x7fac7406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.709+0000 7fac8be00640 1 -- 172.21.15.6:0/2548641990 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fac8008a5e0 con 0x7fac94152400 2024-03-22T13:00:07.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.710+0000 7fac93e00640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac7406bc50 0x7fac7406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:07.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.711+0000 7fac93e00640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac7406bc50 0x7fac7406e110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fac84036000 tx=0x7fac8403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:07.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.719+0000 7fac8be00640 1 -- 172.21.15.6:0/2548641990 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac80058640 con 0x7fac94152400 2024-03-22T13:00:07.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:07.971+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac9406f5b0 con 0x7fac94152400 2024-03-22T13:00:08.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.147+0000 7fac8be00640 1 -- 172.21.15.6:0/2548641990 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fac8005c6e0 con 0x7fac94152400 2024-03-22T13:00:08.147 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:08.148 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:08.148 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:08.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.150+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac7406bc50 msgr2=0x7fac7406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.150+0000 7fac9b000640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac7406bc50 0x7fac7406e110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fac84036000 tx=0x7fac8403b040 comp rx=0 tx=0).stop 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.150+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 msgr2=0x7fac94156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.150+0000 7fac9b000640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac94156670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fac8000c3c0 tx=0x7fac8000c880 comp rx=0 tx=0).stop 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 shutdown_connections 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac7406bc50 0x7fac7406e110 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac9415bc80 0x7fac94153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac94152400 0x7fac94156670 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 --2- 172.21.15.6:0/2548641990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac9406f8c0 0x7fac94157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 >> 172.21.15.6:0/2548641990 conn(0x7fac940cb930 msgr2=0x7fac94151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 shutdown_connections 2024-03-22T13:00:08.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:08.151+0000 7fac9b000640 1 -- 172.21.15.6:0/2548641990 wait complete. 2024-03-22T13:00:08.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:08.771 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:09.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.964+0000 7f312be00640 1 Processor -- start 2024-03-22T13:00:09.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.964+0000 7f312be00640 1 -- start start 2024-03-22T13:00:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.965+0000 7f312be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0ce8a0 0x7f312c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.965+0000 7f312be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f312c15ae30 0x7f312c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.965+0000 7f312be00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312c15b260 con 0x7f312c0cc380 2024-03-22T13:00:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.965+0000 7f312be00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312c0cee00 con 0x7f312c15ae30 2024-03-22T13:00:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.965+0000 7f312be00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c0cf0c0 con 0x7f312c0ce8a0 2024-03-22T13:00:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.966+0000 7f312b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f312c15ae30 0x7f312c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.966+0000 7f312a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0ce8a0 0x7f312c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.966+0000 7f312b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f312c15ae30 0x7f312c0cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47124/0 (socket says 172.21.15.6:47124) 2024-03-22T13:00:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.966+0000 7f312b400640 1 -- 172.21.15.6:0/3143545352 learned_addr learned my addr 172.21.15.6:0/3143545352 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:09.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.970+0000 7f312b400640 1 -- 172.21.15.6:0/3143545352 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0ce8a0 msgr2=0x7f312c06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:09.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.970+0000 7f312b400640 1 --2- 172.21.15.6:0/3143545352 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0ce8a0 0x7f312c06c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:09.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.970+0000 7f312b400640 1 -- 172.21.15.6:0/3143545352 >> v1:172.21.15.92:6789/0 conn(0x7f312c0cc380 legacy=0x7f312c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:09.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.970+0000 7f312b400640 1 -- 172.21.15.6:0/3143545352 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c0cf3a0 con 0x7f312c15ae30 2024-03-22T13:00:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.976+0000 7f312b400640 1 --2- 172.21.15.6:0/3143545352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c0cc760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3124009870 tx=0x7f312402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee06c82a4688352 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.979+0000 7f3129600640 1 -- 172.21.15.6:0/3143545352 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124034070 con 0x7f312c15ae30 2024-03-22T13:00:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.981+0000 7f3129600640 1 -- 172.21.15.6:0/3143545352 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3124030840 con 0x7f312c15ae30 2024-03-22T13:00:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.981+0000 7f3129600640 1 -- 172.21.15.6:0/3143545352 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124030c10 con 0x7f312c15ae30 2024-03-22T13:00:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 -- 172.21.15.6:0/3143545352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 msgr2=0x7f312c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 --2- 172.21.15.6:0/3143545352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c0cc760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3124009870 tx=0x7f312402b980 comp rx=0 tx=0).stop 2024-03-22T13:00:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 -- 172.21.15.6:0/3143545352 shutdown_connections 2024-03-22T13:00:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 --2- 172.21.15.6:0/3143545352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c0cc760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 --2- 172.21.15.6:0/3143545352 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0ce8a0 0x7f312c06c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 -- 172.21.15.6:0/3143545352 >> 172.21.15.6:0/3143545352 conn(0x7f312c0b1480 msgr2=0x7f312c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:09.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 -- 172.21.15.6:0/3143545352 shutdown_connections 2024-03-22T13:00:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.982+0000 7f312be00640 1 -- 172.21.15.6:0/3143545352 wait complete. 2024-03-22T13:00:09.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.984+0000 7f312be00640 1 Processor -- start 2024-03-22T13:00:09.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.984+0000 7f312be00640 1 -- start start 2024-03-22T13:00:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.985+0000 7f312be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0cc380 0x7f312c1e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.985+0000 7f312be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0ce8a0 0x7f312c1e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.985+0000 7f312be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c1e3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.985+0000 7f312be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15a910 con 0x7f312c0ce8a0 2024-03-22T13:00:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.985+0000 7f312be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c15a630 con 0x7f312c15ae30 2024-03-22T13:00:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.985+0000 7f312be00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c1549d0 con 0x7f312c0cc380 2024-03-22T13:00:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.986+0000 7f312b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c1e3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.986+0000 7f312aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0cc380 0x7f312c1e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.986+0000 7f312a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0ce8a0 0x7f312c1e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.986+0000 7f312b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c1e3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47140/0 (socket says 172.21.15.6:47140) 2024-03-22T13:00:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.986+0000 7f312b400640 1 -- 172.21.15.6:0/2718377515 learned_addr learned my addr 172.21.15.6:0/2718377515 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.990+0000 7f312a000640 1 -- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0cc380 msgr2=0x7f312c1e5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.990+0000 7f312a000640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0cc380 0x7f312c1e5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.990+0000 7f312a000640 1 -- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 msgr2=0x7f312c1e3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.990+0000 7f312a000640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c1e3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.990+0000 7f312a000640 1 -- 172.21.15.6:0/2718377515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c1436f0 con 0x7f312c0ce8a0 2024-03-22T13:00:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:09.993+0000 7f312a000640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0ce8a0 0x7f312c1e5ff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f311400e850 tx=0x7f311400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:10.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.003+0000 7f3122a00640 1 -- 172.21.15.6:0/2718377515 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114017070 con 0x7f312c0ce8a0 2024-03-22T13:00:10.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.003+0000 7f3122a00640 1 -- 172.21.15.6:0/2718377515 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311400c940 con 0x7f312c0ce8a0 2024-03-22T13:00:10.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.003+0000 7f3122a00640 1 -- 172.21.15.6:0/2718377515 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311400cd10 con 0x7f312c0ce8a0 2024-03-22T13:00:10.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.003+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c15bef0 con 0x7f312c0ce8a0 2024-03-22T13:00:10.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.004+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c15c4b0 con 0x7f312c0ce8a0 2024-03-22T13:00:10.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.005+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c0684e0 con 0x7f312c0ce8a0 2024-03-22T13:00:10.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.431+0000 7f3122a00640 1 -- 172.21.15.6:0/2718377515 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f311401a070 con 0x7f312c0ce8a0 2024-03-22T13:00:10.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.433+0000 7f3122a00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f310806bc50 0x7f310806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:10.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.433+0000 7f312aa00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f310806bc50 0x7f310806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:10.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.434+0000 7f312aa00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f310806bc50 0x7f310806e110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f311c001e70 tx=0x7f311c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:10.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.469+0000 7f3122a00640 1 -- 172.21.15.6:0/2718377515 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f31140575d0 con 0x7f312c0ce8a0 2024-03-22T13:00:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.469+0000 7f3122a00640 1 -- 172.21.15.6:0/2718377515 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f312c0684e0 con 0x7f312c0ce8a0 2024-03-22T13:00:10.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.710+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f312c075650 con 0x7f312c0ce8a0 2024-03-22T13:00:10.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.732+0000 7f3122a00640 1 -- 172.21.15.6:0/2718377515 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f312c1436f0 con 0x7f312c0ce8a0 2024-03-22T13:00:10.733 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:10.733 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:10.733 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:10.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f310806bc50 msgr2=0x7f310806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:10.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f310806bc50 0x7f310806e110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f311c001e70 tx=0x7f311c009040 comp rx=0 tx=0).stop 2024-03-22T13:00:10.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0ce8a0 msgr2=0x7f312c1e5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:10.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0ce8a0 0x7f312c1e5ff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f311400e850 tx=0x7f311400ed10 comp rx=0 tx=0).stop 2024-03-22T13:00:10.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 shutdown_connections 2024-03-22T13:00:10.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f310806bc50 0x7f310806e110 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:10.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f312c15ae30 0x7f312c1e3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:10.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f312c0ce8a0 0x7f312c1e5ff0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:10.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 --2- 172.21.15.6:0/2718377515 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f312c0cc380 0x7f312c1e5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:10.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 >> 172.21.15.6:0/2718377515 conn(0x7f312c0b1480 msgr2=0x7f312c1496b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:10.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 shutdown_connections 2024-03-22T13:00:10.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:10.736+0000 7f312be00640 1 -- 172.21.15.6:0/2718377515 wait complete. 2024-03-22T13:00:11.367 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:11.367 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:12.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.565+0000 7fe989200640 1 Processor -- start 2024-03-22T13:00:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.565+0000 7fe989200640 1 -- start start 2024-03-22T13:00:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.566+0000 7fe989200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9840c5c20 0x7fe984156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.567+0000 7fe989200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840c80f0 con 0x7fe984150f50 2024-03-22T13:00:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.567+0000 7fe989200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840c3bf0 con 0x7fe984157410 2024-03-22T13:00:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.567+0000 7fe989200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840c3760 con 0x7fe9840c5c20 2024-03-22T13:00:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.567+0000 7fe981e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe984150f50 0x7fe9840c79e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46384/0 (socket says 172.21.15.6:46384) 2024-03-22T13:00:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.567+0000 7fe981e00640 1 -- 172.21.15.6:0/3752236653 learned_addr learned my addr 172.21.15.6:0/3752236653 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.567+0000 7fe981400640 1 --2- 172.21.15.6:0/3752236653 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9840c5c20 0x7fe984156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:12.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.570+0000 7fe980a00640 1 -- 172.21.15.6:0/3752236653 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493470414 0 0) 0x7fe974001110 con 0x7fe984150f50 2024-03-22T13:00:12.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.571+0000 7fe980a00640 1 -- 172.21.15.6:0/3752236653 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968005730 con 0x7fe984150f50 2024-03-22T13:00:12.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.571+0000 7fe981400640 1 -- 172.21.15.6:0/3752236653 >> v1:172.21.15.92:6789/0 conn(0x7fe984157410 legacy=0x7fe98406b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:12.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.571+0000 7fe981400640 1 -- 172.21.15.6:0/3752236653 >> v1:172.21.15.6:6789/0 conn(0x7fe984150f50 legacy=0x7fe9840c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:12.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.571+0000 7fe981400640 1 -- 172.21.15.6:0/3752236653 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe984071e70 con 0x7fe9840c5c20 2024-03-22T13:00:12.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.573+0000 7fe981400640 1 --2- 172.21.15.6:0/3752236653 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9840c5c20 0x7fe984156ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9700097d0 tx=0x7fe97002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e13d22fd649b261 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.576+0000 7fe980a00640 1 -- 172.21.15.6:0/3752236653 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe970034070 con 0x7fe9840c5c20 2024-03-22T13:00:12.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.577+0000 7fe980a00640 1 -- 172.21.15.6:0/3752236653 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe970030880 con 0x7fe9840c5c20 2024-03-22T13:00:12.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.578+0000 7fe989200640 1 -- 172.21.15.6:0/3752236653 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9840c5c20 msgr2=0x7fe984156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:12.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.578+0000 7fe989200640 1 --2- 172.21.15.6:0/3752236653 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9840c5c20 0x7fe984156ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe9700097d0 tx=0x7fe97002b9c0 comp rx=0 tx=0).stop 2024-03-22T13:00:12.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.578+0000 7fe989200640 1 -- 172.21.15.6:0/3752236653 shutdown_connections 2024-03-22T13:00:12.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.578+0000 7fe989200640 1 --2- 172.21.15.6:0/3752236653 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9840c5c20 0x7fe984156ed0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:12.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.578+0000 7fe989200640 1 -- 172.21.15.6:0/3752236653 >> 172.21.15.6:0/3752236653 conn(0x7fe9840cb930 msgr2=0x7fe9840cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:12.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.578+0000 7fe989200640 1 -- 172.21.15.6:0/3752236653 shutdown_connections 2024-03-22T13:00:12.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.578+0000 7fe989200640 1 -- 172.21.15.6:0/3752236653 wait complete. 2024-03-22T13:00:12.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.580+0000 7fe989200640 1 Processor -- start 2024-03-22T13:00:12.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.580+0000 7fe989200640 1 -- start start 2024-03-22T13:00:12.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.581+0000 7fe989200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9840c5c20 0x7fe98414f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:12.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.581+0000 7fe989200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe984150f50 0x7fe98414f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:12.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.581+0000 7fe989200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe984157410 0x7fe984154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:12.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.581+0000 7fe989200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840c3110 con 0x7fe984150f50 2024-03-22T13:00:12.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.581+0000 7fe989200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840c2e30 con 0x7fe9840c5c20 2024-03-22T13:00:12.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.581+0000 7fe989200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840c33f0 con 0x7fe984157410 2024-03-22T13:00:12.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.582+0000 7fe981400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe984150f50 0x7fe98414f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:12.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.582+0000 7fe982800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe984157410 0x7fe984154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:12.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.582+0000 7fe981400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe984150f50 0x7fe98414f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34114/0 (socket says 172.21.15.6:34114) 2024-03-22T13:00:12.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.582+0000 7fe981400640 1 -- 172.21.15.6:0/1783312405 learned_addr learned my addr 172.21.15.6:0/1783312405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:12.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.587+0000 7fe982800640 1 -- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9840c5c20 msgr2=0x7fe98414f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:12.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.587+0000 7fe982800640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9840c5c20 0x7fe98414f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:12.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.587+0000 7fe982800640 1 -- 172.21.15.6:0/1783312405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe984150f50 msgr2=0x7fe98414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:12.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.587+0000 7fe982800640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe984150f50 0x7fe98414f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:12.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.587+0000 7fe982800640 1 -- 172.21.15.6:0/1783312405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9841e2970 con 0x7fe984157410 2024-03-22T13:00:12.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.591+0000 7fe982800640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe984157410 0x7fe984154fb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c690 tx=0x7fe97c00cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:12.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.596+0000 7fe97a000640 1 -- 172.21.15.6:0/1783312405 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c003b70 con 0x7fe984157410 2024-03-22T13:00:12.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.596+0000 7fe97a000640 1 -- 172.21.15.6:0/1783312405 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c003d10 con 0x7fe984157410 2024-03-22T13:00:12.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.596+0000 7fe97a000640 1 -- 172.21.15.6:0/1783312405 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c005310 con 0x7fe984157410 2024-03-22T13:00:12.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.596+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9841e2c50 con 0x7fe984157410 2024-03-22T13:00:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.597+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98414e690 con 0x7fe984157410 2024-03-22T13:00:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.598+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9840756c0 con 0x7fe984157410 2024-03-22T13:00:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.635+0000 7fe97a000640 1 -- 172.21.15.6:0/1783312405 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe97c0054b0 con 0x7fe984157410 2024-03-22T13:00:12.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.638+0000 7fe97a000640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe96406bc20 0x7fe96406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:12.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.639+0000 7fe981e00640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe96406bc20 0x7fe96406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:12.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.640+0000 7fe981e00640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe96406bc20 0x7fe96406e0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe974000f60 tx=0x7fe974001110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.880+0000 7fe97a000640 1 -- 172.21.15.6:0/1783312405 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe97c0057a0 con 0x7fe984157410 2024-03-22T13:00:12.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:12.897+0000 7fe97a000640 1 -- 172.21.15.6:0/1783312405 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe97c059df0 con 0x7fe984157410 2024-03-22T13:00:13.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.180+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9840c2690 con 0x7fe984157410 2024-03-22T13:00:13.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.201+0000 7fe97a000640 1 -- 172.21.15.6:0/1783312405 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fe97c0596f0 con 0x7fe984157410 2024-03-22T13:00:13.201 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:13.201 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:13.201 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.203+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe96406bc20 msgr2=0x7fe96406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.203+0000 7fe989200640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe96406bc20 0x7fe96406e0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe974000f60 tx=0x7fe974001110 comp rx=0 tx=0).stop 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.203+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe984157410 msgr2=0x7fe984154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.203+0000 7fe989200640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe984157410 0x7fe984154fb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00c690 tx=0x7fe97c00cb50 comp rx=0 tx=0).stop 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 shutdown_connections 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe984157410 0x7fe984154fb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe984150f50 0x7fe98414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe96406bc20 0x7fe96406e0e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 --2- 172.21.15.6:0/1783312405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9840c5c20 0x7fe98414f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 >> 172.21.15.6:0/1783312405 conn(0x7fe9840cb930 msgr2=0x7fe9840730a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 shutdown_connections 2024-03-22T13:00:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:13.204+0000 7fe989200640 1 -- 172.21.15.6:0/1783312405 wait complete. 2024-03-22T13:00:13.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:13.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:14.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:15.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.018+0000 7f9e55a00640 1 Processor -- start 2024-03-22T13:00:15.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.018+0000 7f9e55a00640 1 -- start start 2024-03-22T13:00:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.020+0000 7f9e55a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e50152400 0x7f9e5015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.020+0000 7f9e55a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9e5006f8c0 0x7f9e500c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.020+0000 7f9e55a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e501590b0 con 0x7f9e5015bc80 2024-03-22T13:00:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.020+0000 7f9e55a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e500724d0 con 0x7f9e50152400 2024-03-22T13:00:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.020+0000 7f9e55a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e500721f0 con 0x7f9e5006f8c0 2024-03-22T13:00:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.021+0000 7f9e4f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9e5006f8c0 0x7f9e500c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:15.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.021+0000 7f9e4ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e50152400 0x7f9e5015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.021+0000 7f9e4ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e50152400 0x7f9e5015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38620/0 (socket says 172.21.15.6:38620) 2024-03-22T13:00:15.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.021+0000 7f9e4ea00640 1 -- 172.21.15.6:0/1073899529 learned_addr learned my addr 172.21.15.6:0/1073899529 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:15.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.024+0000 7f9e4ea00640 1 -- 172.21.15.6:0/1073899529 >> v2:172.21.15.92:3300/0 conn(0x7f9e5006f8c0 msgr2=0x7f9e500c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:15.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.024+0000 7f9e4ea00640 1 --2- 172.21.15.6:0/1073899529 >> v2:172.21.15.92:3300/0 conn(0x7f9e5006f8c0 0x7f9e500c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.024+0000 7f9e4ea00640 1 -- 172.21.15.6:0/1073899529 >> v1:172.21.15.92:6789/0 conn(0x7f9e5015bc80 legacy=0x7f9e500716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:15.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.024+0000 7f9e4ea00640 1 -- 172.21.15.6:0/1073899529 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e501582f0 con 0x7f9e50152400 2024-03-22T13:00:15.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.027+0000 7f9e4ea00640 1 --2- 172.21.15.6:0/1073899529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e50152400 0x7f9e5015b740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9e5015c860 tx=0x7f9e4002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=411115a48d04b864 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:15.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.032+0000 7f9e4e000640 1 -- 172.21.15.6:0/1073899529 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4002cbb0 con 0x7f9e50152400 2024-03-22T13:00:15.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.032+0000 7f9e4e000640 1 -- 172.21.15.6:0/1073899529 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4002cd50 con 0x7f9e50152400 2024-03-22T13:00:15.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.032+0000 7f9e4e000640 1 -- 172.21.15.6:0/1073899529 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40034610 con 0x7f9e50152400 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 -- 172.21.15.6:0/1073899529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e50152400 msgr2=0x7f9e5015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 --2- 172.21.15.6:0/1073899529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e50152400 0x7f9e5015b740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9e5015c860 tx=0x7f9e4002c010 comp rx=0 tx=0).stop 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 -- 172.21.15.6:0/1073899529 shutdown_connections 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 --2- 172.21.15.6:0/1073899529 >> v2:172.21.15.92:3300/0 conn(0x7f9e5006f8c0 0x7f9e500c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 --2- 172.21.15.6:0/1073899529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e50152400 0x7f9e5015b740 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 -- 172.21.15.6:0/1073899529 >> 172.21.15.6:0/1073899529 conn(0x7f9e500cb930 msgr2=0x7f9e500cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 -- 172.21.15.6:0/1073899529 shutdown_connections 2024-03-22T13:00:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.033+0000 7f9e55a00640 1 -- 172.21.15.6:0/1073899529 wait complete. 2024-03-22T13:00:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.036+0000 7f9e55a00640 1 Processor -- start 2024-03-22T13:00:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.036+0000 7f9e55a00640 1 -- start start 2024-03-22T13:00:15.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.037+0000 7f9e55a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e5006f8c0 0x7f9e501e23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:15.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.037+0000 7f9e55a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e50152400 0x7f9e50159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:15.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.037+0000 7f9e55a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e5015bc80 0x7f9e500c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:15.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.037+0000 7f9e55a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e50158b90 con 0x7f9e5006f8c0 2024-03-22T13:00:15.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.037+0000 7f9e55a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e501588b0 con 0x7f9e5015bc80 2024-03-22T13:00:15.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.037+0000 7f9e55a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e501527e0 con 0x7f9e50152400 2024-03-22T13:00:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.038+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e5006f8c0 0x7f9e501e23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.038+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e5015bc80 0x7f9e500c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.038+0000 7f9e46000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e50152400 0x7f9e50159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.038+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e5006f8c0 0x7f9e501e23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38628/0 (socket says 172.21.15.6:38628) 2024-03-22T13:00:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.038+0000 7f9e4ea00640 1 -- 172.21.15.6:0/78737325 learned_addr learned my addr 172.21.15.6:0/78737325 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.042+0000 7f9e4ea00640 1 -- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e50152400 msgr2=0x7f9e50159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.042+0000 7f9e4ea00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e50152400 0x7f9e50159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.042+0000 7f9e4ea00640 1 -- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e5015bc80 msgr2=0x7f9e500c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.042+0000 7f9e4ea00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e5015bc80 0x7f9e500c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.042+0000 7f9e4ea00640 1 -- 172.21.15.6:0/78737325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5014ea60 con 0x7f9e5006f8c0 2024-03-22T13:00:15.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.044+0000 7f9e4ea00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e5006f8c0 0x7f9e501e23c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e40000f80 tx=0x7f9e40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.049+0000 7f9e47400640 1 -- 172.21.15.6:0/78737325 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40043020 con 0x7f9e5006f8c0 2024-03-22T13:00:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.049+0000 7f9e47400640 1 -- 172.21.15.6:0/78737325 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4003e450 con 0x7f9e5006f8c0 2024-03-22T13:00:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.049+0000 7f9e47400640 1 -- 172.21.15.6:0/78737325 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4003e820 con 0x7f9e5006f8c0 2024-03-22T13:00:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.049+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5014ed20 con 0x7f9e5006f8c0 2024-03-22T13:00:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.049+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e500c4240 con 0x7f9e5006f8c0 2024-03-22T13:00:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.050+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e500756c0 con 0x7f9e5006f8c0 2024-03-22T13:00:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.343+0000 7f9e47400640 1 -- 172.21.15.6:0/78737325 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e40046020 con 0x7f9e5006f8c0 2024-03-22T13:00:15.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.344+0000 7f9e47400640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9e2c06bc20 0x7f9e2c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:15.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.344+0000 7f9e46000640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9e2c06bc20 0x7f9e2c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.345+0000 7f9e46000640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9e2c06bc20 0x7f9e2c06e0e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9e501582c0 tx=0x7f9e38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:15.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.374+0000 7f9e47400640 1 -- 172.21.15.6:0/78737325 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9e40079c90 con 0x7f9e5006f8c0 2024-03-22T13:00:15.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.375+0000 7f9e47400640 1 -- 172.21.15.6:0/78737325 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9e4004d030 con 0x7f9e5006f8c0 2024-03-22T13:00:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.628+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e500c6400 con 0x7f9e5006f8c0 2024-03-22T13:00:15.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.652+0000 7f9e47400640 1 -- 172.21.15.6:0/78737325 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f9e4007bbf0 con 0x7f9e5006f8c0 2024-03-22T13:00:15.653 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:15.653 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:15.653 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9e2c06bc20 msgr2=0x7f9e2c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9e2c06bc20 0x7f9e2c06e0e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9e501582c0 tx=0x7f9e38009040 comp rx=0 tx=0).stop 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e5006f8c0 msgr2=0x7f9e501e23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e5006f8c0 0x7f9e501e23c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e40000f80 tx=0x7f9e40004a40 comp rx=0 tx=0).stop 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 shutdown_connections 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9e2c06bc20 0x7f9e2c06e0e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e5015bc80 0x7f9e500c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e50152400 0x7f9e50159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 --2- 172.21.15.6:0/78737325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e5006f8c0 0x7f9e501e23c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 >> 172.21.15.6:0/78737325 conn(0x7f9e500cb930 msgr2=0x7f9e500c6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:15.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 shutdown_connections 2024-03-22T13:00:15.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:15.655+0000 7f9e55a00640 1 -- 172.21.15.6:0/78737325 wait complete. 2024-03-22T13:00:16.277 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:16.277 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:17.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.487+0000 7fa36e800640 1 Processor -- start 2024-03-22T13:00:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.487+0000 7fa36e800640 1 -- start start 2024-03-22T13:00:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.488+0000 7fa36e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3680cc380 0x7fa36806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.488+0000 7fa36e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3680ce8a0 0x7fa36815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.488+0000 7fa36e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680cee00 con 0x7fa3680cc760 2024-03-22T13:00:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.488+0000 7fa36e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3680cf0c0 con 0x7fa3680ce8a0 2024-03-22T13:00:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.488+0000 7fa36e800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3680cf3a0 con 0x7fa3680cc380 2024-03-22T13:00:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.489+0000 7fa36d400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3680cc380 0x7fa36806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.489+0000 7fa36de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3680cc760 0x7fa3680cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51014/0 (socket says 172.21.15.6:51014) 2024-03-22T13:00:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.490+0000 7fa36de00640 1 -- 172.21.15.6:0/2009625970 learned_addr learned my addr 172.21.15.6:0/2009625970 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.490+0000 7fa36ca00640 1 --2- 172.21.15.6:0/2009625970 >> v2:172.21.15.6:3300/0 conn(0x7fa3680ce8a0 0x7fa36815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.493+0000 7fa36d400640 1 -- 172.21.15.6:0/2009625970 >> v2:172.21.15.6:3300/0 conn(0x7fa3680ce8a0 msgr2=0x7fa36815ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.493+0000 7fa36d400640 1 --2- 172.21.15.6:0/2009625970 >> v2:172.21.15.6:3300/0 conn(0x7fa3680ce8a0 0x7fa36815ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.493+0000 7fa36d400640 1 -- 172.21.15.6:0/2009625970 >> v1:172.21.15.6:6789/0 conn(0x7fa3680cc760 legacy=0x7fa3680cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.493+0000 7fa36d400640 1 -- 172.21.15.6:0/2009625970 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680cf680 con 0x7fa3680cc380 2024-03-22T13:00:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.496+0000 7fa36d400640 1 --2- 172.21.15.6:0/2009625970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc380 0x7fa36806c230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa36808d0e0 tx=0x7fa35802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d253c2c728e58ac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.501+0000 7fa367e00640 1 -- 172.21.15.6:0/2009625970 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358035020 con 0x7fa3680cc380 2024-03-22T13:00:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.501+0000 7fa367e00640 1 -- 172.21.15.6:0/2009625970 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa358031830 con 0x7fa3680cc380 2024-03-22T13:00:17.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.501+0000 7fa367e00640 1 -- 172.21.15.6:0/2009625970 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358031c00 con 0x7fa3680cc380 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 -- 172.21.15.6:0/2009625970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc380 msgr2=0x7fa36806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 --2- 172.21.15.6:0/2009625970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc380 0x7fa36806c230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa36808d0e0 tx=0x7fa35802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 -- 172.21.15.6:0/2009625970 shutdown_connections 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 --2- 172.21.15.6:0/2009625970 >> v2:172.21.15.6:3300/0 conn(0x7fa3680ce8a0 0x7fa36815ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 --2- 172.21.15.6:0/2009625970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc380 0x7fa36806c230 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 -- 172.21.15.6:0/2009625970 >> 172.21.15.6:0/2009625970 conn(0x7fa3680b1480 msgr2=0x7fa3680b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 -- 172.21.15.6:0/2009625970 shutdown_connections 2024-03-22T13:00:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.502+0000 7fa36e800640 1 -- 172.21.15.6:0/2009625970 wait complete. 2024-03-22T13:00:17.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.504+0000 7fa36e800640 1 Processor -- start 2024-03-22T13:00:17.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.504+0000 7fa36e800640 1 -- start start 2024-03-22T13:00:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.505+0000 7fa36e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3680cc380 0x7fa3680c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.505+0000 7fa36e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc760 0x7fa3681f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.505+0000 7fa36e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680ce8a0 0x7fa3681523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.505+0000 7fa36e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3681549d0 con 0x7fa3680cc380 2024-03-22T13:00:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.505+0000 7fa36e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36815a910 con 0x7fa3680cc760 2024-03-22T13:00:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.505+0000 7fa36e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680ccb70 con 0x7fa3680ce8a0 2024-03-22T13:00:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.506+0000 7fa36d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3680cc380 0x7fa3680c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.506+0000 7fa36ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc760 0x7fa3681f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.506+0000 7fa36de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680ce8a0 0x7fa3681523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.506+0000 7fa36d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3680cc380 0x7fa3680c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38648/0 (socket says 172.21.15.6:38648) 2024-03-22T13:00:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.506+0000 7fa36d400640 1 -- 172.21.15.6:0/470892372 learned_addr learned my addr 172.21.15.6:0/470892372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.510+0000 7fa36d400640 1 -- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680ce8a0 msgr2=0x7fa3681523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.510+0000 7fa36d400640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680ce8a0 0x7fa3681523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.510+0000 7fa36d400640 1 -- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc760 msgr2=0x7fa3681f3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.510+0000 7fa36d400640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc760 0x7fa3681f3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.510+0000 7fa36d400640 1 -- 172.21.15.6:0/470892372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680c7fd0 con 0x7fa3680cc380 2024-03-22T13:00:17.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.513+0000 7fa36d400640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3680cc380 0x7fa3680c6c70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa358009890 tx=0x7fa3580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:17.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.516+0000 7fa365600640 1 -- 172.21.15.6:0/470892372 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358041070 con 0x7fa3680cc380 2024-03-22T13:00:17.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.516+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3681f31e0 con 0x7fa3680cc380 2024-03-22T13:00:17.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.516+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3680c8570 con 0x7fa3680cc380 2024-03-22T13:00:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.517+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa368075650 con 0x7fa3680cc380 2024-03-22T13:00:17.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.518+0000 7fa365600640 1 -- 172.21.15.6:0/470892372 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa358031800 con 0x7fa3680cc380 2024-03-22T13:00:17.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.518+0000 7fa365600640 1 -- 172.21.15.6:0/470892372 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358035040 con 0x7fa3680cc380 2024-03-22T13:00:17.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.804+0000 7fa365600640 1 -- 172.21.15.6:0/470892372 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa358038020 con 0x7fa3680cc380 2024-03-22T13:00:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.806+0000 7fa365600640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa34806bc00 0x7fa34806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:17.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.806+0000 7fa36ca00640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa34806bc00 0x7fa34806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:17.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.807+0000 7fa36ca00640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa34806bc00 0x7fa34806e0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa3680b0e80 tx=0x7fa354009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:17.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.821+0000 7fa365600640 1 -- 172.21.15.6:0/470892372 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa358078c70 con 0x7fa3680cc380 2024-03-22T13:00:17.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:17.823+0000 7fa365600640 1 -- 172.21.15.6:0/470892372 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa35807abd0 con 0x7fa3680cc380 2024-03-22T13:00:18.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.085+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa36814ea50 con 0x7fa3680cc380 2024-03-22T13:00:18.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.108+0000 7fa365600640 1 -- 172.21.15.6:0/470892372 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fa35807ec70 con 0x7fa3680cc380 2024-03-22T13:00:18.109 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:18.109 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:18.109 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:18.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa34806bc00 msgr2=0x7fa34806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:18.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa34806bc00 0x7fa34806e0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa3680b0e80 tx=0x7fa354009190 comp rx=0 tx=0).stop 2024-03-22T13:00:18.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3680cc380 msgr2=0x7fa3680c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:18.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3680cc380 0x7fa3680c6c70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa358009890 tx=0x7fa3580046c0 comp rx=0 tx=0).stop 2024-03-22T13:00:18.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 shutdown_connections 2024-03-22T13:00:18.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680ce8a0 0x7fa3681523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:18.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa34806bc00 0x7fa34806e0c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:18.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3680cc760 0x7fa3681f3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:18.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 --2- 172.21.15.6:0/470892372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3680cc380 0x7fa3680c6c70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:18.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 >> 172.21.15.6:0/470892372 conn(0x7fa3680b1480 msgr2=0x7fa368158ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:18.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 shutdown_connections 2024-03-22T13:00:18.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:18.111+0000 7fa36e800640 1 -- 172.21.15.6:0/470892372 wait complete. 2024-03-22T13:00:18.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:18.733 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:19.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:19.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.929+0000 7f54aee00640 1 Processor -- start 2024-03-22T13:00:19.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.929+0000 7f54aee00640 1 -- start start 2024-03-22T13:00:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.930+0000 7f54aee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a806c230 0x7f54a815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.930+0000 7f54aee00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80cee00 con 0x7f54a80ce8a0 2024-03-22T13:00:19.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.930+0000 7f54aee00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a80cf0c0 con 0x7f54a80cc380 2024-03-22T13:00:19.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.930+0000 7f54aee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a806c660 con 0x7f54a806c230 2024-03-22T13:00:19.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.931+0000 7f54ae400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a806c230 0x7f54a815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:19.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.931+0000 7f54ada00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f54a80cc380 0x7f54a80cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35964/0 (socket says 172.21.15.6:35964) 2024-03-22T13:00:19.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.931+0000 7f54ada00640 1 -- 172.21.15.6:0/1485220381 learned_addr learned my addr 172.21.15.6:0/1485220381 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:19.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.931+0000 7f54ad000640 1 --1- 172.21.15.6:0/1485220381 >> v1:172.21.15.6:6789/0 conn(0x7f54a80ce8a0 0x7f54a8162b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51020/0 (socket says 172.21.15.6:51020) 2024-03-22T13:00:19.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.934+0000 7f54ae400640 1 -- 172.21.15.6:0/1485220381 >> v1:172.21.15.92:6789/0 conn(0x7f54a80cc380 legacy=0x7f54a80cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:19.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.934+0000 7f54ae400640 1 -- 172.21.15.6:0/1485220381 >> v1:172.21.15.6:6789/0 conn(0x7f54a80ce8a0 legacy=0x7f54a8162b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:19.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.934+0000 7f54ae400640 1 -- 172.21.15.6:0/1485220381 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a80cf3a0 con 0x7f54a806c230 2024-03-22T13:00:19.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.936+0000 7f54ae400640 1 --2- 172.21.15.6:0/1485220381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a806c230 0x7f54a815ae30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009dc0 tx=0x7f54a002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78e4051fa08d71b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:19.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.940+0000 7f54a7e00640 1 -- 172.21.15.6:0/1485220381 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0039070 con 0x7f54a806c230 2024-03-22T13:00:19.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.940+0000 7f54a7e00640 1 -- 172.21.15.6:0/1485220381 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a002ccd0 con 0x7f54a806c230 2024-03-22T13:00:19.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.940+0000 7f54a7e00640 1 -- 172.21.15.6:0/1485220381 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0034660 con 0x7f54a806c230 2024-03-22T13:00:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.941+0000 7f54aee00640 1 -- 172.21.15.6:0/1485220381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a806c230 msgr2=0x7f54a815ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.941+0000 7f54aee00640 1 --2- 172.21.15.6:0/1485220381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a806c230 0x7f54a815ae30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009dc0 tx=0x7f54a002bf20 comp rx=0 tx=0).stop 2024-03-22T13:00:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.941+0000 7f54aee00640 1 -- 172.21.15.6:0/1485220381 shutdown_connections 2024-03-22T13:00:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.941+0000 7f54aee00640 1 --2- 172.21.15.6:0/1485220381 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a806c230 0x7f54a815ae30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.941+0000 7f54aee00640 1 -- 172.21.15.6:0/1485220381 >> 172.21.15.6:0/1485220381 conn(0x7f54a80b1480 msgr2=0x7f54a80b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.941+0000 7f54aee00640 1 -- 172.21.15.6:0/1485220381 shutdown_connections 2024-03-22T13:00:19.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.941+0000 7f54aee00640 1 -- 172.21.15.6:0/1485220381 wait complete. 2024-03-22T13:00:19.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.943+0000 7f54aee00640 1 Processor -- start 2024-03-22T13:00:19.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.943+0000 7f54aee00640 1 -- start start 2024-03-22T13:00:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.944+0000 7f54aee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54a806c230 0x7f54a814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.944+0000 7f54aee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a80cc380 0x7f54a81f0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.944+0000 7f54aee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54a80ce8a0 0x7f54a81521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.944+0000 7f54aee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a815a910 con 0x7f54a80ce8a0 2024-03-22T13:00:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.944+0000 7f54aee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a815a630 con 0x7f54a806c230 2024-03-22T13:00:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.944+0000 7f54aee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81549d0 con 0x7f54a80cc380 2024-03-22T13:00:19.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.945+0000 7f54ad000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a80cc380 0x7f54a81f0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:19.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.945+0000 7f54ae400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54a80ce8a0 0x7f54a81521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:19.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.945+0000 7f54ad000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a80cc380 0x7f54a81f0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47728/0 (socket says 172.21.15.6:47728) 2024-03-22T13:00:19.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.945+0000 7f54ad000640 1 -- 172.21.15.6:0/2586593520 learned_addr learned my addr 172.21.15.6:0/2586593520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:19.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.945+0000 7f54ada00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54a806c230 0x7f54a814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.949+0000 7f54ada00640 1 -- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a80cc380 msgr2=0x7f54a81f0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.949+0000 7f54ada00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a80cc380 0x7f54a81f0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.949+0000 7f54ada00640 1 -- 172.21.15.6:0/2586593520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54a80ce8a0 msgr2=0x7f54a81521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.949+0000 7f54ada00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54a80ce8a0 0x7f54a81521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.949+0000 7f54ada00640 1 -- 172.21.15.6:0/2586593520 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a80b5180 con 0x7f54a806c230 2024-03-22T13:00:19.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.952+0000 7f54ada00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54a806c230 0x7f54a814e420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f549c00c470 tx=0x7f549c001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:19.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.954+0000 7f54a5600640 1 -- 172.21.15.6:0/2586593520 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c017070 con 0x7f54a806c230 2024-03-22T13:00:19.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.954+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a80b5460 con 0x7f54a806c230 2024-03-22T13:00:19.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.954+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a80b4010 con 0x7f54a806c230 2024-03-22T13:00:19.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.955+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a8075650 con 0x7f54a806c230 2024-03-22T13:00:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.981+0000 7f54a5600640 1 -- 172.21.15.6:0/2586593520 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c005ce0 con 0x7f54a806c230 2024-03-22T13:00:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.981+0000 7f54a5600640 1 -- 172.21.15.6:0/2586593520 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c021060 con 0x7f54a806c230 2024-03-22T13:00:19.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.983+0000 7f54a5600640 1 -- 172.21.15.6:0/2586593520 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f549c01a070 con 0x7f54a806c230 2024-03-22T13:00:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.984+0000 7f54a5600640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f548806bbd0 0x7f548806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.985+0000 7f54ad000640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f548806bbd0 0x7f548806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:19.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.985+0000 7f54a5600640 1 -- 172.21.15.6:0/2586593520 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f549c056b10 con 0x7f54a806c230 2024-03-22T13:00:19.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:19.986+0000 7f54ad000640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f548806bbd0 0x7f548806e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54a80c2de0 tx=0x7f5494007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:20.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.274+0000 7f54a5600640 1 -- 172.21.15.6:0/2586593520 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f549c058a70 con 0x7f54a806c230 2024-03-22T13:00:20.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.525+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54a80b4970 con 0x7f54a806c230 2024-03-22T13:00:20.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.548+0000 7f54a5600640 1 -- 172.21.15.6:0/2586593520 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f549c05cb10 con 0x7f54a806c230 2024-03-22T13:00:20.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:20.549 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:20.549 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:20.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f548806bbd0 msgr2=0x7f548806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:20.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f548806bbd0 0x7f548806e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54a80c2de0 tx=0x7f5494007f10 comp rx=0 tx=0).stop 2024-03-22T13:00:20.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54a806c230 msgr2=0x7f54a814e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:20.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54a806c230 0x7f54a814e420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f549c00c470 tx=0x7f549c001e70 comp rx=0 tx=0).stop 2024-03-22T13:00:20.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 shutdown_connections 2024-03-22T13:00:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54a80ce8a0 0x7f54a81521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54a80cc380 0x7f54a81f0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f548806bbd0 0x7f548806e090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 --2- 172.21.15.6:0/2586593520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54a806c230 0x7f54a814e420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 >> 172.21.15.6:0/2586593520 conn(0x7f54a80b1480 msgr2=0x7f54a80ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.550+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 shutdown_connections 2024-03-22T13:00:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:20.551+0000 7f54aee00640 1 -- 172.21.15.6:0/2586593520 wait complete. 2024-03-22T13:00:21.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:21.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:22.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.401+0000 7fc7c8000640 1 Processor -- start 2024-03-22T13:00:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.401+0000 7fc7c8000640 1 -- start start 2024-03-22T13:00:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.403+0000 7fc7c8000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7c00cc3b0 0x7fc7c00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.403+0000 7fc7c8000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 0x7fc7c00cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.403+0000 7fc7c8000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c00ce230 con 0x7fc7c00cccd0 2024-03-22T13:00:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.403+0000 7fc7c8000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c00ce510 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.403+0000 7fc7c8000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c00ce7f0 con 0x7fc7c00cf770 2024-03-22T13:00:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.404+0000 7fc7c6000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc7c00cccd0 0x7fc7c015ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35966/0 (socket says 172.21.15.6:35966) 2024-03-22T13:00:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.404+0000 7fc7c6000640 1 -- 172.21.15.6:0/4157271290 learned_addr learned my addr 172.21.15.6:0/4157271290 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:22.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.405+0000 7fc7c5600640 1 --2- 172.21.15.6:0/4157271290 >> v2:172.21.15.6:3300/0 conn(0x7fc7c00cc3b0 0x7fc7c00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:22.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.408+0000 7fc7c5600640 1 -- 172.21.15.6:0/4157271290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 msgr2=0x7fc7c00cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:22.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.408+0000 7fc7c5600640 1 --2- 172.21.15.6:0/4157271290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 0x7fc7c00cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:22.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.408+0000 7fc7c5600640 1 -- 172.21.15.6:0/4157271290 >> v1:172.21.15.92:6789/0 conn(0x7fc7c00cccd0 legacy=0x7fc7c015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.408+0000 7fc7c5600640 1 -- 172.21.15.6:0/4157271290 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00cead0 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.410+0000 7fc7c5600640 1 --2- 172.21.15.6:0/4157271290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00cf230 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40097a0 tx=0x7fc7b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b2dbcdb62f637d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:22.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.418+0000 7fc7bfe00640 1 -- 172.21.15.6:0/4157271290 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b40315a0 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.418+0000 7fc7bfe00640 1 -- 172.21.15.6:0/4157271290 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b4037070 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.418+0000 7fc7bfe00640 1 -- 172.21.15.6:0/4157271290 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4031960 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 -- 172.21.15.6:0/4157271290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 msgr2=0x7fc7c00cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:22.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 --2- 172.21.15.6:0/4157271290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00cf230 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40097a0 tx=0x7fc7b402b840 comp rx=0 tx=0).stop 2024-03-22T13:00:22.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 -- 172.21.15.6:0/4157271290 shutdown_connections 2024-03-22T13:00:22.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 --2- 172.21.15.6:0/4157271290 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 0x7fc7c00cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:22.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 --2- 172.21.15.6:0/4157271290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00cf230 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:22.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 -- 172.21.15.6:0/4157271290 >> 172.21.15.6:0/4157271290 conn(0x7fc7c00b16d0 msgr2=0x7fc7c00b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:22.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 -- 172.21.15.6:0/4157271290 shutdown_connections 2024-03-22T13:00:22.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.419+0000 7fc7c8000640 1 -- 172.21.15.6:0/4157271290 wait complete. 2024-03-22T13:00:22.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.421+0000 7fc7c8000640 1 Processor -- start 2024-03-22T13:00:22.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.421+0000 7fc7c8000640 1 -- start start 2024-03-22T13:00:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.422+0000 7fc7c8000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.422+0000 7fc7c8000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7c00cccd0 0x7fc7c00c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.422+0000 7fc7c8000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 0x7fc7c01f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.422+0000 7fc7c8000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c015a5d0 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.422+0000 7fc7c8000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c006dc10 con 0x7fc7c00cccd0 2024-03-22T13:00:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.422+0000 7fc7c8000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c0154bb0 con 0x7fc7c00cf770 2024-03-22T13:00:22.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.423+0000 7fc7c5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:22.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.423+0000 7fc7c5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38658/0 (socket says 172.21.15.6:38658) 2024-03-22T13:00:22.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.423+0000 7fc7c5600640 1 -- 172.21.15.6:0/522642039 learned_addr learned my addr 172.21.15.6:0/522642039 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:22.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.429+0000 7fc7c5600640 1 -- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 msgr2=0x7fc7c01f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:22.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.429+0000 7fc7c5600640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 0x7fc7c01f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:22.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.429+0000 7fc7c5600640 1 -- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7c00cccd0 msgr2=0x7fc7c00c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:22.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.429+0000 7fc7c5600640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7c00cccd0 0x7fc7c00c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:22.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.429+0000 7fc7c5600640 1 -- 172.21.15.6:0/522642039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c0157a50 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.433+0000 7fc7c5600640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00c9180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4034040 tx=0x7fc7b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:22.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.438+0000 7fc7bd600640 1 -- 172.21.15.6:0/522642039 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4032840 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.438+0000 7fc7bd600640 1 -- 172.21.15.6:0/522642039 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b4032a00 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.438+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c0156f50 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.438+0000 7fc7bd600640 1 -- 172.21.15.6:0/522642039 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4032df0 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.438+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c01f4b10 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.439+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c0073240 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.475+0000 7fc7bd600640 1 -- 172.21.15.6:0/522642039 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc7b404e420 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.476+0000 7fc7bd600640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7a006bc20 0x7fc7a006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:22.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.476+0000 7fc7c4c00640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7a006bc20 0x7fc7a006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:22.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.477+0000 7fc7c4c00640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7a006bc20 0x7fc7a006e0e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00b5230 tx=0x7fc7ac006bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:22.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.480+0000 7fc7bd600640 1 -- 172.21.15.6:0/522642039 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc7b402f030 con 0x7fc7c00cc3b0 2024-03-22T13:00:22.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:22.794+0000 7fc7bd600640 1 -- 172.21.15.6:0/522642039 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc7c0073240 con 0x7fc7c00cc3b0 2024-03-22T13:00:23.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.047+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7c00756c0 con 0x7fc7c00cc3b0 2024-03-22T13:00:23.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.065+0000 7fc7bd600640 1 -- 172.21.15.6:0/522642039 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fc7b4040070 con 0x7fc7c00cc3b0 2024-03-22T13:00:23.066 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:23.066 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:23.066 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:23.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7a006bc20 msgr2=0x7fc7a006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:23.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7a006bc20 0x7fc7a006e0e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00b5230 tx=0x7fc7ac006bf0 comp rx=0 tx=0).stop 2024-03-22T13:00:23.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 msgr2=0x7fc7c00c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00c9180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4034040 tx=0x7fc7b40046c0 comp rx=0 tx=0).stop 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 shutdown_connections 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7c00cf770 0x7fc7c01f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7a006bc20 0x7fc7a006e0e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7c00cccd0 0x7fc7c00c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 --2- 172.21.15.6:0/522642039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c00cc3b0 0x7fc7c00c9180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 >> 172.21.15.6:0/522642039 conn(0x7fc7c00b16d0 msgr2=0x7fc7c00b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.067+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 shutdown_connections 2024-03-22T13:00:23.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:23.068+0000 7fc7c8000640 1 -- 172.21.15.6:0/522642039 wait complete. 2024-03-22T13:00:23.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:23.692 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:24.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.895+0000 7f3e28a00640 1 Processor -- start 2024-03-22T13:00:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.895+0000 7f3e28a00640 1 -- start start 2024-03-22T13:00:24.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.897+0000 7f3e28a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e240cc380 0x7f3e2406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:24.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.897+0000 7f3e28a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3e240ce8a0 0x7f3e2415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:24.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.897+0000 7f3e28a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e240cee00 con 0x7f3e240cc760 2024-03-22T13:00:24.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.897+0000 7f3e28a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e240cf0c0 con 0x7f3e240cc380 2024-03-22T13:00:24.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.897+0000 7f3e28a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e240cf3a0 con 0x7f3e240ce8a0 2024-03-22T13:00:24.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.899+0000 7f3e22a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3e240ce8a0 0x7f3e2415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:24.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.899+0000 7f3e23400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e240cc380 0x7f3e2406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:24.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.899+0000 7f3e23e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3e240cc760 0x7f3e240cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57542/0 (socket says 172.21.15.6:57542) 2024-03-22T13:00:24.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.899+0000 7f3e23e00640 1 -- 172.21.15.6:0/3758288164 learned_addr learned my addr 172.21.15.6:0/3758288164 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:24.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.902+0000 7f3e22a00640 1 -- 172.21.15.6:0/3758288164 >> v2:172.21.15.6:3300/0 conn(0x7f3e240cc380 msgr2=0x7f3e2406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:24.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.902+0000 7f3e22a00640 1 --2- 172.21.15.6:0/3758288164 >> v2:172.21.15.6:3300/0 conn(0x7f3e240cc380 0x7f3e2406c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:24.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.902+0000 7f3e22a00640 1 -- 172.21.15.6:0/3758288164 >> v1:172.21.15.6:6789/0 conn(0x7f3e240cc760 legacy=0x7f3e240cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:24.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.902+0000 7f3e22a00640 1 -- 172.21.15.6:0/3758288164 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240cf680 con 0x7f3e240ce8a0 2024-03-22T13:00:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.906+0000 7f3e22a00640 1 --2- 172.21.15.6:0/3758288164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 0x7f3e2415ae30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e240cb7f0 tx=0x7f3e1002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9708605571b9ffae server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:24.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.910+0000 7f3e22000640 1 -- 172.21.15.6:0/3758288164 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10035020 con 0x7f3e240ce8a0 2024-03-22T13:00:24.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.910+0000 7f3e22000640 1 -- 172.21.15.6:0/3758288164 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e10031830 con 0x7f3e240ce8a0 2024-03-22T13:00:24.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.910+0000 7f3e22000640 1 -- 172.21.15.6:0/3758288164 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10031c00 con 0x7f3e240ce8a0 2024-03-22T13:00:24.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.911+0000 7f3e28a00640 1 -- 172.21.15.6:0/3758288164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 msgr2=0x7f3e2415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:24.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.911+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3758288164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 0x7f3e2415ae30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e240cb7f0 tx=0x7f3e1002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:00:24.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.911+0000 7f3e28a00640 1 -- 172.21.15.6:0/3758288164 shutdown_connections 2024-03-22T13:00:24.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.911+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3758288164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 0x7f3e2415ae30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:24.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.911+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3758288164 >> v2:172.21.15.6:3300/0 conn(0x7f3e240cc380 0x7f3e2406c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:24.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.911+0000 7f3e28a00640 1 -- 172.21.15.6:0/3758288164 >> 172.21.15.6:0/3758288164 conn(0x7f3e240b1480 msgr2=0x7f3e240b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:24.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.912+0000 7f3e28a00640 1 -- 172.21.15.6:0/3758288164 shutdown_connections 2024-03-22T13:00:24.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.912+0000 7f3e28a00640 1 -- 172.21.15.6:0/3758288164 wait complete. 2024-03-22T13:00:24.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.914+0000 7f3e28a00640 1 Processor -- start 2024-03-22T13:00:24.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.914+0000 7f3e28a00640 1 -- start start 2024-03-22T13:00:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.916+0000 7f3e28a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e240cc380 0x7f3e240c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.916+0000 7f3e28a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e240cc760 0x7f3e2415b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.916+0000 7f3e28a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 0x7f3e2415c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.916+0000 7f3e28a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e241549d0 con 0x7f3e240cc380 2024-03-22T13:00:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.916+0000 7f3e28a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2415a910 con 0x7f3e240ce8a0 2024-03-22T13:00:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.916+0000 7f3e28a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240ccb70 con 0x7f3e240cc760 2024-03-22T13:00:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.917+0000 7f3e23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e240cc380 0x7f3e240c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.917+0000 7f3e22a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e240cc760 0x7f3e2415b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.917+0000 7f3e23e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 0x7f3e2415c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.917+0000 7f3e23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e240cc380 0x7f3e240c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58208/0 (socket says 172.21.15.6:58208) 2024-03-22T13:00:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.917+0000 7f3e23400640 1 -- 172.21.15.6:0/3852980611 learned_addr learned my addr 172.21.15.6:0/3852980611 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:24.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.921+0000 7f3e22a00640 1 -- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 msgr2=0x7f3e2415c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:24.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.921+0000 7f3e22a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 0x7f3e2415c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:24.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.921+0000 7f3e22a00640 1 -- 172.21.15.6:0/3852980611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e240cc380 msgr2=0x7f3e240c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:24.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.921+0000 7f3e22a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e240cc380 0x7f3e240c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:24.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.921+0000 7f3e22a00640 1 -- 172.21.15.6:0/3852980611 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240c8c30 con 0x7f3e240cc760 2024-03-22T13:00:24.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.923+0000 7f3e22a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e240cc760 0x7f3e2415b6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3e10033000 tx=0x7f3e100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.927+0000 7f3e1b400640 1 -- 172.21.15.6:0/3852980611 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10040070 con 0x7f3e240cc760 2024-03-22T13:00:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.927+0000 7f3e1b400640 1 -- 172.21.15.6:0/3852980611 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1003a8f0 con 0x7f3e240cc760 2024-03-22T13:00:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.927+0000 7f3e1b400640 1 -- 172.21.15.6:0/3852980611 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10035040 con 0x7f3e240cc760 2024-03-22T13:00:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.927+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e241f0040 con 0x7f3e240cc760 2024-03-22T13:00:24.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.927+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e241f0600 con 0x7f3e240cc760 2024-03-22T13:00:24.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.928+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e24075650 con 0x7f3e240cc760 2024-03-22T13:00:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.949+0000 7f3e1b400640 1 -- 172.21.15.6:0/3852980611 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3e10038020 con 0x7f3e240cc760 2024-03-22T13:00:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.951+0000 7f3e1b400640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e0406bc20 0x7f3e0406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:24.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.951+0000 7f3e1b400640 1 -- 172.21.15.6:0/3852980611 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3e10078f80 con 0x7f3e240cc760 2024-03-22T13:00:24.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.951+0000 7f3e23400640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e0406bc20 0x7f3e0406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:24.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:24.952+0000 7f3e23400640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e0406bc20 0x7f3e0406e0e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009a40 tx=0x7f3e1400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.202+0000 7f3e1b400640 1 -- 172.21.15.6:0/3852980611 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3e10031800 con 0x7f3e240cc760 2024-03-22T13:00:25.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.458+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e2415b060 con 0x7f3e240cc760 2024-03-22T13:00:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.516+0000 7f3e1b400640 1 -- 172.21.15.6:0/3852980611 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f3e1007aee0 con 0x7f3e240cc760 2024-03-22T13:00:25.516 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:25.516 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e0406bc20 msgr2=0x7f3e0406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e0406bc20 0x7f3e0406e0e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009a40 tx=0x7f3e1400b040 comp rx=0 tx=0).stop 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e240cc760 msgr2=0x7f3e2415b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e240cc760 0x7f3e2415b6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3e10033000 tx=0x7f3e100046c0 comp rx=0 tx=0).stop 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 shutdown_connections 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e0406bc20 0x7f3e0406e0e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240ce8a0 0x7f3e2415c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e240cc760 0x7f3e2415b6b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 --2- 172.21.15.6:0/3852980611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e240cc380 0x7f3e240c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 >> 172.21.15.6:0/3852980611 conn(0x7f3e240b1480 msgr2=0x7f3e240cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 shutdown_connections 2024-03-22T13:00:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:25.519+0000 7f3e28a00640 1 -- 172.21.15.6:0/3852980611 wait complete. 2024-03-22T13:00:26.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:26.140 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:27.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:27.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.341+0000 7f9c4b800640 1 Processor -- start 2024-03-22T13:00:27.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.341+0000 7f9c4b800640 1 -- start start 2024-03-22T13:00:27.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.343+0000 7f9c4b800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c44152400 0x7f9c4415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:27.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.343+0000 7f9c4b800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c441590b0 con 0x7f9c4415bc80 2024-03-22T13:00:27.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.343+0000 7f9c4b800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c440721f0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.343+0000 7f9c4b800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c440724d0 con 0x7f9c44152400 2024-03-22T13:00:27.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.344+0000 7f9c48e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c44152400 0x7f9c4415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:27.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.344+0000 7f9c49800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9c4406f8c0 0x7f9c440c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:58816/0 (socket says 172.21.15.6:58816) 2024-03-22T13:00:27.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.344+0000 7f9c49800640 1 -- 172.21.15.6:0/3632517477 learned_addr learned my addr 172.21.15.6:0/3632517477 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:27.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.348+0000 7f9c43400640 1 -- 172.21.15.6:0/3632517477 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079977612 0 0) 0x7f9c440721f0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.348+0000 7f9c43400640 1 -- 172.21.15.6:0/3632517477 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c2c003520 con 0x7f9c4406f8c0 2024-03-22T13:00:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.350+0000 7f9c48e00640 1 -- 172.21.15.6:0/3632517477 >> v1:172.21.15.92:6789/0 conn(0x7f9c4406f8c0 legacy=0x7f9c440c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.350+0000 7f9c48e00640 1 -- 172.21.15.6:0/3632517477 >> v1:172.21.15.6:6789/0 conn(0x7f9c4415bc80 legacy=0x7f9c440716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:27.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.350+0000 7f9c48e00640 1 -- 172.21.15.6:0/3632517477 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c441582f0 con 0x7f9c44152400 2024-03-22T13:00:27.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.354+0000 7f9c48e00640 1 --2- 172.21.15.6:0/3632517477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c4415b740 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c380097a0 tx=0x7f9c3802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417e2f9a00d6da4b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.360+0000 7f9c43400640 1 -- 172.21.15.6:0/3632517477 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3802f810 con 0x7f9c44152400 2024-03-22T13:00:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.360+0000 7f9c43400640 1 -- 172.21.15.6:0/3632517477 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3802f9b0 con 0x7f9c44152400 2024-03-22T13:00:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.360+0000 7f9c43400640 1 -- 172.21.15.6:0/3632517477 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3802fd80 con 0x7f9c44152400 2024-03-22T13:00:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.361+0000 7f9c4b800640 1 -- 172.21.15.6:0/3632517477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 msgr2=0x7f9c4415b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.361+0000 7f9c4b800640 1 --2- 172.21.15.6:0/3632517477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c4415b740 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c380097a0 tx=0x7f9c3802b9f0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.361+0000 7f9c4b800640 1 -- 172.21.15.6:0/3632517477 shutdown_connections 2024-03-22T13:00:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.361+0000 7f9c4b800640 1 --2- 172.21.15.6:0/3632517477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c4415b740 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.361+0000 7f9c4b800640 1 -- 172.21.15.6:0/3632517477 >> 172.21.15.6:0/3632517477 conn(0x7f9c440cb930 msgr2=0x7f9c440cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.361+0000 7f9c4b800640 1 -- 172.21.15.6:0/3632517477 shutdown_connections 2024-03-22T13:00:27.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.361+0000 7f9c4b800640 1 -- 172.21.15.6:0/3632517477 wait complete. 2024-03-22T13:00:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.363+0000 7f9c4b800640 1 Processor -- start 2024-03-22T13:00:27.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.363+0000 7f9c4b800640 1 -- start start 2024-03-22T13:00:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.364+0000 7f9c4b800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c4406f8c0 0x7f9c441594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.364+0000 7f9c4b800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c44159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.364+0000 7f9c4b800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c4415bc80 0x7f9c44148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.364+0000 7f9c4b800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44158b90 con 0x7f9c44152400 2024-03-22T13:00:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.364+0000 7f9c4b800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c441588b0 con 0x7f9c4415bc80 2024-03-22T13:00:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.364+0000 7f9c4b800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c441527e0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.365+0000 7f9c43e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c44159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.365+0000 7f9c48e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c4406f8c0 0x7f9c441594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.365+0000 7f9c49800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c4415bc80 0x7f9c44148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.365+0000 7f9c43e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c44159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58226/0 (socket says 172.21.15.6:58226) 2024-03-22T13:00:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.365+0000 7f9c48e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c4406f8c0 0x7f9c441594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40652/0 (socket says 172.21.15.6:40652) 2024-03-22T13:00:27.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.365+0000 7f9c43e00640 1 -- 172.21.15.6:0/1117572692 learned_addr learned my addr 172.21.15.6:0/1117572692 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:27.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.369+0000 7f9c48e00640 1 -- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c4415bc80 msgr2=0x7f9c44148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:27.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.369+0000 7f9c48e00640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c4415bc80 0x7f9c44148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.369+0000 7f9c48e00640 1 -- 172.21.15.6:0/1117572692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 msgr2=0x7f9c44159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:27.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.369+0000 7f9c48e00640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c44159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.369+0000 7f9c48e00640 1 -- 172.21.15.6:0/1117572692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c44157120 con 0x7f9c4406f8c0 2024-03-22T13:00:27.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.372+0000 7f9c48e00640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c4406f8c0 0x7f9c441594c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9c38033000 tx=0x7f9c380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:27.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.377+0000 7f9c40c00640 1 -- 172.21.15.6:0/1117572692 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c380376d0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.377+0000 7f9c40c00640 1 -- 172.21.15.6:0/1117572692 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c380378b0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.377+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c44157400 con 0x7f9c4406f8c0 2024-03-22T13:00:27.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.377+0000 7f9c40c00640 1 -- 172.21.15.6:0/1117572692 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38037ca0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.377+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c441579c0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.378+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c440756c0 con 0x7f9c4406f8c0 2024-03-22T13:00:27.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.404+0000 7f9c40c00640 1 -- 172.21.15.6:0/1117572692 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9c38049020 con 0x7f9c4406f8c0 2024-03-22T13:00:27.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.405+0000 7f9c40c00640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9c2406bc50 0x7f9c2406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:27.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.406+0000 7f9c43e00640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9c2406bc50 0x7f9c2406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:27.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.406+0000 7f9c40c00640 1 -- 172.21.15.6:0/1117572692 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9c38079e10 con 0x7f9c4406f8c0 2024-03-22T13:00:27.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.407+0000 7f9c43e00640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9c2406bc50 0x7f9c2406e110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9c441f0d50 tx=0x7f9c30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.678+0000 7f9c40c00640 1 -- 172.21.15.6:0/1117572692 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9c38033030 con 0x7f9c4406f8c0 2024-03-22T13:00:27.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.954+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c440c6400 con 0x7f9c4406f8c0 2024-03-22T13:00:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.978+0000 7f9c40c00640 1 -- 172.21.15.6:0/1117572692 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f9c3807bd70 con 0x7f9c4406f8c0 2024-03-22T13:00:27.978 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:27.978 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:27.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9c2406bc50 msgr2=0x7f9c2406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9c2406bc50 0x7f9c2406e110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9c441f0d50 tx=0x7f9c30009040 comp rx=0 tx=0).stop 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c4406f8c0 msgr2=0x7f9c441594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c4406f8c0 0x7f9c441594c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9c38033000 tx=0x7f9c380046c0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 shutdown_connections 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9c2406bc50 0x7f9c2406e110 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c4415bc80 0x7f9c44148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c44152400 0x7f9c44159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 --2- 172.21.15.6:0/1117572692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c4406f8c0 0x7f9c441594c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.981+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 >> 172.21.15.6:0/1117572692 conn(0x7f9c440cb930 msgr2=0x7f9c4414fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.982+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 shutdown_connections 2024-03-22T13:00:27.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:27.982+0000 7f9c4b800640 1 -- 172.21.15.6:0/1117572692 wait complete. 2024-03-22T13:00:28.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:28.601 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:29.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.796+0000 7fe0c7000640 1 Processor -- start 2024-03-22T13:00:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.796+0000 7fe0c7000640 1 -- start start 2024-03-22T13:00:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.797+0000 7fe0c7000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0c0158a60 0x7fe0c014e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.797+0000 7fe0c7000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c006e0f0 con 0x7fe0c00c5210 2024-03-22T13:00:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.797+0000 7fe0c7000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c006d350 con 0x7fe0c00c55f0 2024-03-22T13:00:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.797+0000 7fe0c7000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c00c6f90 con 0x7fe0c0158a60 2024-03-22T13:00:29.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.798+0000 7fe0c6600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0c0158a60 0x7fe0c014e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.799+0000 7fe0c6600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0c0158a60 0x7fe0c014e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:42196/0 (socket says 172.21.15.6:42196) 2024-03-22T13:00:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.799+0000 7fe0c6600640 1 -- 172.21.15.6:0/70548781 learned_addr learned my addr 172.21.15.6:0/70548781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.803+0000 7fe0c6600640 1 -- 172.21.15.6:0/70548781 >> v1:172.21.15.92:6789/0 conn(0x7fe0c00c55f0 legacy=0x7fe0c006ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.803+0000 7fe0c6600640 1 -- 172.21.15.6:0/70548781 >> v1:172.21.15.6:6789/0 conn(0x7fe0c00c5210 legacy=0x7fe0c00c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.803+0000 7fe0c6600640 1 -- 172.21.15.6:0/70548781 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00c6cb0 con 0x7fe0c0158a60 2024-03-22T13:00:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.808+0000 7fe0c6600640 1 --2- 172.21.15.6:0/70548781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c0158a60 0x7fe0c014e4b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8009e50 tx=0x7fe0b802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34b773189ced87f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.814+0000 7fe0bfe00640 1 -- 172.21.15.6:0/70548781 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b802cb20 con 0x7fe0c0158a60 2024-03-22T13:00:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.815+0000 7fe0bfe00640 1 -- 172.21.15.6:0/70548781 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b802ccc0 con 0x7fe0c0158a60 2024-03-22T13:00:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.815+0000 7fe0bfe00640 1 -- 172.21.15.6:0/70548781 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8034610 con 0x7fe0c0158a60 2024-03-22T13:00:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.816+0000 7fe0c7000640 1 -- 172.21.15.6:0/70548781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c0158a60 msgr2=0x7fe0c014e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.816+0000 7fe0c7000640 1 --2- 172.21.15.6:0/70548781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c0158a60 0x7fe0c014e4b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8009e50 tx=0x7fe0b802bf80 comp rx=0 tx=0).stop 2024-03-22T13:00:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.816+0000 7fe0c7000640 1 -- 172.21.15.6:0/70548781 shutdown_connections 2024-03-22T13:00:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.816+0000 7fe0c7000640 1 --2- 172.21.15.6:0/70548781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c0158a60 0x7fe0c014e4b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.816+0000 7fe0c7000640 1 -- 172.21.15.6:0/70548781 >> 172.21.15.6:0/70548781 conn(0x7fe0c00cb890 msgr2=0x7fe0c00cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.816+0000 7fe0c7000640 1 -- 172.21.15.6:0/70548781 shutdown_connections 2024-03-22T13:00:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.816+0000 7fe0c7000640 1 -- 172.21.15.6:0/70548781 wait complete. 2024-03-22T13:00:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.818+0000 7fe0c7000640 1 Processor -- start 2024-03-22T13:00:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.818+0000 7fe0c7000640 1 -- start start 2024-03-22T13:00:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.819+0000 7fe0c7000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00c5210 0x7fe0c01528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.819+0000 7fe0c7000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00c55f0 0x7fe0c01ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.819+0000 7fe0c7000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c0158a60 0x7fe0c01565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.819+0000 7fe0c7000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c006dbd0 con 0x7fe0c00c5210 2024-03-22T13:00:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.819+0000 7fe0c7000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c006d8f0 con 0x7fe0c00c55f0 2024-03-22T13:00:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.819+0000 7fe0c7000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00c69a0 con 0x7fe0c0158a60 2024-03-22T13:00:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.820+0000 7fe0c6600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c0158a60 0x7fe0c01565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.820+0000 7fe0c5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00c55f0 0x7fe0c01ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.821+0000 7fe0c5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00c55f0 0x7fe0c01ee4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:42212/0 (socket says 172.21.15.6:42212) 2024-03-22T13:00:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.821+0000 7fe0c6600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c0158a60 0x7fe0c01565c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40666/0 (socket says 172.21.15.6:40666) 2024-03-22T13:00:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.821+0000 7fe0c5200640 1 -- 172.21.15.6:0/1283172894 learned_addr learned my addr 172.21.15.6:0/1283172894 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.825+0000 7fe0c5c00640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00c5210 0x7fe0c01528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.826+0000 7fe0c6600640 1 -- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00c55f0 msgr2=0x7fe0c01ee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.826+0000 7fe0c6600640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00c55f0 0x7fe0c01ee4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.826+0000 7fe0c6600640 1 -- 172.21.15.6:0/1283172894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00c5210 msgr2=0x7fe0c01528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.826+0000 7fe0c6600640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00c5210 0x7fe0c01528d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.826+0000 7fe0c6600640 1 -- 172.21.15.6:0/1283172894 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c01eea10 con 0x7fe0c0158a60 2024-03-22T13:00:29.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.829+0000 7fe0c6600640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c0158a60 0x7fe0c01565c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8032000 tx=0x7fe0b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.833+0000 7fe0bd600640 1 -- 172.21.15.6:0/1283172894 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8043020 con 0x7fe0c0158a60 2024-03-22T13:00:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.833+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c01454e0 con 0x7fe0c0158a60 2024-03-22T13:00:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.833+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c0145aa0 con 0x7fe0c0158a60 2024-03-22T13:00:29.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:29.834+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c0075650 con 0x7fe0c0158a60 2024-03-22T13:00:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.197+0000 7fe0bd600640 1 -- 172.21.15.6:0/1283172894 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b802cb30 con 0x7fe0c0158a60 2024-03-22T13:00:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.197+0000 7fe0bd600640 1 -- 172.21.15.6:0/1283172894 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b803e650 con 0x7fe0c0158a60 2024-03-22T13:00:30.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.199+0000 7fe0bd600640 1 -- 172.21.15.6:0/1283172894 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe0b8046020 con 0x7fe0c0158a60 2024-03-22T13:00:30.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.201+0000 7fe0bd600640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe0a006bbd0 0x7fe0a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:30.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.202+0000 7fe0c5c00640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe0a006bbd0 0x7fe0a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.202+0000 7fe0c5c00640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe0a006bbd0 0x7fe0a006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0015d0 tx=0x7fe0ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.204+0000 7fe0bd600640 1 -- 172.21.15.6:0/1283172894 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe0b8079f30 con 0x7fe0c0158a60 2024-03-22T13:00:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.216+0000 7fe0bd600640 1 -- 172.21.15.6:0/1283172894 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe0b804d030 con 0x7fe0c0158a60 2024-03-22T13:00:30.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:00:30.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:00:30.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.516+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0c0155900 con 0x7fe0c0158a60 2024-03-22T13:00:30.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.530+0000 7fe0bd600640 1 -- 172.21.15.6:0/1283172894 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fe0b807c590 con 0x7fe0c0158a60 2024-03-22T13:00:30.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:30.530 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:30.531 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe0a006bbd0 msgr2=0x7fe0a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe0a006bbd0 0x7fe0a006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0015d0 tx=0x7fe0ac009040 comp rx=0 tx=0).stop 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c0158a60 msgr2=0x7fe0c01565c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c0158a60 0x7fe0c01565c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8032000 tx=0x7fe0b8004a40 comp rx=0 tx=0).stop 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 shutdown_connections 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c0158a60 0x7fe0c01565c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe0a006bbd0 0x7fe0a006e090 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:30.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00c55f0 0x7fe0c01ee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 --2- 172.21.15.6:0/1283172894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00c5210 0x7fe0c01528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 >> 172.21.15.6:0/1283172894 conn(0x7fe0c00cb890 msgr2=0x7fe0c00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 shutdown_connections 2024-03-22T13:00:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:30.534+0000 7fe0c7000640 1 -- 172.21.15.6:0/1283172894 wait complete. 2024-03-22T13:00:31.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:31.156 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:32.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:32.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.369+0000 7f458bc00640 1 Processor -- start 2024-03-22T13:00:32.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.369+0000 7f458bc00640 1 -- start start 2024-03-22T13:00:32.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.371+0000 7f458bc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45840c5c20 0x7f4584156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:32.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.371+0000 7f458bc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45840c80f0 con 0x7f4584150f50 2024-03-22T13:00:32.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.372+0000 7f458bc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45840c3bf0 con 0x7f4584157410 2024-03-22T13:00:32.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.372+0000 7f458bc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840c3760 con 0x7f45840c5c20 2024-03-22T13:00:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.372+0000 7f4583e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45840c5c20 0x7f4584156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.372+0000 7f4589200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4584150f50 0x7f45840c79e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57562/0 (socket says 172.21.15.6:57562) 2024-03-22T13:00:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.372+0000 7f4589200640 1 -- 172.21.15.6:0/642997303 learned_addr learned my addr 172.21.15.6:0/642997303 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.375+0000 7f4583e00640 1 -- 172.21.15.6:0/642997303 >> v1:172.21.15.92:6789/0 conn(0x7f4584157410 legacy=0x7f458406b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.375+0000 7f4583e00640 1 -- 172.21.15.6:0/642997303 >> v1:172.21.15.6:6789/0 conn(0x7f4584150f50 legacy=0x7f45840c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:32.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.375+0000 7f4583e00640 1 -- 172.21.15.6:0/642997303 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584071e70 con 0x7f45840c5c20 2024-03-22T13:00:32.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.377+0000 7f4583e00640 1 --2- 172.21.15.6:0/642997303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45840c5c20 0x7f4584156ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f45700097d0 tx=0x7f457002b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d55ace8724a6bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.381+0000 7f4583400640 1 -- 172.21.15.6:0/642997303 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570034070 con 0x7f45840c5c20 2024-03-22T13:00:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.381+0000 7f4583400640 1 -- 172.21.15.6:0/642997303 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4570030890 con 0x7f45840c5c20 2024-03-22T13:00:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f4583400640 1 -- 172.21.15.6:0/642997303 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570030c60 con 0x7f45840c5c20 2024-03-22T13:00:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f458bc00640 1 -- 172.21.15.6:0/642997303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45840c5c20 msgr2=0x7f4584156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f458bc00640 1 --2- 172.21.15.6:0/642997303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45840c5c20 0x7f4584156ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f45700097d0 tx=0x7f457002b9d0 comp rx=0 tx=0).stop 2024-03-22T13:00:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f458bc00640 1 -- 172.21.15.6:0/642997303 shutdown_connections 2024-03-22T13:00:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f458bc00640 1 --2- 172.21.15.6:0/642997303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45840c5c20 0x7f4584156ed0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f458bc00640 1 -- 172.21.15.6:0/642997303 >> 172.21.15.6:0/642997303 conn(0x7f45840cb930 msgr2=0x7f45840cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f458bc00640 1 -- 172.21.15.6:0/642997303 shutdown_connections 2024-03-22T13:00:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.382+0000 7f458bc00640 1 -- 172.21.15.6:0/642997303 wait complete. 2024-03-22T13:00:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.384+0000 7f458bc00640 1 Processor -- start 2024-03-22T13:00:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.384+0000 7f458bc00640 1 -- start start 2024-03-22T13:00:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.385+0000 7f458bc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45840c5c20 0x7f45841e0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.385+0000 7f458bc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4584150f50 0x7f458414ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.385+0000 7f458bc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4584157410 0x7f4584145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.385+0000 7f458bc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840c3110 con 0x7f45840c5c20 2024-03-22T13:00:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.385+0000 7f458bc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840c2e30 con 0x7f4584157410 2024-03-22T13:00:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.385+0000 7f458bc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840c33f0 con 0x7f4584150f50 2024-03-22T13:00:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.386+0000 7f4583e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4584150f50 0x7f458414ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.386+0000 7f4589200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45840c5c20 0x7f45841e0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.386+0000 7f4589c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4584157410 0x7f4584145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.386+0000 7f4583e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4584150f50 0x7f458414ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40688/0 (socket says 172.21.15.6:40688) 2024-03-22T13:00:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.386+0000 7f4583e00640 1 -- 172.21.15.6:0/3180387242 learned_addr learned my addr 172.21.15.6:0/3180387242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.390+0000 7f4589200640 1 -- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4584150f50 msgr2=0x7f458414ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.390+0000 7f4589200640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4584150f50 0x7f458414ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.390+0000 7f4589200640 1 -- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4584157410 msgr2=0x7f4584145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.390+0000 7f4589200640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4584157410 0x7f4584145200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.390+0000 7f4589200640 1 -- 172.21.15.6:0/3180387242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458415a8b0 con 0x7f45840c5c20 2024-03-22T13:00:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.393+0000 7f4589200640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45840c5c20 0x7f45841e0c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f457800c470 tx=0x7f4578001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.397+0000 7f4580c00640 1 -- 172.21.15.6:0/3180387242 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578017070 con 0x7f45840c5c20 2024-03-22T13:00:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.397+0000 7f4580c00640 1 -- 172.21.15.6:0/3180387242 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578005ce0 con 0x7f45840c5c20 2024-03-22T13:00:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.397+0000 7f4580c00640 1 -- 172.21.15.6:0/3180387242 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45780072d0 con 0x7f45840c5c20 2024-03-22T13:00:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.397+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458414f750 con 0x7f45840c5c20 2024-03-22T13:00:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.398+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458414fc00 con 0x7f45840c5c20 2024-03-22T13:00:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.399+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45840756c0 con 0x7f45840c5c20 2024-03-22T13:00:32.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.422+0000 7f4580c00640 1 -- 172.21.15.6:0/3180387242 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f457801a070 con 0x7f45840c5c20 2024-03-22T13:00:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.423+0000 7f4580c00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f456406bbd0 0x7f456406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.424+0000 7f4583e00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f456406bbd0 0x7f456406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:32.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.425+0000 7f4583e00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f456406bbd0 0x7f456406e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f45840cb850 tx=0x7f457003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:32.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.739+0000 7f4580c00640 1 -- 172.21.15.6:0/3180387242 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4578056a70 con 0x7f45840c5c20 2024-03-22T13:00:32.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:32.739+0000 7f4580c00640 1 -- 172.21.15.6:0/3180387242 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f45840c80f0 con 0x7f45840c5c20 2024-03-22T13:00:33.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.005+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458415b2d0 con 0x7f45840c5c20 2024-03-22T13:00:33.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.029+0000 7f4580c00640 1 -- 172.21.15.6:0/3180387242 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f45780589d0 con 0x7f45840c5c20 2024-03-22T13:00:33.029 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:33.029 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:33.029 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f456406bbd0 msgr2=0x7f456406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f456406bbd0 0x7f456406e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f45840cb850 tx=0x7f457003b040 comp rx=0 tx=0).stop 2024-03-22T13:00:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45840c5c20 msgr2=0x7f45841e0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45840c5c20 0x7f45841e0c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f457800c470 tx=0x7f4578001e70 comp rx=0 tx=0).stop 2024-03-22T13:00:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 shutdown_connections 2024-03-22T13:00:33.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f456406bbd0 0x7f456406e090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4584157410 0x7f4584145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4584150f50 0x7f458414ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 --2- 172.21.15.6:0/3180387242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45840c5c20 0x7f45841e0c80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 >> 172.21.15.6:0/3180387242 conn(0x7f45840cb930 msgr2=0x7f4584157960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 shutdown_connections 2024-03-22T13:00:33.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:33.031+0000 7f458bc00640 1 -- 172.21.15.6:0/3180387242 wait complete. 2024-03-22T13:00:33.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:33.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:34.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:34.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.856+0000 7ff71fe00640 1 Processor -- start 2024-03-22T13:00:34.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.857+0000 7ff71fe00640 1 -- start start 2024-03-22T13:00:34.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.858+0000 7ff71fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7201589e0 0x7ff72015f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:34.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.858+0000 7ff71fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff7200c6880 0x7ff72006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:34.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.858+0000 7ff71fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7200715b0 con 0x7ff720072c10 2024-03-22T13:00:34.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.858+0000 7ff71fe00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7200c6dd0 con 0x7ff7200c6880 2024-03-22T13:00:34.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.858+0000 7ff71fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff720161f20 con 0x7ff7201589e0 2024-03-22T13:00:34.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.859+0000 7ff71ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7201589e0 0x7ff72015f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:34.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.859+0000 7ff71f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff7200c6880 0x7ff72006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:34.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.859+0000 7ff71ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7201589e0 0x7ff72015f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:58958/0 (socket says 172.21.15.6:58958) 2024-03-22T13:00:34.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.859+0000 7ff71ea00640 1 -- 172.21.15.6:0/410773319 learned_addr learned my addr 172.21.15.6:0/410773319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:34.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.863+0000 7ff71f400640 1 -- 172.21.15.6:0/410773319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7201589e0 msgr2=0x7ff72015f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:34.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.863+0000 7ff71f400640 1 --2- 172.21.15.6:0/410773319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7201589e0 0x7ff72015f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:34.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.863+0000 7ff71f400640 1 -- 172.21.15.6:0/410773319 >> v1:172.21.15.6:6789/0 conn(0x7ff720072c10 legacy=0x7ff7201623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:34.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.863+0000 7ff71f400640 1 -- 172.21.15.6:0/410773319 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720071890 con 0x7ff7200c6880 2024-03-22T13:00:34.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.866+0000 7ff71f400640 1 --2- 172.21.15.6:0/410773319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7200c6880 0x7ff72006d1d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff718009d90 tx=0x7ff71802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4aa462e3b248964 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:34.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71d600640 1 -- 172.21.15.6:0/410773319 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718039070 con 0x7ff7200c6880 2024-03-22T13:00:34.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71d600640 1 -- 172.21.15.6:0/410773319 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71802cbc0 con 0x7ff7200c6880 2024-03-22T13:00:34.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71d600640 1 -- 172.21.15.6:0/410773319 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718034440 con 0x7ff7200c6880 2024-03-22T13:00:34.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 -- 172.21.15.6:0/410773319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7200c6880 msgr2=0x7ff72006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:34.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 --2- 172.21.15.6:0/410773319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7200c6880 0x7ff72006d1d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff718009d90 tx=0x7ff71802be10 comp rx=0 tx=0).stop 2024-03-22T13:00:34.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 -- 172.21.15.6:0/410773319 shutdown_connections 2024-03-22T13:00:34.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 --2- 172.21.15.6:0/410773319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7200c6880 0x7ff72006d1d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:34.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 --2- 172.21.15.6:0/410773319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7201589e0 0x7ff72015f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:34.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 -- 172.21.15.6:0/410773319 >> 172.21.15.6:0/410773319 conn(0x7ff7200cb890 msgr2=0x7ff7200cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:34.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 -- 172.21.15.6:0/410773319 shutdown_connections 2024-03-22T13:00:34.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.871+0000 7ff71fe00640 1 -- 172.21.15.6:0/410773319 wait complete. 2024-03-22T13:00:34.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.874+0000 7ff71fe00640 1 Processor -- start 2024-03-22T13:00:34.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.874+0000 7ff71fe00640 1 -- start start 2024-03-22T13:00:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.876+0000 7ff71fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff720072c10 0x7ff72006d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.876+0000 7ff71fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7200c6880 0x7ff72014f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.876+0000 7ff71fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7201589e0 0x7ff72014fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.876+0000 7ff71fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720072450 con 0x7ff7200c6880 2024-03-22T13:00:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.876+0000 7ff71fe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72015f670 con 0x7ff7201589e0 2024-03-22T13:00:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.876+0000 7ff71fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff720158dc0 con 0x7ff720072c10 2024-03-22T13:00:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.877+0000 7ff71ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff720072c10 0x7ff72006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.877+0000 7ff71f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7201589e0 0x7ff72014fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.877+0000 7ff71ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff720072c10 0x7ff72006d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:58960/0 (socket says 172.21.15.6:58960) 2024-03-22T13:00:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.877+0000 7ff71ea00640 1 -- 172.21.15.6:0/1161590058 learned_addr learned my addr 172.21.15.6:0/1161590058 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.880+0000 7ff71e000640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7200c6880 0x7ff72014f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.882+0000 7ff71f400640 1 -- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff720072c10 msgr2=0x7ff72006d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.882+0000 7ff71f400640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff720072c10 0x7ff72006d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.882+0000 7ff71f400640 1 -- 172.21.15.6:0/1161590058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7200c6880 msgr2=0x7ff72014f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.882+0000 7ff71f400640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7200c6880 0x7ff72014f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.882+0000 7ff71f400640 1 -- 172.21.15.6:0/1161590058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720150210 con 0x7ff7201589e0 2024-03-22T13:00:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.885+0000 7ff71f400640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7201589e0 0x7ff72014fb10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff720067e10 tx=0x7ff718004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.893+0000 7ff716a00640 1 -- 172.21.15.6:0/1161590058 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718039050 con 0x7ff7201589e0 2024-03-22T13:00:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.893+0000 7ff716a00640 1 -- 172.21.15.6:0/1161590058 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71803a040 con 0x7ff7201589e0 2024-03-22T13:00:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.893+0000 7ff716a00640 1 -- 172.21.15.6:0/1161590058 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718009450 con 0x7ff7201589e0 2024-03-22T13:00:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.893+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7201f7920 con 0x7ff7201589e0 2024-03-22T13:00:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.893+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7201f7ee0 con 0x7ff7201589e0 2024-03-22T13:00:34.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:34.894+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff720075650 con 0x7ff7201589e0 2024-03-22T13:00:35.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.191+0000 7ff716a00640 1 -- 172.21.15.6:0/1161590058 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff718047020 con 0x7ff7201589e0 2024-03-22T13:00:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.193+0000 7ff716a00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff70006bbd0 0x7ff70006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.193+0000 7ff71ea00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff70006bbd0 0x7ff70006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:35.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.194+0000 7ff71ea00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff70006bbd0 0x7ff70006e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff7100094a0 tx=0x7ff710009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.195+0000 7ff716a00640 1 -- 172.21.15.6:0/1161590058 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff718079ce0 con 0x7ff7201589e0 2024-03-22T13:00:35.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.210+0000 7ff716a00640 1 -- 172.21.15.6:0/1161590058 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff720150210 con 0x7ff7201589e0 2024-03-22T13:00:35.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.468+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff720071f50 con 0x7ff7201589e0 2024-03-22T13:00:35.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.484+0000 7ff716a00640 1 -- 172.21.15.6:0/1161590058 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7ff71807bc40 con 0x7ff7201589e0 2024-03-22T13:00:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:35.484 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:35.484 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff70006bbd0 msgr2=0x7ff70006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff70006bbd0 0x7ff70006e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff7100094a0 tx=0x7ff710009240 comp rx=0 tx=0).stop 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7201589e0 msgr2=0x7ff72014fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7201589e0 0x7ff72014fb10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff720067e10 tx=0x7ff718004a40 comp rx=0 tx=0).stop 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 shutdown_connections 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff70006bbd0 0x7ff70006e090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7201589e0 0x7ff72014fb10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7200c6880 0x7ff72014f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 --2- 172.21.15.6:0/1161590058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff720072c10 0x7ff72006d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.486+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 >> 172.21.15.6:0/1161590058 conn(0x7ff7200cb890 msgr2=0x7ff72006f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.487+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 shutdown_connections 2024-03-22T13:00:35.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:35.487+0000 7ff71fe00640 1 -- 172.21.15.6:0/1161590058 wait complete. 2024-03-22T13:00:36.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:36.105 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:37.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:37.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.319+0000 7f32bde00640 1 Processor -- start 2024-03-22T13:00:37.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.320+0000 7f32bde00640 1 -- start start 2024-03-22T13:00:37.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.321+0000 7f32bde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32b8154240 0x7f32b815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:37.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.321+0000 7f32bde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b80c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:37.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.321+0000 7f32bde00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b8154620 con 0x7f32b80c68c0 2024-03-22T13:00:37.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.321+0000 7f32bde00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b815e130 con 0x7f32b8154240 2024-03-22T13:00:37.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.321+0000 7f32bde00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b80c6e00 con 0x7f32b80c7ee0 2024-03-22T13:00:37.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.322+0000 7f32afe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b80c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:37.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.323+0000 7f32afe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b80c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:58962/0 (socket says 172.21.15.6:58962) 2024-03-22T13:00:37.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.323+0000 7f32afe00640 1 -- 172.21.15.6:0/2673618298 learned_addr learned my addr 172.21.15.6:0/2673618298 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:37.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.323+0000 7f32b6a00640 1 --2- 172.21.15.6:0/2673618298 >> v2:172.21.15.6:3300/0 conn(0x7f32b8154240 0x7f32b815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.325+0000 7f32afe00640 1 -- 172.21.15.6:0/2673618298 >> v2:172.21.15.6:3300/0 conn(0x7f32b8154240 msgr2=0x7f32b815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.325+0000 7f32afe00640 1 --2- 172.21.15.6:0/2673618298 >> v2:172.21.15.6:3300/0 conn(0x7f32b8154240 0x7f32b815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.325+0000 7f32afe00640 1 -- 172.21.15.6:0/2673618298 >> v1:172.21.15.6:6789/0 conn(0x7f32b80c68c0 legacy=0x7f32b806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.325+0000 7f32afe00640 1 -- 172.21.15.6:0/2673618298 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80c70e0 con 0x7f32b80c7ee0 2024-03-22T13:00:37.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.327+0000 7f32afe00640 1 --2- 172.21.15.6:0/2673618298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b80c4030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009870 tx=0x7f32a002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49f7314d5fb03b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:37.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.331+0000 7f32b6000640 1 -- 172.21.15.6:0/2673618298 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a002f840 con 0x7f32b80c7ee0 2024-03-22T13:00:37.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.331+0000 7f32b6000640 1 -- 172.21.15.6:0/2673618298 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a002f9e0 con 0x7f32b80c7ee0 2024-03-22T13:00:37.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.331+0000 7f32b6000640 1 -- 172.21.15.6:0/2673618298 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a002fdb0 con 0x7f32b80c7ee0 2024-03-22T13:00:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.332+0000 7f32bde00640 1 -- 172.21.15.6:0/2673618298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 msgr2=0x7f32b80c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.332+0000 7f32bde00640 1 --2- 172.21.15.6:0/2673618298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b80c4030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009870 tx=0x7f32a002ba20 comp rx=0 tx=0).stop 2024-03-22T13:00:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.333+0000 7f32bde00640 1 -- 172.21.15.6:0/2673618298 shutdown_connections 2024-03-22T13:00:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.333+0000 7f32bde00640 1 --2- 172.21.15.6:0/2673618298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b80c4030 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.333+0000 7f32bde00640 1 --2- 172.21.15.6:0/2673618298 >> v2:172.21.15.6:3300/0 conn(0x7f32b8154240 0x7f32b815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.333+0000 7f32bde00640 1 -- 172.21.15.6:0/2673618298 >> 172.21.15.6:0/2673618298 conn(0x7f32b80cb930 msgr2=0x7f32b80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.333+0000 7f32bde00640 1 -- 172.21.15.6:0/2673618298 shutdown_connections 2024-03-22T13:00:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.333+0000 7f32bde00640 1 -- 172.21.15.6:0/2673618298 wait complete. 2024-03-22T13:00:37.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.335+0000 7f32bde00640 1 Processor -- start 2024-03-22T13:00:37.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.336+0000 7f32bde00640 1 -- start start 2024-03-22T13:00:37.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.337+0000 7f32bde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b80c68c0 0x7f32b80c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:37.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.337+0000 7f32bde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b8157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:37.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.337+0000 7f32bde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32b8154240 0x7f32b815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:37.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.337+0000 7f32bde00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b815ddc0 con 0x7f32b8154240 2024-03-22T13:00:37.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.337+0000 7f32bde00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b815dae0 con 0x7f32b80c68c0 2024-03-22T13:00:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.337+0000 7f32bde00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b815e5c0 con 0x7f32b80c7ee0 2024-03-22T13:00:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.338+0000 7f32b6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b80c68c0 0x7f32b80c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.338+0000 7f32afe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b8157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.338+0000 7f32b7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32b8154240 0x7f32b815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.338+0000 7f32b6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b80c68c0 0x7f32b80c51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:59796/0 (socket says 172.21.15.6:59796) 2024-03-22T13:00:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.338+0000 7f32b6a00640 1 -- 172.21.15.6:0/2523653680 learned_addr learned my addr 172.21.15.6:0/2523653680 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.342+0000 7f32b6a00640 1 -- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 msgr2=0x7f32b8157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.342+0000 7f32b6a00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b8157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.342+0000 7f32b6a00640 1 -- 172.21.15.6:0/2523653680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32b8154240 msgr2=0x7f32b815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.342+0000 7f32b6a00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32b8154240 0x7f32b815c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.342+0000 7f32b6a00640 1 -- 172.21.15.6:0/2523653680 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b81f3e30 con 0x7f32b80c68c0 2024-03-22T13:00:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.344+0000 7f32b6a00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b80c68c0 0x7f32b80c51a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f32a800e8a0 tx=0x7f32a800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.348+0000 7f32af400640 1 -- 172.21.15.6:0/2523653680 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8017070 con 0x7f32b80c68c0 2024-03-22T13:00:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.348+0000 7f32af400640 1 -- 172.21.15.6:0/2523653680 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a800c940 con 0x7f32b80c68c0 2024-03-22T13:00:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.348+0000 7f32af400640 1 -- 172.21.15.6:0/2523653680 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a800cd30 con 0x7f32b80c68c0 2024-03-22T13:00:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.348+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b81f4cb0 con 0x7f32b80c68c0 2024-03-22T13:00:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.348+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b81f4380 con 0x7f32b80c68c0 2024-03-22T13:00:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.350+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b80756c0 con 0x7f32b80c68c0 2024-03-22T13:00:37.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.621+0000 7f32af400640 1 -- 172.21.15.6:0/2523653680 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f32a801a070 con 0x7f32b80c68c0 2024-03-22T13:00:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.622+0000 7f32af400640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f329406bc20 0x7f329406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.623+0000 7f32afe00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f329406bc20 0x7f329406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.624+0000 7f32afe00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f329406bc20 0x7f329406e0e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32a0036000 tx=0x7f32a0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.625+0000 7f32af400640 1 -- 172.21.15.6:0/2523653680 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f32a80575a0 con 0x7f32b80c68c0 2024-03-22T13:00:37.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.644+0000 7f32af400640 1 -- 172.21.15.6:0/2523653680 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f32a8059500 con 0x7f32b80c68c0 2024-03-22T13:00:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.922+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32b815b030 con 0x7f32b80c68c0 2024-03-22T13:00:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.937+0000 7f32af400640 1 -- 172.21.15.6:0/2523653680 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f32a805d5a0 con 0x7f32b80c68c0 2024-03-22T13:00:37.937 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:37.937 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f329406bc20 msgr2=0x7f329406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f329406bc20 0x7f329406e0e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32a0036000 tx=0x7f32a0008030 comp rx=0 tx=0).stop 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b80c68c0 msgr2=0x7f32b80c51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b80c68c0 0x7f32b80c51a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f32a800e8a0 tx=0x7f32a800ed60 comp rx=0 tx=0).stop 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 shutdown_connections 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32b8154240 0x7f32b815c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32b80c7ee0 0x7f32b8157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f329406bc20 0x7f329406e0e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 --2- 172.21.15.6:0/2523653680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b80c68c0 0x7f32b80c51a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 >> 172.21.15.6:0/2523653680 conn(0x7f32b80cb930 msgr2=0x7f32b8153460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 shutdown_connections 2024-03-22T13:00:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:37.939+0000 7f32bde00640 1 -- 172.21.15.6:0/2523653680 wait complete. 2024-03-22T13:00:38.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:38.561 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:39.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:39.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.757+0000 7f35ef200640 1 Processor -- start 2024-03-22T13:00:39.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.757+0000 7f35ef200640 1 -- start start 2024-03-22T13:00:39.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.758+0000 7f35ef200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f35f00cc640 0x7f35f006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:39.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.758+0000 7f35ef200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35f00cca20 0x7f35f00ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:39.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.759+0000 7f35ef200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f006dd00 con 0x7f35f015b7a0 2024-03-22T13:00:39.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.759+0000 7f35ef200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f006d330 con 0x7f35f00cca20 2024-03-22T13:00:39.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.759+0000 7f35ef200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f006c440 con 0x7f35f00cc640 2024-03-22T13:00:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.790+0000 7f35ee800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f35f015b7a0 0x7f35f015d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43606/0 (socket says 172.21.15.6:43606) 2024-03-22T13:00:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.790+0000 7f35ee800640 1 -- 172.21.15.6:0/1890919521 learned_addr learned my addr 172.21.15.6:0/1890919521 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.792+0000 7f35ed400640 1 --2- 172.21.15.6:0/1890919521 >> v2:172.21.15.6:3300/0 conn(0x7f35f00cca20 0x7f35f00ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:39.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.799+0000 7f35ed400640 1 -- 172.21.15.6:0/1890919521 >> v2:172.21.15.92:3300/0 conn(0x7f35f00cc640 msgr2=0x7f35f006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:39.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.799+0000 7f35ed400640 1 --2- 172.21.15.6:0/1890919521 >> v2:172.21.15.92:3300/0 conn(0x7f35f00cc640 0x7f35f006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:39.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.799+0000 7f35ed400640 1 -- 172.21.15.6:0/1890919521 >> v1:172.21.15.6:6789/0 conn(0x7f35f015b7a0 legacy=0x7f35f015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:39.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.799+0000 7f35ed400640 1 -- 172.21.15.6:0/1890919521 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f006c720 con 0x7f35f00cca20 2024-03-22T13:00:39.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.836+0000 7f35ed400640 1 --2- 172.21.15.6:0/1890919521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cca20 0x7f35f00ca340 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f35f00cc320 tx=0x7f35dc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38cd5e6b02aec97e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.850+0000 7f35eca00640 1 -- 172.21.15.6:0/1890919521 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc034070 con 0x7f35f00cca20 2024-03-22T13:00:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.850+0000 7f35eca00640 1 -- 172.21.15.6:0/1890919521 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc030610 con 0x7f35f00cca20 2024-03-22T13:00:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.850+0000 7f35eca00640 1 -- 172.21.15.6:0/1890919521 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0309e0 con 0x7f35f00cca20 2024-03-22T13:00:39.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.851+0000 7f35ef200640 1 -- 172.21.15.6:0/1890919521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cca20 msgr2=0x7f35f00ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:39.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.851+0000 7f35ef200640 1 --2- 172.21.15.6:0/1890919521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cca20 0x7f35f00ca340 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f35f00cc320 tx=0x7f35dc02b960 comp rx=0 tx=0).stop 2024-03-22T13:00:39.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.851+0000 7f35ef200640 1 -- 172.21.15.6:0/1890919521 shutdown_connections 2024-03-22T13:00:39.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.851+0000 7f35ef200640 1 --2- 172.21.15.6:0/1890919521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cca20 0x7f35f00ca340 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:39.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.851+0000 7f35ef200640 1 --2- 172.21.15.6:0/1890919521 >> v2:172.21.15.92:3300/0 conn(0x7f35f00cc640 0x7f35f006d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:39.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.851+0000 7f35ef200640 1 -- 172.21.15.6:0/1890919521 >> 172.21.15.6:0/1890919521 conn(0x7f35f00cf9c0 msgr2=0x7f35f00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:39.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.852+0000 7f35ef200640 1 -- 172.21.15.6:0/1890919521 shutdown_connections 2024-03-22T13:00:39.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.852+0000 7f35ef200640 1 -- 172.21.15.6:0/1890919521 wait complete. 2024-03-22T13:00:39.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.855+0000 7f35ef200640 1 Processor -- start 2024-03-22T13:00:39.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.855+0000 7f35ef200640 1 -- start start 2024-03-22T13:00:39.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.856+0000 7f35ef200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cc640 0x7f35f00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:39.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.856+0000 7f35ef200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35f015b7a0 0x7f35f00cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:39.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.856+0000 7f35ef200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35f01f40e0 0x7f35f01f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:39.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.856+0000 7f35ef200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f006cfc0 con 0x7f35f00cc640 2024-03-22T13:00:39.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.856+0000 7f35ef200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f006cce0 con 0x7f35f01f40e0 2024-03-22T13:00:39.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.856+0000 7f35ef200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f015b000 con 0x7f35f015b7a0 2024-03-22T13:00:39.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.857+0000 7f35ed400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35f015b7a0 0x7f35f00cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:39.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.857+0000 7f35ee800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35f01f40e0 0x7f35f01f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:39.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.858+0000 7f35ed400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35f015b7a0 0x7f35f00cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:58974/0 (socket says 172.21.15.6:58974) 2024-03-22T13:00:39.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.858+0000 7f35ed400640 1 -- 172.21.15.6:0/712123592 learned_addr learned my addr 172.21.15.6:0/712123592 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:39.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.862+0000 7f35ede00640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cc640 0x7f35f00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.863+0000 7f35ed400640 1 -- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35f01f40e0 msgr2=0x7f35f01f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.863+0000 7f35ed400640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35f01f40e0 0x7f35f01f4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.863+0000 7f35ed400640 1 -- 172.21.15.6:0/712123592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cc640 msgr2=0x7f35f00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.863+0000 7f35ed400640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cc640 0x7f35f00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.863+0000 7f35ed400640 1 -- 172.21.15.6:0/712123592 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00c4950 con 0x7f35f015b7a0 2024-03-22T13:00:39.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.863+0000 7f35ee800640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35f01f40e0 0x7f35f01f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:00:39.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.867+0000 7f35ed400640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35f015b7a0 0x7f35f00cda10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000c00 tx=0x7f35dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.871+0000 7f35e6000640 1 -- 172.21.15.6:0/712123592 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc034040 con 0x7f35f015b7a0 2024-03-22T13:00:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.871+0000 7f35e6000640 1 -- 172.21.15.6:0/712123592 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc030410 con 0x7f35f015b7a0 2024-03-22T13:00:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.871+0000 7f35e6000640 1 -- 172.21.15.6:0/712123592 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc004250 con 0x7f35f015b7a0 2024-03-22T13:00:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.871+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f00c4bc0 con 0x7f35f015b7a0 2024-03-22T13:00:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.871+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f00c5180 con 0x7f35f015b7a0 2024-03-22T13:00:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:39.872+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f01f39b0 con 0x7f35f015b7a0 2024-03-22T13:00:40.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.132+0000 7f35e6000640 1 -- 172.21.15.6:0/712123592 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35dc04b060 con 0x7f35f015b7a0 2024-03-22T13:00:40.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.134+0000 7f35e6000640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35d006bc00 0x7f35d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:40.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.134+0000 7f35ede00640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35d006bc00 0x7f35d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:40.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.135+0000 7f35e6000640 1 -- 172.21.15.6:0/712123592 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f35dc078ed0 con 0x7f35f015b7a0 2024-03-22T13:00:40.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.135+0000 7f35ede00640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35d006bc00 0x7f35d006e0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009890 tx=0x7f35e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:40.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.146+0000 7f35e6000640 1 -- 172.21.15.6:0/712123592 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35dc07b530 con 0x7f35f015b7a0 2024-03-22T13:00:40.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.401+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35f0075650 con 0x7f35f015b7a0 2024-03-22T13:00:40.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.416+0000 7f35e6000640 1 -- 172.21.15.6:0/712123592 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f35dc07ae30 con 0x7f35f015b7a0 2024-03-22T13:00:40.416 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:40.416 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:40.416 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35d006bc00 msgr2=0x7f35d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35d006bc00 0x7f35d006e0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009890 tx=0x7f35e0008040 comp rx=0 tx=0).stop 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35f015b7a0 msgr2=0x7f35f00cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35f015b7a0 0x7f35f00cda10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000c00 tx=0x7f35dc0046c0 comp rx=0 tx=0).stop 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 shutdown_connections 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35d006bc00 0x7f35d006e0c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35f01f40e0 0x7f35f01f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35f015b7a0 0x7f35f00cda10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 --2- 172.21.15.6:0/712123592 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35f00cc640 0x7f35f00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:40.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 >> 172.21.15.6:0/712123592 conn(0x7f35f00cf9c0 msgr2=0x7f35f0158ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:40.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 shutdown_connections 2024-03-22T13:00:40.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:40.418+0000 7f35ef200640 1 -- 172.21.15.6:0/712123592 wait complete. 2024-03-22T13:00:41.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:41.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:42.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:42.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.239+0000 7f0979000640 1 Processor -- start 2024-03-22T13:00:42.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.239+0000 7f0979000640 1 -- start start 2024-03-22T13:00:42.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.240+0000 7f0979000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09740cc380 0x7f097406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:42.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.240+0000 7f0979000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f097415ae30 0x7f09740cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:42.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.240+0000 7f0979000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097415b240 con 0x7f09740ce8a0 2024-03-22T13:00:42.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.240+0000 7f0979000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09740cee00 con 0x7f097415ae30 2024-03-22T13:00:42.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.240+0000 7f0979000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09740cf0c0 con 0x7f09740cc380 2024-03-22T13:00:42.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.241+0000 7f0973400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09740cc380 0x7f097406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:42.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.241+0000 7f0973e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f097415ae30 0x7f09740cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:42.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.241+0000 7f096aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f09740ce8a0 0x7f09740cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43620/0 (socket says 172.21.15.6:43620) 2024-03-22T13:00:42.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.241+0000 7f096aa00640 1 -- 172.21.15.6:0/2027544937 learned_addr learned my addr 172.21.15.6:0/2027544937 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:42.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.244+0000 7f0973400640 1 -- 172.21.15.6:0/2027544937 >> v2:172.21.15.6:3300/0 conn(0x7f097415ae30 msgr2=0x7f09740cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:42.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.244+0000 7f0973400640 1 --2- 172.21.15.6:0/2027544937 >> v2:172.21.15.6:3300/0 conn(0x7f097415ae30 0x7f09740cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.244+0000 7f0973400640 1 -- 172.21.15.6:0/2027544937 >> v1:172.21.15.6:6789/0 conn(0x7f09740ce8a0 legacy=0x7f09740cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:42.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.244+0000 7f0973400640 1 -- 172.21.15.6:0/2027544937 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09740cf3a0 con 0x7f09740cc380 2024-03-22T13:00:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.246+0000 7f0973400640 1 --2- 172.21.15.6:0/2027544937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740cc380 0x7f097406c230 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f097408d0e0 tx=0x7f096402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee56b005c12578b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:42.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.251+0000 7f0972a00640 1 -- 172.21.15.6:0/2027544937 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964035020 con 0x7f09740cc380 2024-03-22T13:00:42.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.251+0000 7f0972a00640 1 -- 172.21.15.6:0/2027544937 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0964031830 con 0x7f09740cc380 2024-03-22T13:00:42.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.251+0000 7f0972a00640 1 -- 172.21.15.6:0/2027544937 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0964031c00 con 0x7f09740cc380 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 -- 172.21.15.6:0/2027544937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740cc380 msgr2=0x7f097406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 --2- 172.21.15.6:0/2027544937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740cc380 0x7f097406c230 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f097408d0e0 tx=0x7f096402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 -- 172.21.15.6:0/2027544937 shutdown_connections 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 --2- 172.21.15.6:0/2027544937 >> v2:172.21.15.6:3300/0 conn(0x7f097415ae30 0x7f09740cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 --2- 172.21.15.6:0/2027544937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740cc380 0x7f097406c230 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 -- 172.21.15.6:0/2027544937 >> 172.21.15.6:0/2027544937 conn(0x7f09740b1480 msgr2=0x7f09740b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 -- 172.21.15.6:0/2027544937 shutdown_connections 2024-03-22T13:00:42.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.252+0000 7f0979000640 1 -- 172.21.15.6:0/2027544937 wait complete. 2024-03-22T13:00:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.254+0000 7f0979000640 1 Processor -- start 2024-03-22T13:00:42.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.255+0000 7f0979000640 1 -- start start 2024-03-22T13:00:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.255+0000 7f0979000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09740cc380 0x7f09741f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.255+0000 7f0979000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740ce8a0 0x7f097415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:42.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.255+0000 7f0979000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f097415ae30 0x7f097415d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.255+0000 7f0979000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097415a910 con 0x7f097415ae30 2024-03-22T13:00:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.256+0000 7f0979000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097415a630 con 0x7f09740ce8a0 2024-03-22T13:00:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.256+0000 7f0979000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09741549d0 con 0x7f09740cc380 2024-03-22T13:00:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.256+0000 7f0973e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f097415ae30 0x7f097415d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.256+0000 7f096aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740ce8a0 0x7f097415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:42.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.256+0000 7f0973400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09740cc380 0x7f09741f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:42.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.257+0000 7f0973e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f097415ae30 0x7f097415d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47726/0 (socket says 172.21.15.6:47726) 2024-03-22T13:00:42.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.257+0000 7f0973e00640 1 -- 172.21.15.6:0/2356983760 learned_addr learned my addr 172.21.15.6:0/2356983760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:42.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.260+0000 7f0973e00640 1 -- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09740cc380 msgr2=0x7f09741f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:42.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.260+0000 7f0973e00640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09740cc380 0x7f09741f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.260+0000 7f0973e00640 1 -- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740ce8a0 msgr2=0x7f097415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.260+0000 7f0973e00640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740ce8a0 0x7f097415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.260+0000 7f0973e00640 1 -- 172.21.15.6:0/2356983760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09740ccd80 con 0x7f097415ae30 2024-03-22T13:00:42.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.263+0000 7f0973e00640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f097415ae30 0x7f097415d070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f096c00efc0 tx=0x7f096c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:42.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.268+0000 7f096be00640 1 -- 172.21.15.6:0/2356983760 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c018020 con 0x7f097415ae30 2024-03-22T13:00:42.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.268+0000 7f096be00640 1 -- 172.21.15.6:0/2356983760 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c014480 con 0x7f097415ae30 2024-03-22T13:00:42.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.268+0000 7f096be00640 1 -- 172.21.15.6:0/2356983760 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c014870 con 0x7f097415ae30 2024-03-22T13:00:42.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.268+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0974068410 con 0x7f097415ae30 2024-03-22T13:00:42.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.268+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0974156fc0 con 0x7f097415ae30 2024-03-22T13:00:42.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.270+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0974075650 con 0x7f097415ae30 2024-03-22T13:00:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.297+0000 7f096be00640 1 -- 172.21.15.6:0/2356983760 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f096c005d10 con 0x7f097415ae30 2024-03-22T13:00:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.297+0000 7f096be00640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f095006bc20 0x7f095006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.298+0000 7f0973400640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f095006bc20 0x7f095006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.299+0000 7f0973400640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f095006bc20 0x7f095006e0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0964036000 tx=0x7f096403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:42.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.591+0000 7f096be00640 1 -- 172.21.15.6:0/2356983760 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f096c058200 con 0x7f097415ae30 2024-03-22T13:00:42.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.591+0000 7f096be00640 1 -- 172.21.15.6:0/2356983760 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f096c022070 con 0x7f097415ae30 2024-03-22T13:00:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.858+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0974158d10 con 0x7f097415ae30 2024-03-22T13:00:42.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.927+0000 7f096be00640 1 -- 172.21.15.6:0/2356983760 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f096c010040 con 0x7f097415ae30 2024-03-22T13:00:42.927 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:42.928 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:42.928 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f095006bc20 msgr2=0x7f095006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:42.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f095006bc20 0x7f095006e0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0964036000 tx=0x7f096403c040 comp rx=0 tx=0).stop 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f097415ae30 msgr2=0x7f097415d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f097415ae30 0x7f097415d070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f096c00efc0 tx=0x7f096c00c490 comp rx=0 tx=0).stop 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 shutdown_connections 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f097415ae30 0x7f097415d070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f095006bc20 0x7f095006e0e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09740ce8a0 0x7f097415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 --2- 172.21.15.6:0/2356983760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09740cc380 0x7f09741f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.930+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 >> 172.21.15.6:0/2356983760 conn(0x7f09740b1480 msgr2=0x7f097406c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.931+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 shutdown_connections 2024-03-22T13:00:42.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:42.931+0000 7f0979000640 1 -- 172.21.15.6:0/2356983760 wait complete. 2024-03-22T13:00:43.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:43.552 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:44.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:44.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.753+0000 7f1911600640 1 Processor -- start 2024-03-22T13:00:44.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.753+0000 7f1911600640 1 -- start start 2024-03-22T13:00:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.754+0000 7f1911600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cc3b0 0x7f190c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.754+0000 7f1911600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f190c0ccea0 0x7f190c15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.754+0000 7f1911600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190c0ce230 con 0x7f190c0cf770 2024-03-22T13:00:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.754+0000 7f1911600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c0ce510 con 0x7f190c0ccea0 2024-03-22T13:00:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.754+0000 7f1911600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0ce7f0 con 0x7f190c0cc3b0 2024-03-22T13:00:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.755+0000 7f190b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f190c0ccea0 0x7f190c15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.755+0000 7f190aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cc3b0 0x7f190c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.756+0000 7f190aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cc3b0 0x7f190c0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44256/0 (socket says 172.21.15.6:44256) 2024-03-22T13:00:44.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.756+0000 7f190aa00640 1 -- 172.21.15.6:0/2965522820 learned_addr learned my addr 172.21.15.6:0/2965522820 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.758+0000 7f190aa00640 1 -- 172.21.15.6:0/2965522820 >> v2:172.21.15.92:3300/0 conn(0x7f190c0ccea0 msgr2=0x7f190c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.758+0000 7f190aa00640 1 --2- 172.21.15.6:0/2965522820 >> v2:172.21.15.92:3300/0 conn(0x7f190c0ccea0 0x7f190c15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.758+0000 7f190aa00640 1 -- 172.21.15.6:0/2965522820 >> v1:172.21.15.92:6789/0 conn(0x7f190c0cf770 legacy=0x7f190c0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:44.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.758+0000 7f190aa00640 1 -- 172.21.15.6:0/2965522820 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c0cead0 con 0x7f190c0cc3b0 2024-03-22T13:00:44.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.761+0000 7f190aa00640 1 --2- 172.21.15.6:0/2965522820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cc3b0 0x7f190c0cf230 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009870 tx=0x7f18fc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da454f248f8bfe6e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:44.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.763+0000 7f1909600640 1 -- 172.21.15.6:0/2965522820 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc035020 con 0x7f190c0cc3b0 2024-03-22T13:00:44.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.766+0000 7f1909600640 1 -- 172.21.15.6:0/2965522820 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc031640 con 0x7f190c0cc3b0 2024-03-22T13:00:44.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.766+0000 7f1909600640 1 -- 172.21.15.6:0/2965522820 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc02d040 con 0x7f190c0cc3b0 2024-03-22T13:00:44.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.766+0000 7f1911600640 1 -- 172.21.15.6:0/2965522820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cc3b0 msgr2=0x7f190c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:44.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.766+0000 7f1911600640 1 --2- 172.21.15.6:0/2965522820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cc3b0 0x7f190c0cf230 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009870 tx=0x7f18fc02b8f0 comp rx=0 tx=0).stop 2024-03-22T13:00:44.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.767+0000 7f1911600640 1 -- 172.21.15.6:0/2965522820 shutdown_connections 2024-03-22T13:00:44.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.767+0000 7f1911600640 1 --2- 172.21.15.6:0/2965522820 >> v2:172.21.15.92:3300/0 conn(0x7f190c0ccea0 0x7f190c15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:44.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.767+0000 7f1911600640 1 --2- 172.21.15.6:0/2965522820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cc3b0 0x7f190c0cf230 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:44.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.767+0000 7f1911600640 1 -- 172.21.15.6:0/2965522820 >> 172.21.15.6:0/2965522820 conn(0x7f190c0b16d0 msgr2=0x7f190c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:44.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.767+0000 7f1911600640 1 -- 172.21.15.6:0/2965522820 shutdown_connections 2024-03-22T13:00:44.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.767+0000 7f1911600640 1 -- 172.21.15.6:0/2965522820 wait complete. 2024-03-22T13:00:44.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.768+0000 7f1911600640 1 Processor -- start 2024-03-22T13:00:44.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.769+0000 7f1911600640 1 -- start start 2024-03-22T13:00:44.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.769+0000 7f1911600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f190c0cc3b0 0x7f190c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:44.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.769+0000 7f1911600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f190c0ccea0 0x7f190c1f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:44.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.769+0000 7f1911600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cf770 0x7f190c1f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:44.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.770+0000 7f1911600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c15a5d0 con 0x7f190c0ccea0 2024-03-22T13:00:44.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.770+0000 7f1911600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c06dc10 con 0x7f190c0cc3b0 2024-03-22T13:00:44.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.770+0000 7f1911600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c154bb0 con 0x7f190c0cf770 2024-03-22T13:00:44.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.770+0000 7f190aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f190c0cc3b0 0x7f190c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:44.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.770+0000 7f190b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cf770 0x7f190c1f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:44.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.770+0000 7f190a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f190c0ccea0 0x7f190c1f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:44.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.771+0000 7f190aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f190c0cc3b0 0x7f190c06ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:35006/0 (socket says 172.21.15.6:35006) 2024-03-22T13:00:44.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.771+0000 7f190aa00640 1 -- 172.21.15.6:0/1274716811 learned_addr learned my addr 172.21.15.6:0/1274716811 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.775+0000 7f190aa00640 1 -- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cf770 msgr2=0x7f190c1f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.775+0000 7f190aa00640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cf770 0x7f190c1f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.775+0000 7f190aa00640 1 -- 172.21.15.6:0/1274716811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f190c0ccea0 msgr2=0x7f190c1f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.775+0000 7f190aa00640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f190c0ccea0 0x7f190c1f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:44.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.776+0000 7f190aa00640 1 -- 172.21.15.6:0/1274716811 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c153750 con 0x7f190c0cc3b0 2024-03-22T13:00:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.792+0000 7f190aa00640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f190c0cc3b0 0x7f190c06ca00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f18fc000c00 tx=0x7f18fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.804+0000 7f1902a00640 1 -- 172.21.15.6:0/1274716811 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc040070 con 0x7f190c0cc3b0 2024-03-22T13:00:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.804+0000 7f1902a00640 1 -- 172.21.15.6:0/1274716811 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc0399f0 con 0x7f190c0cc3b0 2024-03-22T13:00:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.804+0000 7f1902a00640 1 -- 172.21.15.6:0/1274716811 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc035040 con 0x7f190c0cc3b0 2024-03-22T13:00:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.804+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c0b52a0 con 0x7f190c0cc3b0 2024-03-22T13:00:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.805+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c153a10 con 0x7f190c0cc3b0 2024-03-22T13:00:44.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:44.806+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c0b0f70 con 0x7f190c0cc3b0 2024-03-22T13:00:45.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.077+0000 7f1902a00640 1 -- 172.21.15.6:0/1274716811 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f18fc04b060 con 0x7f190c0cc3b0 2024-03-22T13:00:45.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.078+0000 7f1902a00640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f18ec06bc20 0x7f18ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.079+0000 7f190a000640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f18ec06bc20 0x7f18ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.080+0000 7f190a000640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f18ec06bc20 0x7f18ec06e0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f190c1f1460 tx=0x7f18f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.080+0000 7f1902a00640 1 -- 172.21.15.6:0/1274716811 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f18fc078be0 con 0x7f190c0cc3b0 2024-03-22T13:00:45.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.112+0000 7f1902a00640 1 -- 172.21.15.6:0/1274716811 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f190c0b0f70 con 0x7f190c0cc3b0 2024-03-22T13:00:45.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.367+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f190c0756c0 con 0x7f190c0cc3b0 2024-03-22T13:00:45.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.388+0000 7f1902a00640 1 -- 172.21.15.6:0/1274716811 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f18fc07ab40 con 0x7f190c0cc3b0 2024-03-22T13:00:45.388 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:45.388 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:45.389 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f18ec06bc20 msgr2=0x7f18ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f18ec06bc20 0x7f18ec06e0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f190c1f1460 tx=0x7f18f8009040 comp rx=0 tx=0).stop 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f190c0cc3b0 msgr2=0x7f190c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f190c0cc3b0 0x7f190c06ca00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f18fc000c00 tx=0x7f18fc0046c0 comp rx=0 tx=0).stop 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 shutdown_connections 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f190c0cf770 0x7f190c1f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f190c0ccea0 0x7f190c1f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f18ec06bc20 0x7f18ec06e0e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 --2- 172.21.15.6:0/1274716811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f190c0cc3b0 0x7f190c06ca00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 >> 172.21.15.6:0/1274716811 conn(0x7f190c0b16d0 msgr2=0x7f190c0cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 shutdown_connections 2024-03-22T13:00:45.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:45.391+0000 7f1911600640 1 -- 172.21.15.6:0/1274716811 wait complete. 2024-03-22T13:00:46.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:46.015 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:47.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.216+0000 7f7730200640 1 Processor -- start 2024-03-22T13:00:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.216+0000 7f7730200640 1 -- start start 2024-03-22T13:00:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.218+0000 7f7730200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7728154240 0x7f772815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.218+0000 7f7730200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77280c7ee0 0x7f77280c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.218+0000 7f7730200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7728154620 con 0x7f77280c68c0 2024-03-22T13:00:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.218+0000 7f7730200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772815e130 con 0x7f7728154240 2024-03-22T13:00:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.218+0000 7f7730200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77280c6e00 con 0x7f77280c7ee0 2024-03-22T13:00:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.219+0000 7f772ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77280c7ee0 0x7f77280c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.219+0000 7f772ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77280c7ee0 0x7f77280c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44270/0 (socket says 172.21.15.6:44270) 2024-03-22T13:00:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.219+0000 7f772ce00640 1 -- 172.21.15.6:0/2241433483 learned_addr learned my addr 172.21.15.6:0/2241433483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.219+0000 7f772d800640 1 --2- 172.21.15.6:0/2241433483 >> v2:172.21.15.6:3300/0 conn(0x7f7728154240 0x7f772815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:47.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.223+0000 7f772ce00640 1 -- 172.21.15.6:0/2241433483 >> v2:172.21.15.6:3300/0 conn(0x7f7728154240 msgr2=0x7f772815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:47.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.223+0000 7f772ce00640 1 --2- 172.21.15.6:0/2241433483 >> v2:172.21.15.6:3300/0 conn(0x7f7728154240 0x7f772815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.223+0000 7f772ce00640 1 -- 172.21.15.6:0/2241433483 >> v1:172.21.15.6:6789/0 conn(0x7f77280c68c0 legacy=0x7f772806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:47.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.223+0000 7f772ce00640 1 -- 172.21.15.6:0/2241433483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280c70e0 con 0x7f77280c7ee0 2024-03-22T13:00:47.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.226+0000 7f772ce00640 1 --2- 172.21.15.6:0/2241433483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77280c7ee0 0x7f77280c4030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7714009870 tx=0x7f771402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a42fabcedab50acb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:47.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.231+0000 7f7727e00640 1 -- 172.21.15.6:0/2241433483 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771402f830 con 0x7f77280c7ee0 2024-03-22T13:00:47.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.231+0000 7f7727e00640 1 -- 172.21.15.6:0/2241433483 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771402f9d0 con 0x7f77280c7ee0 2024-03-22T13:00:47.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.231+0000 7f7727e00640 1 -- 172.21.15.6:0/2241433483 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771402fda0 con 0x7f77280c7ee0 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 -- 172.21.15.6:0/2241433483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77280c7ee0 msgr2=0x7f77280c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 --2- 172.21.15.6:0/2241433483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77280c7ee0 0x7f77280c4030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7714009870 tx=0x7f771402ba10 comp rx=0 tx=0).stop 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 -- 172.21.15.6:0/2241433483 shutdown_connections 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 --2- 172.21.15.6:0/2241433483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77280c7ee0 0x7f77280c4030 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 --2- 172.21.15.6:0/2241433483 >> v2:172.21.15.6:3300/0 conn(0x7f7728154240 0x7f772815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 -- 172.21.15.6:0/2241433483 >> 172.21.15.6:0/2241433483 conn(0x7f77280cb930 msgr2=0x7f77280cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 -- 172.21.15.6:0/2241433483 shutdown_connections 2024-03-22T13:00:47.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.232+0000 7f7730200640 1 -- 172.21.15.6:0/2241433483 wait complete. 2024-03-22T13:00:47.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.234+0000 7f7730200640 1 Processor -- start 2024-03-22T13:00:47.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.234+0000 7f7730200640 1 -- start start 2024-03-22T13:00:47.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.235+0000 7f7730200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77280c68c0 0x7f77281586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:47.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.235+0000 7f7730200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77280c7ee0 0x7f772814a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:47.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.235+0000 7f7730200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7728154240 0x7f772814a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:47.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.235+0000 7f7730200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772815ddc0 con 0x7f77280c68c0 2024-03-22T13:00:47.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.235+0000 7f7730200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772815dae0 con 0x7f77280c7ee0 2024-03-22T13:00:47.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.235+0000 7f7730200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772815e5c0 con 0x7f7728154240 2024-03-22T13:00:47.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.235+0000 7f772d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77280c68c0 0x7f77281586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:47.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.236+0000 7f772e200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7728154240 0x7f772814a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:47.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.236+0000 7f772ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77280c7ee0 0x7f772814a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:47.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.236+0000 7f772d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77280c68c0 0x7f77281586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40254/0 (socket says 172.21.15.6:40254) 2024-03-22T13:00:47.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.236+0000 7f772d800640 1 -- 172.21.15.6:0/3842475293 learned_addr learned my addr 172.21.15.6:0/3842475293 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:47.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.240+0000 7f772d800640 1 -- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7728154240 msgr2=0x7f772814a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:47.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.240+0000 7f772d800640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7728154240 0x7f772814a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.240+0000 7f772d800640 1 -- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77280c7ee0 msgr2=0x7f772814a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:47.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.240+0000 7f772d800640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77280c7ee0 0x7f772814a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.240+0000 7f772d800640 1 -- 172.21.15.6:0/3842475293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7728159670 con 0x7f77280c68c0 2024-03-22T13:00:47.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.242+0000 7f772d800640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77280c68c0 0x7f77281586b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f771c00e870 tx=0x7f771c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:47.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.247+0000 7f7725600640 1 -- 172.21.15.6:0/3842475293 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c017070 con 0x7f77280c68c0 2024-03-22T13:00:47.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.247+0000 7f7725600640 1 -- 172.21.15.6:0/3842475293 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c00c940 con 0x7f77280c68c0 2024-03-22T13:00:47.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.247+0000 7f7725600640 1 -- 172.21.15.6:0/3842475293 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c00cd30 con 0x7f77280c68c0 2024-03-22T13:00:47.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.247+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7728159950 con 0x7f77280c68c0 2024-03-22T13:00:47.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.247+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7728159f10 con 0x7f77280c68c0 2024-03-22T13:00:47.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.249+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77280756c0 con 0x7f77280c68c0 2024-03-22T13:00:47.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.274+0000 7f7725600640 1 -- 172.21.15.6:0/3842475293 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f771c01a070 con 0x7f77280c68c0 2024-03-22T13:00:47.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.275+0000 7f7725600640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f770806bc20 0x7f770806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:47.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.276+0000 7f772ce00640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f770806bc20 0x7f770806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:47.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.277+0000 7f772ce00640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f770806bc20 0x7f770806e0e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7714036000 tx=0x7f7714008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:47.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.280+0000 7f7725600640 1 -- 172.21.15.6:0/3842475293 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f771c0575c0 con 0x7f77280c68c0 2024-03-22T13:00:47.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.590+0000 7f7725600640 1 -- 172.21.15.6:0/3842475293 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f771c059520 con 0x7f77280c68c0 2024-03-22T13:00:47.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.834+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77280c7ab0 con 0x7f77280c68c0 2024-03-22T13:00:47.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.856+0000 7f7725600640 1 -- 172.21.15.6:0/3842475293 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f771c05d5c0 con 0x7f77280c68c0 2024-03-22T13:00:47.856 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:47.856 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:47.857 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:47.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f770806bc20 msgr2=0x7f770806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:47.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f770806bc20 0x7f770806e0e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7714036000 tx=0x7f7714008030 comp rx=0 tx=0).stop 2024-03-22T13:00:47.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77280c68c0 msgr2=0x7f77281586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:47.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77280c68c0 0x7f77281586b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f771c00e870 tx=0x7f771c00ed30 comp rx=0 tx=0).stop 2024-03-22T13:00:47.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 shutdown_connections 2024-03-22T13:00:47.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7728154240 0x7f772814a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f770806bc20 0x7f770806e0e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77280c7ee0 0x7f772814a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 --2- 172.21.15.6:0/3842475293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77280c68c0 0x7f77281586b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 >> 172.21.15.6:0/3842475293 conn(0x7f77280cb930 msgr2=0x7f77280ca590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 shutdown_connections 2024-03-22T13:00:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:47.859+0000 7f7730200640 1 -- 172.21.15.6:0/3842475293 wait complete. 2024-03-22T13:00:48.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:48.481 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:49.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.678+0000 7fce66a00640 1 Processor -- start 2024-03-22T13:00:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.678+0000 7fce66a00640 1 -- start start 2024-03-22T13:00:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.680+0000 7fce66a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce60154240 0x7fce6015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.680+0000 7fce66a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce600c7ee0 0x7fce600c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.680+0000 7fce66a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce600c68c0 0x7fce6006b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.680+0000 7fce66a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6015e130 con 0x7fce600c68c0 2024-03-22T13:00:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.680+0000 7fce66a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce60154620 con 0x7fce600c7ee0 2024-03-22T13:00:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.680+0000 7fce66a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce600c6e00 con 0x7fce60154240 2024-03-22T13:00:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.681+0000 7fce5fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce60154240 0x7fce6015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.681+0000 7fce5f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce600c7ee0 0x7fce600c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.681+0000 7fce64a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce600c68c0 0x7fce6006b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.681+0000 7fce5fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce60154240 0x7fce6015ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44286/0 (socket says 172.21.15.6:44286) 2024-03-22T13:00:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.681+0000 7fce5fe00640 1 -- 172.21.15.6:0/3533583374 learned_addr learned my addr 172.21.15.6:0/3533583374 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.684+0000 7fce5fe00640 1 -- 172.21.15.6:0/3533583374 >> v2:172.21.15.92:3300/0 conn(0x7fce600c7ee0 msgr2=0x7fce600c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.684+0000 7fce5fe00640 1 --2- 172.21.15.6:0/3533583374 >> v2:172.21.15.92:3300/0 conn(0x7fce600c7ee0 0x7fce600c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.684+0000 7fce5fe00640 1 -- 172.21.15.6:0/3533583374 >> v2:172.21.15.6:3300/0 conn(0x7fce600c68c0 msgr2=0x7fce6006b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.684+0000 7fce5fe00640 1 --2- 172.21.15.6:0/3533583374 >> v2:172.21.15.6:3300/0 conn(0x7fce600c68c0 0x7fce6006b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.684+0000 7fce5fe00640 1 -- 172.21.15.6:0/3533583374 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600c70e0 con 0x7fce60154240 2024-03-22T13:00:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.688+0000 7fce5fe00640 1 --2- 172.21.15.6:0/3533583374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce60154240 0x7fce6015ac90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fce600713c0 tx=0x7fce5002b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca5ccf987d126ba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:49.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.693+0000 7fce5ea00640 1 -- 172.21.15.6:0/3533583374 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50034070 con 0x7fce60154240 2024-03-22T13:00:49.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.693+0000 7fce5ea00640 1 -- 172.21.15.6:0/3533583374 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce50030860 con 0x7fce60154240 2024-03-22T13:00:49.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.693+0000 7fce5ea00640 1 -- 172.21.15.6:0/3533583374 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50030c30 con 0x7fce60154240 2024-03-22T13:00:49.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.693+0000 7fce66a00640 1 -- 172.21.15.6:0/3533583374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce60154240 msgr2=0x7fce6015ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:49.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.693+0000 7fce66a00640 1 --2- 172.21.15.6:0/3533583374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce60154240 0x7fce6015ac90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fce600713c0 tx=0x7fce5002b9a0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.694+0000 7fce66a00640 1 -- 172.21.15.6:0/3533583374 shutdown_connections 2024-03-22T13:00:49.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.694+0000 7fce66a00640 1 --2- 172.21.15.6:0/3533583374 >> v2:172.21.15.6:3300/0 conn(0x7fce600c68c0 0x7fce6006b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.694+0000 7fce66a00640 1 --2- 172.21.15.6:0/3533583374 >> v2:172.21.15.92:3300/0 conn(0x7fce600c7ee0 0x7fce600c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.694+0000 7fce66a00640 1 --2- 172.21.15.6:0/3533583374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce60154240 0x7fce6015ac90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.694+0000 7fce66a00640 1 -- 172.21.15.6:0/3533583374 >> 172.21.15.6:0/3533583374 conn(0x7fce600cb930 msgr2=0x7fce600cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:49.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.694+0000 7fce66a00640 1 -- 172.21.15.6:0/3533583374 shutdown_connections 2024-03-22T13:00:49.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.694+0000 7fce66a00640 1 -- 172.21.15.6:0/3533583374 wait complete. 2024-03-22T13:00:49.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.695+0000 7fce66a00640 1 Processor -- start 2024-03-22T13:00:49.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.696+0000 7fce66a00640 1 -- start start 2024-03-22T13:00:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.696+0000 7fce66a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce600c68c0 0x7fce6015c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.696+0000 7fce66a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce600c7ee0 0x7fce6006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.696+0000 7fce66a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce60154240 0x7fce60156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.696+0000 7fce66a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6015ddc0 con 0x7fce60154240 2024-03-22T13:00:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.696+0000 7fce66a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6015dae0 con 0x7fce600c7ee0 2024-03-22T13:00:49.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.696+0000 7fce66a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6015e5c0 con 0x7fce600c68c0 2024-03-22T13:00:49.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.697+0000 7fce5fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce600c68c0 0x7fce6015c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:49.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.697+0000 7fce64a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce60154240 0x7fce60156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:49.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.697+0000 7fce5f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce600c7ee0 0x7fce6006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:49.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.698+0000 7fce5fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce600c68c0 0x7fce6015c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44292/0 (socket says 172.21.15.6:44292) 2024-03-22T13:00:49.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.698+0000 7fce5fe00640 1 -- 172.21.15.6:0/1079376935 learned_addr learned my addr 172.21.15.6:0/1079376935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.701+0000 7fce64a00640 1 -- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce600c68c0 msgr2=0x7fce6015c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.701+0000 7fce64a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce600c68c0 0x7fce6015c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.701+0000 7fce64a00640 1 -- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce600c7ee0 msgr2=0x7fce6006f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.701+0000 7fce64a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce600c7ee0 0x7fce6006f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.701+0000 7fce64a00640 1 -- 172.21.15.6:0/1079376935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce601e4510 con 0x7fce60154240 2024-03-22T13:00:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.704+0000 7fce64a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce60154240 0x7fce60156a90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce5800ef00 tx=0x7fce5800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.708+0000 7fce57e00640 1 -- 172.21.15.6:0/1079376935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58018020 con 0x7fce60154240 2024-03-22T13:00:49.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.709+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce601e5640 con 0x7fce60154240 2024-03-22T13:00:49.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.709+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce601e3750 con 0x7fce60154240 2024-03-22T13:00:49.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.710+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce60067ae0 con 0x7fce60154240 2024-03-22T13:00:49.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:49.930+0000 7fce57e00640 1 -- 172.21.15.6:0/1079376935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5801d400 con 0x7fce60154240 2024-03-22T13:00:50.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.042+0000 7fce57e00640 1 -- 172.21.15.6:0/1079376935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58009e60 con 0x7fce60154240 2024-03-22T13:00:50.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.044+0000 7fce57e00640 1 -- 172.21.15.6:0/1079376935 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fce58005d10 con 0x7fce60154240 2024-03-22T13:00:50.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.046+0000 7fce57e00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce4006bbd0 0x7fce4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:50.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.046+0000 7fce5fe00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce4006bbd0 0x7fce4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:50.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.048+0000 7fce5fe00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce4006bbd0 0x7fce4006e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fce50009630 tx=0x7fce5003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:50.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.074+0000 7fce57e00640 1 -- 172.21.15.6:0/1079376935 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fce58057da0 con 0x7fce60154240 2024-03-22T13:00:50.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.075+0000 7fce57e00640 1 -- 172.21.15.6:0/1079376935 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fce60067ae0 con 0x7fce60154240 2024-03-22T13:00:50.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.336+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce600756c0 con 0x7fce60154240 2024-03-22T13:00:50.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.358+0000 7fce57e00640 1 -- 172.21.15.6:0/1079376935 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fce58022030 con 0x7fce60154240 2024-03-22T13:00:50.358 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:50.358 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:50.359 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce4006bbd0 msgr2=0x7fce4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce4006bbd0 0x7fce4006e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fce50009630 tx=0x7fce5003b040 comp rx=0 tx=0).stop 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce60154240 msgr2=0x7fce60156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce60154240 0x7fce60156a90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce5800ef00 tx=0x7fce5800b480 comp rx=0 tx=0).stop 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 shutdown_connections 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce60154240 0x7fce60156a90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce4006bbd0 0x7fce4006e090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce600c7ee0 0x7fce6006f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 --2- 172.21.15.6:0/1079376935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce600c68c0 0x7fce6015c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 >> 172.21.15.6:0/1079376935 conn(0x7fce600cb930 msgr2=0x7fce600c48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 shutdown_connections 2024-03-22T13:00:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:50.361+0000 7fce66a00640 1 -- 172.21.15.6:0/1079376935 wait complete. 2024-03-22T13:00:50.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:50.983 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:51.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.181+0000 7f6755400640 1 Processor -- start 2024-03-22T13:00:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.181+0000 7f6755400640 1 -- start start 2024-03-22T13:00:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.183+0000 7f6755400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67500cc380 0x7f675006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.183+0000 7f6755400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500ce8a0 0x7f675015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.183+0000 7f6755400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500cee00 con 0x7f67500cc760 2024-03-22T13:00:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.183+0000 7f6755400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67500cf0c0 con 0x7f67500cc380 2024-03-22T13:00:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.183+0000 7f6755400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67500cf3a0 con 0x7f67500ce8a0 2024-03-22T13:00:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.184+0000 7f6747400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500ce8a0 0x7f675015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.184+0000 7f674fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67500cc380 0x7f675006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.184+0000 7f6747400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500ce8a0 0x7f675015ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44308/0 (socket says 172.21.15.6:44308) 2024-03-22T13:00:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.184+0000 7f6747400640 1 -- 172.21.15.6:0/197689753 learned_addr learned my addr 172.21.15.6:0/197689753 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.186+0000 7f6747400640 1 -- 172.21.15.6:0/197689753 >> v2:172.21.15.6:3300/0 conn(0x7f67500cc380 msgr2=0x7f675006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.186+0000 7f6747400640 1 --2- 172.21.15.6:0/197689753 >> v2:172.21.15.6:3300/0 conn(0x7f67500cc380 0x7f675006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.186+0000 7f6747400640 1 -- 172.21.15.6:0/197689753 >> v1:172.21.15.6:6789/0 conn(0x7f67500cc760 legacy=0x7f67500cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.186+0000 7f6747400640 1 -- 172.21.15.6:0/197689753 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500cf680 con 0x7f67500ce8a0 2024-03-22T13:00:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.189+0000 7f6747400640 1 --2- 172.21.15.6:0/197689753 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500ce8a0 0x7f675015ae30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6738009870 tx=0x7f673802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a17ee4d89ff237 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.193+0000 7f674f400640 1 -- 172.21.15.6:0/197689753 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738034070 con 0x7f67500ce8a0 2024-03-22T13:00:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.193+0000 7f674f400640 1 -- 172.21.15.6:0/197689753 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738030840 con 0x7f67500ce8a0 2024-03-22T13:00:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.193+0000 7f674f400640 1 -- 172.21.15.6:0/197689753 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738030c10 con 0x7f67500ce8a0 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 -- 172.21.15.6:0/197689753 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500ce8a0 msgr2=0x7f675015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 --2- 172.21.15.6:0/197689753 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500ce8a0 0x7f675015ae30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6738009870 tx=0x7f673802b980 comp rx=0 tx=0).stop 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 -- 172.21.15.6:0/197689753 shutdown_connections 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 --2- 172.21.15.6:0/197689753 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500ce8a0 0x7f675015ae30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 --2- 172.21.15.6:0/197689753 >> v2:172.21.15.6:3300/0 conn(0x7f67500cc380 0x7f675006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 -- 172.21.15.6:0/197689753 >> 172.21.15.6:0/197689753 conn(0x7f67500b1480 msgr2=0x7f67500b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 -- 172.21.15.6:0/197689753 shutdown_connections 2024-03-22T13:00:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.194+0000 7f6755400640 1 -- 172.21.15.6:0/197689753 wait complete. 2024-03-22T13:00:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.196+0000 7f6755400640 1 Processor -- start 2024-03-22T13:00:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.196+0000 7f6755400640 1 -- start start 2024-03-22T13:00:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.197+0000 7f6755400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500cc380 0x7f675015b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.197+0000 7f6755400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67500cc760 0x7f67500c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.197+0000 7f6755400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67500ce8a0 0x7f67500c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.197+0000 7f6755400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67501549d0 con 0x7f67500ce8a0 2024-03-22T13:00:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.197+0000 7f6755400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675015a910 con 0x7f67500cc760 2024-03-22T13:00:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.197+0000 7f6755400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67500ccb90 con 0x7f67500cc380 2024-03-22T13:00:52.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.198+0000 7f6747400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67500cc760 0x7f67500c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:52.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.198+0000 7f6754a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67500ce8a0 0x7f67500c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:52.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.198+0000 7f674fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500cc380 0x7f675015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:52.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.199+0000 7f6754a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67500ce8a0 0x7f67500c84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40290/0 (socket says 172.21.15.6:40290) 2024-03-22T13:00:52.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.199+0000 7f6754a00640 1 -- 172.21.15.6:0/1476077941 learned_addr learned my addr 172.21.15.6:0/1476077941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:52.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.202+0000 7f6747400640 1 -- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500cc380 msgr2=0x7f675015b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:52.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.202+0000 7f6747400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500cc380 0x7f675015b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.202+0000 7f6747400640 1 -- 172.21.15.6:0/1476077941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67500ce8a0 msgr2=0x7f67500c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:52.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.202+0000 7f6747400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67500ce8a0 0x7f67500c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.202+0000 7f6747400640 1 -- 172.21.15.6:0/1476077941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67501f0200 con 0x7f67500cc760 2024-03-22T13:00:52.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.205+0000 7f6747400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67500cc760 0x7f67500c7fb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6738033000 tx=0x7f6738004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:52.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.209+0000 7f674cc00640 1 -- 172.21.15.6:0/1476077941 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738048070 con 0x7f67500cc760 2024-03-22T13:00:52.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.209+0000 7f674cc00640 1 -- 172.21.15.6:0/1476077941 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738030640 con 0x7f67500cc760 2024-03-22T13:00:52.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.209+0000 7f674cc00640 1 -- 172.21.15.6:0/1476077941 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738034040 con 0x7f67500cc760 2024-03-22T13:00:52.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.209+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67501f04e0 con 0x7f67500cc760 2024-03-22T13:00:52.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.209+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67501f0aa0 con 0x7f67500cc760 2024-03-22T13:00:52.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.211+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6750075650 con 0x7f67500cc760 2024-03-22T13:00:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.497+0000 7f674cc00640 1 -- 172.21.15.6:0/1476077941 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f673804b070 con 0x7f67500cc760 2024-03-22T13:00:52.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.500+0000 7f674cc00640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f672c06bc20 0x7f672c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.500+0000 7f674fe00640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f672c06bc20 0x7f672c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.501+0000 7f674fe00640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f672c06bc20 0x7f672c06e0e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6740009aa0 tx=0x7f6740009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:52.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.503+0000 7f674cc00640 1 -- 172.21.15.6:0/1476077941 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f67380307f0 con 0x7f67500cc760 2024-03-22T13:00:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.520+0000 7f674cc00640 1 -- 172.21.15.6:0/1476077941 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6738052020 con 0x7f67500cc760 2024-03-22T13:00:52.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.785+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67500c9d30 con 0x7f67500cc760 2024-03-22T13:00:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.799+0000 7f674cc00640 1 -- 172.21.15.6:0/1476077941 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f673807b590 con 0x7f67500cc760 2024-03-22T13:00:52.799 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:52.799 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f672c06bc20 msgr2=0x7f672c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f672c06bc20 0x7f672c06e0e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6740009aa0 tx=0x7f6740009190 comp rx=0 tx=0).stop 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67500cc760 msgr2=0x7f67500c7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67500cc760 0x7f67500c7fb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6738033000 tx=0x7f6738004730 comp rx=0 tx=0).stop 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 shutdown_connections 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67500ce8a0 0x7f67500c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f672c06bc20 0x7f672c06e0e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67500cc760 0x7f67500c7fb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 --2- 172.21.15.6:0/1476077941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67500cc380 0x7f675015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 >> 172.21.15.6:0/1476077941 conn(0x7f67500b1480 msgr2=0x7f67501e55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 shutdown_connections 2024-03-22T13:00:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:52.802+0000 7f6755400640 1 -- 172.21.15.6:0/1476077941 wait complete. 2024-03-22T13:00:53.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:53.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:54.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:54.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.633+0000 7f62c4c00640 1 Processor -- start 2024-03-22T13:00:54.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.633+0000 7f62c4c00640 1 -- start start 2024-03-22T13:00:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.635+0000 7f62c4c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c00c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.635+0000 7f62c4c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c01590b0 con 0x7f62c0152400 2024-03-22T13:00:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.635+0000 7f62c4c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c00721f0 con 0x7f62c006f8c0 2024-03-22T13:00:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.635+0000 7f62c4c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00724d0 con 0x7f62c00c5040 2024-03-22T13:00:54.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.636+0000 7f62be800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c00c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:54.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.636+0000 7f62b7e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f62c006f8c0 0x7f62c00716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:39292/0 (socket says 172.21.15.6:39292) 2024-03-22T13:00:54.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.636+0000 7f62b7e00640 1 -- 172.21.15.6:0/109208737 learned_addr learned my addr 172.21.15.6:0/109208737 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:54.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.640+0000 7f62be800640 1 -- 172.21.15.6:0/109208737 >> v1:172.21.15.92:6789/0 conn(0x7f62c006f8c0 legacy=0x7f62c00716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:54.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.640+0000 7f62be800640 1 -- 172.21.15.6:0/109208737 >> v1:172.21.15.6:6789/0 conn(0x7f62c0152400 legacy=0x7f62c015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:54.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.640+0000 7f62be800640 1 -- 172.21.15.6:0/109208737 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c01582f0 con 0x7f62c00c5040 2024-03-22T13:00:54.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.643+0000 7f62be800640 1 --2- 172.21.15.6:0/109208737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c00c5450 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009dc0 tx=0x7f62b802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a394c022675ff3f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.646+0000 7f62bd400640 1 -- 172.21.15.6:0/109208737 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8039070 con 0x7f62c00c5040 2024-03-22T13:00:54.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.648+0000 7f62bd400640 1 -- 172.21.15.6:0/109208737 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b802ccd0 con 0x7f62c00c5040 2024-03-22T13:00:54.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.648+0000 7f62bd400640 1 -- 172.21.15.6:0/109208737 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8034660 con 0x7f62c00c5040 2024-03-22T13:00:54.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.649+0000 7f62c4c00640 1 -- 172.21.15.6:0/109208737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 msgr2=0x7f62c00c5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:54.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.649+0000 7f62c4c00640 1 --2- 172.21.15.6:0/109208737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c00c5450 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009dc0 tx=0x7f62b802bf20 comp rx=0 tx=0).stop 2024-03-22T13:00:54.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.649+0000 7f62c4c00640 1 -- 172.21.15.6:0/109208737 shutdown_connections 2024-03-22T13:00:54.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.649+0000 7f62c4c00640 1 --2- 172.21.15.6:0/109208737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c00c5450 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:54.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.649+0000 7f62c4c00640 1 -- 172.21.15.6:0/109208737 >> 172.21.15.6:0/109208737 conn(0x7f62c00cb930 msgr2=0x7f62c00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:54.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.649+0000 7f62c4c00640 1 -- 172.21.15.6:0/109208737 shutdown_connections 2024-03-22T13:00:54.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.649+0000 7f62c4c00640 1 -- 172.21.15.6:0/109208737 wait complete. 2024-03-22T13:00:54.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.651+0000 7f62c4c00640 1 Processor -- start 2024-03-22T13:00:54.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.651+0000 7f62c4c00640 1 -- start start 2024-03-22T13:00:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.652+0000 7f62c4c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62c006f8c0 0x7f62c006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.652+0000 7f62c4c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c01455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.652+0000 7f62c4c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62c0152400 0x7f62c0145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:54.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.652+0000 7f62c4c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0158b90 con 0x7f62c0152400 2024-03-22T13:00:54.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.652+0000 7f62c4c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c01588b0 con 0x7f62c006f8c0 2024-03-22T13:00:54.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.652+0000 7f62c4c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c01527e0 con 0x7f62c00c5040 2024-03-22T13:00:54.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.653+0000 7f62b7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c01455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:54.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.653+0000 7f62bde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62c006f8c0 0x7f62c006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:54.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.654+0000 7f62b7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c01455a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57630/0 (socket says 172.21.15.6:57630) 2024-03-22T13:00:54.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.654+0000 7f62b7e00640 1 -- 172.21.15.6:0/964061172 learned_addr learned my addr 172.21.15.6:0/964061172 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.657+0000 7f62bde00640 1 -- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 msgr2=0x7f62c01455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.657+0000 7f62bde00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c01455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.657+0000 7f62bde00640 1 -- 172.21.15.6:0/964061172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62c0152400 msgr2=0x7f62c0145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.657+0000 7f62bde00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62c0152400 0x7f62c0145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.657+0000 7f62bde00640 1 -- 172.21.15.6:0/964061172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c0147080 con 0x7f62c006f8c0 2024-03-22T13:00:54.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.660+0000 7f62bde00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62c006f8c0 0x7f62c006b2f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f62b000b3e0 tx=0x7f62b000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.665+0000 7f62b6000640 1 -- 172.21.15.6:0/964061172 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0015020 con 0x7f62c006f8c0 2024-03-22T13:00:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.665+0000 7f62b6000640 1 -- 172.21.15.6:0/964061172 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0011830 con 0x7f62c006f8c0 2024-03-22T13:00:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.665+0000 7f62b6000640 1 -- 172.21.15.6:0/964061172 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0011c20 con 0x7f62c006f8c0 2024-03-22T13:00:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.665+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c0143bd0 con 0x7f62c006f8c0 2024-03-22T13:00:54.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.665+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c006b830 con 0x7f62c006f8c0 2024-03-22T13:00:54.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.666+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c0072d10 con 0x7f62c006f8c0 2024-03-22T13:00:54.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.687+0000 7f62b6000640 1 -- 172.21.15.6:0/964061172 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f62b0018020 con 0x7f62c006f8c0 2024-03-22T13:00:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.688+0000 7f62b6000640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f629c06bc50 0x7f629c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:54.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.689+0000 7f62b7e00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f629c06bc50 0x7f629c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:54.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.689+0000 7f62b7e00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f629c06bc50 0x7f629c06e110 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62c0147830 tx=0x7f62a8007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:54.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.691+0000 7f62b6000640 1 -- 172.21.15.6:0/964061172 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f62b0056ea0 con 0x7f62c006f8c0 2024-03-22T13:00:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:54.943+0000 7f62b6000640 1 -- 172.21.15.6:0/964061172 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f62c0072d10 con 0x7f62c006f8c0 2024-03-22T13:00:55.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.193+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62c00756c0 con 0x7f62c006f8c0 2024-03-22T13:00:55.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.217+0000 7f62b6000640 1 -- 172.21.15.6:0/964061172 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f62b001f070 con 0x7f62c006f8c0 2024-03-22T13:00:55.217 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:55.217 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:55.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.220+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f629c06bc50 msgr2=0x7f629c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.220+0000 7f62c4c00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f629c06bc50 0x7f629c06e110 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62c0147830 tx=0x7f62a8007f10 comp rx=0 tx=0).stop 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.220+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62c006f8c0 msgr2=0x7f62c006b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.220+0000 7f62c4c00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62c006f8c0 0x7f62c006b2f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f62b000b3e0 tx=0x7f62b000b8a0 comp rx=0 tx=0).stop 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 shutdown_connections 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62c0152400 0x7f62c0145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62c00c5040 0x7f62c01455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f629c06bc50 0x7f629c06e110 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 --2- 172.21.15.6:0/964061172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62c006f8c0 0x7f62c006b2f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 >> 172.21.15.6:0/964061172 conn(0x7f62c00cb930 msgr2=0x7f62c00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 shutdown_connections 2024-03-22T13:00:55.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:55.221+0000 7f62c4c00640 1 -- 172.21.15.6:0/964061172 wait complete. 2024-03-22T13:00:55.842 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:55.843 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:56.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:57.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.040+0000 7f9bb7a00640 1 Processor -- start 2024-03-22T13:00:57.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.040+0000 7f9bb7a00640 1 -- start start 2024-03-22T13:00:57.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.042+0000 7f9bb7a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bb00cd300 0x7f9bb015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:57.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.042+0000 7f9bb7a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:57.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.042+0000 7f9bb7a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb00cc380 con 0x7f9bb0154980 2024-03-22T13:00:57.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.042+0000 7f9bb7a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb015a520 con 0x7f9bb00cd300 2024-03-22T13:00:57.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.042+0000 7f9bb7a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb015b3a0 con 0x7f9bb015aef0 2024-03-22T13:00:57.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.043+0000 7f9bb5000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9bb0154980 0x7f9bb00ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55306/0 (socket says 172.21.15.6:55306) 2024-03-22T13:00:57.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.043+0000 7f9bb5000640 1 -- 172.21.15.6:0/855120502 learned_addr learned my addr 172.21.15.6:0/855120502 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:57.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.043+0000 7f9bafe00640 1 --2- 172.21.15.6:0/855120502 >> v2:172.21.15.92:3300/0 conn(0x7f9bb00cd300 0x7f9bb015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:57.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.043+0000 7f9bb5a00640 1 --2- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:57.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.046+0000 7f9baf400640 1 -- 172.21.15.6:0/855120502 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220580170 0 0) 0x7f9bb00cc380 con 0x7f9bb0154980 2024-03-22T13:00:57.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.046+0000 7f9bafe00640 1 -- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 msgr2=0x7f9bb00ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:57.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.047+0000 7f9bafe00640 1 --2- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.047+0000 7f9bafe00640 1 -- 172.21.15.6:0/855120502 >> v1:172.21.15.6:6789/0 conn(0x7f9bb0154980 legacy=0x7f9bb00ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:57.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.047+0000 7f9bafe00640 1 -- 172.21.15.6:0/855120502 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb00cc660 con 0x7f9bb00cd300 2024-03-22T13:00:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.049+0000 7f9bafe00640 1 --2- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb015a9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0097a0 tx=0x7f9b9c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcc3180925b5422f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.053+0000 7f9baf400640 1 -- 172.21.15.6:0/855120502 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c035020 con 0x7f9bb00cd300 2024-03-22T13:00:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.053+0000 7f9baf400640 1 -- 172.21.15.6:0/855120502 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c031830 con 0x7f9bb00cd300 2024-03-22T13:00:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.053+0000 7f9baf400640 1 -- 172.21.15.6:0/855120502 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c031c00 con 0x7f9bb00cd300 2024-03-22T13:00:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.054+0000 7f9bb7a00640 1 -- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 msgr2=0x7f9bb015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.054+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb015a9b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0097a0 tx=0x7f9b9c02b8d0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.054+0000 7f9bb7a00640 1 -- 172.21.15.6:0/855120502 shutdown_connections 2024-03-22T13:00:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.054+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.054+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/855120502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb015a9b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.054+0000 7f9bb7a00640 1 -- 172.21.15.6:0/855120502 >> 172.21.15.6:0/855120502 conn(0x7f9bb00b16c0 msgr2=0x7f9bb00b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.055+0000 7f9bb7a00640 1 -- 172.21.15.6:0/855120502 shutdown_connections 2024-03-22T13:00:57.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.055+0000 7f9bb7a00640 1 -- 172.21.15.6:0/855120502 wait complete. 2024-03-22T13:00:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.057+0000 7f9bb7a00640 1 Processor -- start 2024-03-22T13:00:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.057+0000 7f9bb7a00640 1 -- start start 2024-03-22T13:00:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.059+0000 7f9bb7a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb0159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.059+0000 7f9bb7a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb0154980 0x7f9bb01f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.059+0000 7f9bb7a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.059+0000 7f9bb7a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00ccf00 con 0x7f9bb0154980 2024-03-22T13:00:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.059+0000 7f9bb7a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00ccc20 con 0x7f9bb00cd300 2024-03-22T13:00:57.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.059+0000 7f9bb7a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00cea10 con 0x7f9bb015aef0 2024-03-22T13:00:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.060+0000 7f9bb5a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.060+0000 7f9bb5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb0159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.060+0000 7f9bafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb0154980 0x7f9bb01f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.060+0000 7f9bb5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb0159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58834/0 (socket says 172.21.15.6:58834) 2024-03-22T13:00:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.060+0000 7f9bb5000640 1 -- 172.21.15.6:0/380909821 learned_addr learned my addr 172.21.15.6:0/380909821 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.064+0000 7f9bb5000640 1 -- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 msgr2=0x7f9bb00b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.064+0000 7f9bb5000640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.064+0000 7f9bb5000640 1 -- 172.21.15.6:0/380909821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb0154980 msgr2=0x7f9bb01f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.064+0000 7f9bb5000640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb0154980 0x7f9bb01f3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.064+0000 7f9bb5000640 1 -- 172.21.15.6:0/380909821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb01f0060 con 0x7f9bb00cd300 2024-03-22T13:00:57.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.067+0000 7f9bb5000640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb0159d00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c4d0 tx=0x7f9ba400c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.072+0000 7f9bacc00640 1 -- 172.21.15.6:0/380909821 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4018070 con 0x7f9bb00cd300 2024-03-22T13:00:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.072+0000 7f9bacc00640 1 -- 172.21.15.6:0/380909821 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba40074e0 con 0x7f9bb00cd300 2024-03-22T13:00:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.072+0000 7f9bacc00640 1 -- 172.21.15.6:0/380909821 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4004a20 con 0x7f9bb00cd300 2024-03-22T13:00:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.072+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb01f0340 con 0x7f9bb00cd300 2024-03-22T13:00:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.072+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb01f0900 con 0x7f9bb00cd300 2024-03-22T13:00:57.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.073+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb00756c0 con 0x7f9bb00cd300 2024-03-22T13:00:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.108+0000 7f9bacc00640 1 -- 172.21.15.6:0/380909821 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ba401b070 con 0x7f9bb00cd300 2024-03-22T13:00:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.109+0000 7f9bacc00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b9006bc20 0x7f9b9006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.110+0000 7f9bafe00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b9006bc20 0x7f9b9006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:57.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.111+0000 7f9bafe00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b9006bc20 0x7f9b9006e0e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c036000 tx=0x7f9b9c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:57.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.352+0000 7f9bacc00640 1 -- 172.21.15.6:0/380909821 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9ba4056790 con 0x7f9bb00cd300 2024-03-22T13:00:57.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.371+0000 7f9bacc00640 1 -- 172.21.15.6:0/380909821 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9ba4022020 con 0x7f9bb00cd300 2024-03-22T13:00:57.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.616+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bb00cb5a0 con 0x7f9bb00cd300 2024-03-22T13:00:57.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.631+0000 7f9bacc00640 1 -- 172.21.15.6:0/380909821 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f9ba40586f0 con 0x7f9bb00cd300 2024-03-22T13:00:57.631 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:00:57.632 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:00:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:00:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b9006bc20 msgr2=0x7f9b9006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b9006bc20 0x7f9b9006e0e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c036000 tx=0x7f9b9c008030 comp rx=0 tx=0).stop 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 msgr2=0x7f9bb0159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb0159d00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400c4d0 tx=0x7f9ba400c990 comp rx=0 tx=0).stop 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 shutdown_connections 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb015aef0 0x7f9bb00b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb0154980 0x7f9bb01f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b9006bc20 0x7f9b9006e0e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 --2- 172.21.15.6:0/380909821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb00cd300 0x7f9bb0159d00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.634+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 >> 172.21.15.6:0/380909821 conn(0x7f9bb00b16c0 msgr2=0x7f9bb0151da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.635+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 shutdown_connections 2024-03-22T13:00:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:57.635+0000 7f9bb7a00640 1 -- 172.21.15.6:0/380909821 wait complete. 2024-03-22T13:00:58.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:00:58.255 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:00:59.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:00:59.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.459+0000 7ffb3d800640 1 Processor -- start 2024-03-22T13:00:59.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.459+0000 7ffb3d800640 1 -- start start 2024-03-22T13:00:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.460+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb38152400 0x7ffb3815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.460+0000 7ffb3d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb3806f8c0 0x7ffb380c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.460+0000 7ffb3d800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb381590b0 con 0x7ffb3815bc80 2024-03-22T13:00:59.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.461+0000 7ffb3d800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb380724d0 con 0x7ffb3806f8c0 2024-03-22T13:00:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.461+0000 7ffb3d800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380721f0 con 0x7ffb38152400 2024-03-22T13:00:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.461+0000 7ffb36a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb38152400 0x7ffb3815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.461+0000 7ffb37400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb3806f8c0 0x7ffb380c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.462+0000 7ffb36000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ffb3815bc80 0x7ffb380716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:39298/0 (socket says 172.21.15.6:39298) 2024-03-22T13:00:59.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.462+0000 7ffb36000640 1 -- 172.21.15.6:0/3768750604 learned_addr learned my addr 172.21.15.6:0/3768750604 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:00:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.464+0000 7ffb36a00640 1 -- 172.21.15.6:0/3768750604 >> v2:172.21.15.6:3300/0 conn(0x7ffb3806f8c0 msgr2=0x7ffb380c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:00:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.464+0000 7ffb36a00640 1 --2- 172.21.15.6:0/3768750604 >> v2:172.21.15.6:3300/0 conn(0x7ffb3806f8c0 0x7ffb380c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.464+0000 7ffb36a00640 1 -- 172.21.15.6:0/3768750604 >> v1:172.21.15.92:6789/0 conn(0x7ffb3815bc80 legacy=0x7ffb380716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:59.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.464+0000 7ffb36a00640 1 -- 172.21.15.6:0/3768750604 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb381582f0 con 0x7ffb38152400 2024-03-22T13:00:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.467+0000 7ffb36a00640 1 --2- 172.21.15.6:0/3768750604 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb38152400 0x7ffb3815b740 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009d90 tx=0x7ffb2802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa583a3ae34fc1fa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.473+0000 7ffb35600640 1 -- 172.21.15.6:0/3768750604 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28039070 con 0x7ffb38152400 2024-03-22T13:00:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.473+0000 7ffb35600640 1 -- 172.21.15.6:0/3768750604 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2802cce0 con 0x7ffb38152400 2024-03-22T13:00:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.473+0000 7ffb35600640 1 -- 172.21.15.6:0/3768750604 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28034660 con 0x7ffb38152400 2024-03-22T13:00:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.473+0000 7ffb3d800640 1 -- 172.21.15.6:0/3768750604 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb38152400 msgr2=0x7ffb3815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.473+0000 7ffb3d800640 1 --2- 172.21.15.6:0/3768750604 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb38152400 0x7ffb3815b740 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009d90 tx=0x7ffb2802bf30 comp rx=0 tx=0).stop 2024-03-22T13:00:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.474+0000 7ffb3d800640 1 -- 172.21.15.6:0/3768750604 shutdown_connections 2024-03-22T13:00:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.474+0000 7ffb3d800640 1 --2- 172.21.15.6:0/3768750604 >> v2:172.21.15.6:3300/0 conn(0x7ffb3806f8c0 0x7ffb380c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.474+0000 7ffb3d800640 1 --2- 172.21.15.6:0/3768750604 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb38152400 0x7ffb3815b740 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.474+0000 7ffb3d800640 1 -- 172.21.15.6:0/3768750604 >> 172.21.15.6:0/3768750604 conn(0x7ffb380cb930 msgr2=0x7ffb380cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:00:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.474+0000 7ffb3d800640 1 -- 172.21.15.6:0/3768750604 shutdown_connections 2024-03-22T13:00:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.474+0000 7ffb3d800640 1 -- 172.21.15.6:0/3768750604 wait complete. 2024-03-22T13:00:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.475+0000 7ffb3d800640 1 Processor -- start 2024-03-22T13:00:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.476+0000 7ffb3d800640 1 -- start start 2024-03-22T13:00:59.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3806f8c0 0x7ffb3814c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:59.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb38152400 0x7ffb3814fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:59.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815bc80 0x7ffb381ebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:59.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb3d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38158b90 con 0x7ffb3806f8c0 2024-03-22T13:00:59.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb3d800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb381588b0 con 0x7ffb38152400 2024-03-22T13:00:59.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb3d800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb381527e0 con 0x7ffb3815bc80 2024-03-22T13:00:59.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb37400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815bc80 0x7ffb381ebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:59.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb36a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3806f8c0 0x7ffb3814c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:59.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.477+0000 7ffb36000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb38152400 0x7ffb3814fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:59.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.478+0000 7ffb37400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815bc80 0x7ffb381ebaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57652/0 (socket says 172.21.15.6:57652) 2024-03-22T13:00:59.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.478+0000 7ffb37400640 1 -- 172.21.15.6:0/780188342 learned_addr learned my addr 172.21.15.6:0/780188342 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:00:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.481+0000 7ffb37400640 1 -- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb38152400 msgr2=0x7ffb3814fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.481+0000 7ffb37400640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb38152400 0x7ffb3814fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.481+0000 7ffb37400640 1 -- 172.21.15.6:0/780188342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3806f8c0 msgr2=0x7ffb3814c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:00:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.481+0000 7ffb37400640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3806f8c0 0x7ffb3814c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:00:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.481+0000 7ffb37400640 1 -- 172.21.15.6:0/780188342 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb38148310 con 0x7ffb3815bc80 2024-03-22T13:00:59.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.484+0000 7ffb37400640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815bc80 0x7ffb381ebaa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000e930 tx=0x7ffb3000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:59.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.488+0000 7ffb2ea00640 1 -- 172.21.15.6:0/780188342 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30017070 con 0x7ffb3815bc80 2024-03-22T13:00:59.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.488+0000 7ffb2ea00640 1 -- 172.21.15.6:0/780188342 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3000c980 con 0x7ffb3815bc80 2024-03-22T13:00:59.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.488+0000 7ffb2ea00640 1 -- 172.21.15.6:0/780188342 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000cd70 con 0x7ffb3815bc80 2024-03-22T13:00:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.488+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3814cdc0 con 0x7ffb3815bc80 2024-03-22T13:00:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.488+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb381ef2f0 con 0x7ffb3815bc80 2024-03-22T13:00:59.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.490+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb380756c0 con 0x7ffb3815bc80 2024-03-22T13:00:59.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.509+0000 7ffb2ea00640 1 -- 172.21.15.6:0/780188342 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffb3001a070 con 0x7ffb3815bc80 2024-03-22T13:00:59.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.510+0000 7ffb2ea00640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc50 0x7ffb1406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:00:59.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.511+0000 7ffb36a00640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc50 0x7ffb1406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:00:59.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.511+0000 7ffb36a00640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc50 0x7ffb1406e110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffb28036000 tx=0x7ffb28030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:00:59.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.512+0000 7ffb2ea00640 1 -- 172.21.15.6:0/780188342 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ffb300576e0 con 0x7ffb3815bc80 2024-03-22T13:00:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:00:59.768+0000 7ffb2ea00640 1 -- 172.21.15.6:0/780188342 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffb30059640 con 0x7ffb3815bc80 2024-03-22T13:01:00.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.040+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb38151490 con 0x7ffb3815bc80 2024-03-22T13:01:00.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.052+0000 7ffb2ea00640 1 -- 172.21.15.6:0/780188342 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7ffb3005d6e0 con 0x7ffb3815bc80 2024-03-22T13:01:00.052 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:00.052 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:00.053 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc50 msgr2=0x7ffb1406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc50 0x7ffb1406e110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffb28036000 tx=0x7ffb28030000 comp rx=0 tx=0).stop 2024-03-22T13:01:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815bc80 msgr2=0x7ffb381ebaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815bc80 0x7ffb381ebaa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000e930 tx=0x7ffb3000edf0 comp rx=0 tx=0).stop 2024-03-22T13:01:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 shutdown_connections 2024-03-22T13:01:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb3815bc80 0x7ffb381ebaa0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:00.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ffb1406bc50 0x7ffb1406e110 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:00.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb38152400 0x7ffb3814fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:00.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 --2- 172.21.15.6:0/780188342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3806f8c0 0x7ffb3814c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:00.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 >> 172.21.15.6:0/780188342 conn(0x7ffb380cb930 msgr2=0x7ffb380716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:00.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 shutdown_connections 2024-03-22T13:01:00.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:00.055+0000 7ffb3d800640 1 -- 172.21.15.6:0/780188342 wait complete. 2024-03-22T13:01:00.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:01:00.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:01:00.685 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:00.685 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:01.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.888+0000 7fad4c600640 1 Processor -- start 2024-03-22T13:01:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.889+0000 7fad4c600640 1 -- start start 2024-03-22T13:01:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.890+0000 7fad4c600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad44152400 0x7fad4415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.890+0000 7fad4c600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fad4415bc80 0x7fad440716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.890+0000 7fad4c600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad441590b0 con 0x7fad4406f8c0 2024-03-22T13:01:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.890+0000 7fad4c600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad440724d0 con 0x7fad4415bc80 2024-03-22T13:01:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.890+0000 7fad4c600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad440721f0 con 0x7fad44152400 2024-03-22T13:01:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.891+0000 7fad49c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad44152400 0x7fad4415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.891+0000 7fad49200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fad4415bc80 0x7fad440716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.892+0000 7fad49c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad44152400 0x7fad4415b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57664/0 (socket says 172.21.15.6:57664) 2024-03-22T13:01:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.892+0000 7fad49c00640 1 -- 172.21.15.6:0/2883643748 learned_addr learned my addr 172.21.15.6:0/2883643748 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.895+0000 7fad49c00640 1 -- 172.21.15.6:0/2883643748 >> v2:172.21.15.6:3300/0 conn(0x7fad4415bc80 msgr2=0x7fad440716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.895+0000 7fad49c00640 1 --2- 172.21.15.6:0/2883643748 >> v2:172.21.15.6:3300/0 conn(0x7fad4415bc80 0x7fad440716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.895+0000 7fad49c00640 1 -- 172.21.15.6:0/2883643748 >> v1:172.21.15.6:6789/0 conn(0x7fad4406f8c0 legacy=0x7fad440c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.895+0000 7fad49c00640 1 -- 172.21.15.6:0/2883643748 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad441582f0 con 0x7fad44152400 2024-03-22T13:01:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.898+0000 7fad49c00640 1 --2- 172.21.15.6:0/2883643748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad44152400 0x7fad4415b740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad38009870 tx=0x7fad3802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70bc57ba7757874b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:01.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.903+0000 7fad43e00640 1 -- 172.21.15.6:0/2883643748 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3802f840 con 0x7fad44152400 2024-03-22T13:01:01.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.903+0000 7fad43e00640 1 -- 172.21.15.6:0/2883643748 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3802f9e0 con 0x7fad44152400 2024-03-22T13:01:01.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.903+0000 7fad43e00640 1 -- 172.21.15.6:0/2883643748 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3802fdb0 con 0x7fad44152400 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 -- 172.21.15.6:0/2883643748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad44152400 msgr2=0x7fad4415b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 --2- 172.21.15.6:0/2883643748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad44152400 0x7fad4415b740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad38009870 tx=0x7fad3802ba20 comp rx=0 tx=0).stop 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 -- 172.21.15.6:0/2883643748 shutdown_connections 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 --2- 172.21.15.6:0/2883643748 >> v2:172.21.15.6:3300/0 conn(0x7fad4415bc80 0x7fad440716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 --2- 172.21.15.6:0/2883643748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad44152400 0x7fad4415b740 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 -- 172.21.15.6:0/2883643748 >> 172.21.15.6:0/2883643748 conn(0x7fad440cb930 msgr2=0x7fad440cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 -- 172.21.15.6:0/2883643748 shutdown_connections 2024-03-22T13:01:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.904+0000 7fad4c600640 1 -- 172.21.15.6:0/2883643748 wait complete. 2024-03-22T13:01:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.906+0000 7fad4c600640 1 Processor -- start 2024-03-22T13:01:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.906+0000 7fad4c600640 1 -- start start 2024-03-22T13:01:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad4c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad4406f8c0 0x7fad4414f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad4c600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad44152400 0x7fad44156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad4c600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad4415bc80 0x7fad44159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad4c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad44158b90 con 0x7fad4406f8c0 2024-03-22T13:01:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad4c600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441588b0 con 0x7fad44152400 2024-03-22T13:01:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad4c600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad441527e0 con 0x7fad4415bc80 2024-03-22T13:01:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad49c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad4406f8c0 0x7fad4414f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.907+0000 7fad4a600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad4415bc80 0x7fad44159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.908+0000 7fad49200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad44152400 0x7fad44156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.908+0000 7fad49c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad4406f8c0 0x7fad4414f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37452/0 (socket says 172.21.15.6:37452) 2024-03-22T13:01:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.908+0000 7fad49c00640 1 -- 172.21.15.6:0/4228724671 learned_addr learned my addr 172.21.15.6:0/4228724671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.911+0000 7fad49c00640 1 -- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad4415bc80 msgr2=0x7fad44159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.911+0000 7fad49c00640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad4415bc80 0x7fad44159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.911+0000 7fad49c00640 1 -- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad44152400 msgr2=0x7fad44156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.911+0000 7fad49c00640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad44152400 0x7fad44156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.911+0000 7fad49c00640 1 -- 172.21.15.6:0/4228724671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44159c30 con 0x7fad4406f8c0 2024-03-22T13:01:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.914+0000 7fad49c00640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad4406f8c0 0x7fad4414f400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fad38000c00 tx=0x7fad38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:01.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.919+0000 7fad41600640 1 -- 172.21.15.6:0/4228724671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38037bc0 con 0x7fad4406f8c0 2024-03-22T13:01:01.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.919+0000 7fad41600640 1 -- 172.21.15.6:0/4228724671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad38046070 con 0x7fad4406f8c0 2024-03-22T13:01:01.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.919+0000 7fad41600640 1 -- 172.21.15.6:0/4228724671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38041400 con 0x7fad4406f8c0 2024-03-22T13:01:01.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.919+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad44159f10 con 0x7fad4406f8c0 2024-03-22T13:01:01.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.920+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4415a4d0 con 0x7fad4406f8c0 2024-03-22T13:01:01.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.921+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad440756c0 con 0x7fad4406f8c0 2024-03-22T13:01:01.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.955+0000 7fad41600640 1 -- 172.21.15.6:0/4228724671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fad38040070 con 0x7fad4406f8c0 2024-03-22T13:01:01.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.957+0000 7fad41600640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fad2406bc50 0x7fad2406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:01.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.957+0000 7fad49200640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fad2406bc50 0x7fad2406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:01.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.958+0000 7fad49200640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fad2406bc50 0x7fad2406e110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad4415b040 tx=0x7fad30009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:01.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:01.961+0000 7fad41600640 1 -- 172.21.15.6:0/4228724671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fad38079fd0 con 0x7fad4406f8c0 2024-03-22T13:01:02.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.282+0000 7fad41600640 1 -- 172.21.15.6:0/4228724671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fad38033030 con 0x7fad4406f8c0 2024-03-22T13:01:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.536+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad440c4d80 con 0x7fad4406f8c0 2024-03-22T13:01:02.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.559+0000 7fad41600640 1 -- 172.21.15.6:0/4228724671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fad3807f770 con 0x7fad4406f8c0 2024-03-22T13:01:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:02.560 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:02.560 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fad2406bc50 msgr2=0x7fad2406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fad2406bc50 0x7fad2406e110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad4415b040 tx=0x7fad30009190 comp rx=0 tx=0).stop 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad4406f8c0 msgr2=0x7fad4414f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad4406f8c0 0x7fad4414f400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fad38000c00 tx=0x7fad38004730 comp rx=0 tx=0).stop 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 shutdown_connections 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad4415bc80 0x7fad44159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fad2406bc50 0x7fad2406e110 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad44152400 0x7fad44156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.563+0000 7fad4c600640 1 --2- 172.21.15.6:0/4228724671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad4406f8c0 0x7fad4414f400 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.564+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 >> 172.21.15.6:0/4228724671 conn(0x7fad440cb930 msgr2=0x7fad4414fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:02.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.564+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 shutdown_connections 2024-03-22T13:01:02.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:02.564+0000 7fad4c600640 1 -- 172.21.15.6:0/4228724671 wait complete. 2024-03-22T13:01:03.192 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:03.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:04.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:04.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.393+0000 7fd52b000640 1 Processor -- start 2024-03-22T13:01:04.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.393+0000 7fd52b000640 1 -- start start 2024-03-22T13:01:04.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.395+0000 7fd52b000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd5241541a0 0x7fd5240c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:04.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.395+0000 7fd52b000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd52415a2c0 0x7fd52415a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:04.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.395+0000 7fd52b000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5240c90e0 con 0x7fd524154580 2024-03-22T13:01:04.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.395+0000 7fd52b000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52415b030 con 0x7fd52415a2c0 2024-03-22T13:01:04.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.395+0000 7fd52b000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5240c9350 con 0x7fd5241541a0 2024-03-22T13:01:04.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.403+0000 7fd529c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd5241541a0 0x7fd5240c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:04.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.404+0000 7fd529c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd5241541a0 0x7fd5240c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:39218/0 (socket says 172.21.15.6:39218) 2024-03-22T13:01:04.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.404+0000 7fd529c00640 1 -- 172.21.15.6:0/1670940577 learned_addr learned my addr 172.21.15.6:0/1670940577 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:04.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.407+0000 7fd529c00640 1 -- 172.21.15.6:0/1670940577 >> v2:172.21.15.6:3300/0 conn(0x7fd52415a2c0 msgr2=0x7fd52415a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:04.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.407+0000 7fd529c00640 1 --2- 172.21.15.6:0/1670940577 >> v2:172.21.15.6:3300/0 conn(0x7fd52415a2c0 0x7fd52415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:04.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.407+0000 7fd529c00640 1 -- 172.21.15.6:0/1670940577 >> v1:172.21.15.6:6789/0 conn(0x7fd524154580 legacy=0x7fd52406b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:04.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.408+0000 7fd529c00640 1 -- 172.21.15.6:0/1670940577 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5240c9630 con 0x7fd5241541a0 2024-03-22T13:01:04.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.410+0000 7fd529c00640 1 --2- 172.21.15.6:0/1670940577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5241541a0 0x7fd5240c72a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd524070c80 tx=0x7fd51802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adef47b0b8dcca1f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:04.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.414+0000 7fd523e00640 1 -- 172.21.15.6:0/1670940577 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518035020 con 0x7fd5241541a0 2024-03-22T13:01:04.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd523e00640 1 -- 172.21.15.6:0/1670940577 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518031870 con 0x7fd5241541a0 2024-03-22T13:01:04.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd523e00640 1 -- 172.21.15.6:0/1670940577 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518031c40 con 0x7fd5241541a0 2024-03-22T13:01:04.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd52b000640 1 -- 172.21.15.6:0/1670940577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5241541a0 msgr2=0x7fd5240c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:04.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd52b000640 1 --2- 172.21.15.6:0/1670940577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5241541a0 0x7fd5240c72a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd524070c80 tx=0x7fd51802b910 comp rx=0 tx=0).stop 2024-03-22T13:01:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd52b000640 1 -- 172.21.15.6:0/1670940577 shutdown_connections 2024-03-22T13:01:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd52b000640 1 --2- 172.21.15.6:0/1670940577 >> v2:172.21.15.6:3300/0 conn(0x7fd52415a2c0 0x7fd52415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd52b000640 1 --2- 172.21.15.6:0/1670940577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd5241541a0 0x7fd5240c72a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.415+0000 7fd52b000640 1 -- 172.21.15.6:0/1670940577 >> 172.21.15.6:0/1670940577 conn(0x7fd524070140 msgr2=0x7fd524070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.416+0000 7fd52b000640 1 -- 172.21.15.6:0/1670940577 shutdown_connections 2024-03-22T13:01:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.416+0000 7fd52b000640 1 -- 172.21.15.6:0/1670940577 wait complete. 2024-03-22T13:01:04.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.417+0000 7fd52b000640 1 Processor -- start 2024-03-22T13:01:04.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.417+0000 7fd52b000640 1 -- start start 2024-03-22T13:01:04.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.418+0000 7fd52b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5241541a0 0x7fd5240c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:04.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.418+0000 7fd52b000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd524154580 0x7fd52414e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:04.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.418+0000 7fd52b000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd52415a2c0 0x7fd52414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:04.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.418+0000 7fd52b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240ca110 con 0x7fd5241541a0 2024-03-22T13:01:04.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.418+0000 7fd52b000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240c9bf0 con 0x7fd52415a2c0 2024-03-22T13:01:04.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.418+0000 7fd52b000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52415ac90 con 0x7fd524154580 2024-03-22T13:01:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.419+0000 7fd52a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd52415a2c0 0x7fd52414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.419+0000 7fd529200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd524154580 0x7fd52414e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.420+0000 7fd52a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd52415a2c0 0x7fd52414e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:39224/0 (socket says 172.21.15.6:39224) 2024-03-22T13:01:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.420+0000 7fd52a600640 1 -- 172.21.15.6:0/590156674 learned_addr learned my addr 172.21.15.6:0/590156674 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.423+0000 7fd529200640 1 -- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd52415a2c0 msgr2=0x7fd52414e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.423+0000 7fd529200640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd52415a2c0 0x7fd52414e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.424+0000 7fd529200640 1 -- 172.21.15.6:0/590156674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5241541a0 msgr2=0x7fd5240c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:04.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.424+0000 7fd529200640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5241541a0 0x7fd5240c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:04.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.424+0000 7fd529200640 1 -- 172.21.15.6:0/590156674 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5241f0a10 con 0x7fd524154580 2024-03-22T13:01:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.429+0000 7fd529200640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd524154580 0x7fd52414e1c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd51000c440 tx=0x7fd51000c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.484+0000 7fd521600640 1 -- 172.21.15.6:0/590156674 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510017070 con 0x7fd524154580 2024-03-22T13:01:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.484+0000 7fd521600640 1 -- 172.21.15.6:0/590156674 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5100035f0 con 0x7fd524154580 2024-03-22T13:01:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.484+0000 7fd521600640 1 -- 172.21.15.6:0/590156674 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5100039e0 con 0x7fd524154580 2024-03-22T13:01:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.484+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52414cb80 con 0x7fd524154580 2024-03-22T13:01:04.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.485+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5240c8a20 con 0x7fd524154580 2024-03-22T13:01:04.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.486+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd524075650 con 0x7fd524154580 2024-03-22T13:01:04.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.776+0000 7fd521600640 1 -- 172.21.15.6:0/590156674 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd51001a070 con 0x7fd524154580 2024-03-22T13:01:04.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.778+0000 7fd521600640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd50406bc20 0x7fd50406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:04.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.779+0000 7fd529c00640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd50406bc20 0x7fd50406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:04.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.779+0000 7fd529c00640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd50406bc20 0x7fd50406e0e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd518036000 tx=0x7fd518030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:04.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.780+0000 7fd521600640 1 -- 172.21.15.6:0/590156674 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd510056830 con 0x7fd524154580 2024-03-22T13:01:04.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:04.795+0000 7fd521600640 1 -- 172.21.15.6:0/590156674 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd510021020 con 0x7fd524154580 2024-03-22T13:01:05.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.055+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd524073250 con 0x7fd524154580 2024-03-22T13:01:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.079+0000 7fd521600640 1 -- 172.21.15.6:0/590156674 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fd510058790 con 0x7fd524154580 2024-03-22T13:01:05.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:05.079 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:05.079 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.081+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd50406bc20 msgr2=0x7fd50406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.081+0000 7fd52b000640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd50406bc20 0x7fd50406e0e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd518036000 tx=0x7fd518030000 comp rx=0 tx=0).stop 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd524154580 msgr2=0x7fd52414e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd524154580 0x7fd52414e1c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd51000c440 tx=0x7fd51000c840 comp rx=0 tx=0).stop 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 shutdown_connections 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd50406bc20 0x7fd50406e0e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd52415a2c0 0x7fd52414e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd524154580 0x7fd52414e1c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 --2- 172.21.15.6:0/590156674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5241541a0 0x7fd5240c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 >> 172.21.15.6:0/590156674 conn(0x7fd524070140 msgr2=0x7fd524159b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 shutdown_connections 2024-03-22T13:01:05.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:05.082+0000 7fd52b000640 1 -- 172.21.15.6:0/590156674 wait complete. 2024-03-22T13:01:05.712 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:05.712 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:06.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:06.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.915+0000 7f0dd1c00640 1 Processor -- start 2024-03-22T13:01:06.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.915+0000 7f0dd1c00640 1 -- start start 2024-03-22T13:01:06.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.916+0000 7f0dd1c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:06.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.916+0000 7f0dd1c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0dcc15b0e0 0x7f0dcc15b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:06.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.916+0000 7f0dd1c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0dcc0ca900 0x7f0dcc0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:06.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.916+0000 7f0dd1c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc0cd000 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.916+0000 7f0dd1c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc06c7a0 con 0x7f0dcc0ca900 2024-03-22T13:01:06.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.916+0000 7f0dd1c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc06ca80 con 0x7f0dcc0ca520 2024-03-22T13:01:06.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.917+0000 7f0dca000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0dcc15b0e0 0x7f0dcc15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:06.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.917+0000 7f0dcaa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:06.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.918+0000 7f0dca000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0dcc15b0e0 0x7f0dcc15b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39228/0 (socket says 172.21.15.6:39228) 2024-03-22T13:01:06.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.918+0000 7f0dca000640 1 -- 172.21.15.6:0/2377786115 learned_addr learned my addr 172.21.15.6:0/2377786115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.920+0000 7f0dca000640 1 -- 172.21.15.6:0/2377786115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 msgr2=0x7f0dcc06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.920+0000 7f0dca000640 1 --2- 172.21.15.6:0/2377786115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc06d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.920+0000 7f0dca000640 1 -- 172.21.15.6:0/2377786115 >> v2:172.21.15.92:3300/0 conn(0x7f0dcc0ca900 msgr2=0x7f0dcc0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.920+0000 7f0dca000640 1 --2- 172.21.15.6:0/2377786115 >> v2:172.21.15.92:3300/0 conn(0x7f0dcc0ca900 0x7f0dcc0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.920+0000 7f0dca000640 1 -- 172.21.15.6:0/2377786115 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc06cd60 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.923+0000 7f0dca000640 1 --2- 172.21.15.6:0/2377786115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc15b980 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc08d500 tx=0x7f0db802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ffe55d4c57913c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:06.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.927+0000 7f0dc9600640 1 -- 172.21.15.6:0/2377786115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8035020 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.927+0000 7f0dc9600640 1 -- 172.21.15.6:0/2377786115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db8031870 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.927+0000 7f0dc9600640 1 -- 172.21.15.6:0/2377786115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db8031c40 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2377786115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 msgr2=0x7f0dcc15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2377786115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc15b980 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc08d500 tx=0x7f0db802b910 comp rx=0 tx=0).stop 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2377786115 shutdown_connections 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2377786115 >> v2:172.21.15.92:3300/0 conn(0x7f0dcc0ca900 0x7f0dcc0cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2377786115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc15b980 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2377786115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2377786115 >> 172.21.15.6:0/2377786115 conn(0x7f0dcc0b16b0 msgr2=0x7f0dcc0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2377786115 shutdown_connections 2024-03-22T13:01:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.928+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2377786115 wait complete. 2024-03-22T13:01:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.930+0000 7f0dd1c00640 1 Processor -- start 2024-03-22T13:01:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.930+0000 7f0dd1c00640 1 -- start start 2024-03-22T13:01:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.931+0000 7f0dd1c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.931+0000 7f0dd1c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dcc0ca900 0x7f0dcc157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.931+0000 7f0dd1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc0b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.931+0000 7f0dd1c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0cad60 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.931+0000 7f0dd1c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc06d320 con 0x7f0dcc0ca900 2024-03-22T13:01:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.931+0000 7f0dd1c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc154c20 con 0x7f0dcc0ca520 2024-03-22T13:01:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.932+0000 7f0dcb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc0b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.932+0000 7f0dcaa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.932+0000 7f0dcb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc0b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39240/0 (socket says 172.21.15.6:39240) 2024-03-22T13:01:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.932+0000 7f0dcb400640 1 -- 172.21.15.6:0/2676224355 learned_addr learned my addr 172.21.15.6:0/2676224355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.936+0000 7f0dcb400640 1 -- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 msgr2=0x7f0dcc154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.936+0000 7f0dcb400640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.936+0000 7f0dcb400640 1 -- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dcc0ca900 msgr2=0x7f0dcc157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.936+0000 7f0dcb400640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dcc0ca900 0x7f0dcc157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.936+0000 7f0dcb400640 1 -- 172.21.15.6:0/2676224355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc1555a0 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.939+0000 7f0dcb400640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc0b44e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400b970 tx=0x7f0dc400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.980+0000 7f0dc2a00640 1 -- 172.21.15.6:0/2676224355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4019070 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.980+0000 7f0dc2a00640 1 -- 172.21.15.6:0/2676224355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc400cb60 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.980+0000 7f0dc2a00640 1 -- 172.21.15.6:0/2676224355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4014440 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.980+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc156d00 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.980+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc1572c0 con 0x7f0dcc15b0e0 2024-03-22T13:01:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:06.982+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc0756c0 con 0x7f0dcc15b0e0 2024-03-22T13:01:07.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.297+0000 7f0dc2a00640 1 -- 172.21.15.6:0/2676224355 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0dc401c070 con 0x7f0dcc15b0e0 2024-03-22T13:01:07.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.301+0000 7f0dc2a00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0dac06bc50 0x7f0dac06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:07.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.301+0000 7f0dcaa00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0dac06bc50 0x7f0dac06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:07.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.302+0000 7f0dcaa00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0dac06bc50 0x7f0dac06e110 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009aa0 tx=0x7f0dbc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:07.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.336+0000 7f0dc2a00640 1 -- 172.21.15.6:0/2676224355 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0dc4057140 con 0x7f0dcc15b0e0 2024-03-22T13:01:07.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.337+0000 7f0dc2a00640 1 -- 172.21.15.6:0/2676224355 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0dc4023020 con 0x7f0dcc15b0e0 2024-03-22T13:01:07.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.596+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dcc0cb080 con 0x7f0dcc15b0e0 2024-03-22T13:01:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.623+0000 7f0dc2a00640 1 -- 172.21.15.6:0/2676224355 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f0dc40590a0 con 0x7f0dcc15b0e0 2024-03-22T13:01:07.624 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:07.624 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.626+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0dac06bc50 msgr2=0x7f0dac06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.626+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0dac06bc50 0x7f0dac06e110 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009aa0 tx=0x7f0dbc009190 comp rx=0 tx=0).stop 2024-03-22T13:01:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.626+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 msgr2=0x7f0dcc0b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.626+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc0b44e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400b970 tx=0x7f0dc400be30 comp rx=0 tx=0).stop 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.626+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 shutdown_connections 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.627+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dcc15b0e0 0x7f0dcc0b44e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.627+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0dac06bc50 0x7f0dac06e110 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.627+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dcc0ca900 0x7f0dcc157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.627+0000 7f0dd1c00640 1 --2- 172.21.15.6:0/2676224355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dcc0ca520 0x7f0dcc154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.627+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 >> 172.21.15.6:0/2676224355 conn(0x7f0dcc0b16b0 msgr2=0x7f0dcc0cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.627+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 shutdown_connections 2024-03-22T13:01:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:07.627+0000 7f0dd1c00640 1 -- 172.21.15.6:0/2676224355 wait complete. 2024-03-22T13:01:08.247 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:08.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:09.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.449+0000 7fca3e600640 1 Processor -- start 2024-03-22T13:01:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.449+0000 7fca3e600640 1 -- start start 2024-03-22T13:01:09.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.451+0000 7fca3e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca380cc380 0x7fca3806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:09.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.451+0000 7fca3e600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380ce8a0 0x7fca3815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:09.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.451+0000 7fca3e600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca380cee00 con 0x7fca380cc760 2024-03-22T13:01:09.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.451+0000 7fca3e600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca380cf0c0 con 0x7fca380cc380 2024-03-22T13:01:09.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.451+0000 7fca3e600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca380cf3a0 con 0x7fca380ce8a0 2024-03-22T13:01:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.452+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380ce8a0 0x7fca3815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.452+0000 7fca3d200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca380cc380 0x7fca3806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.452+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380ce8a0 0x7fca3815ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:58492/0 (socket says 172.21.15.6:58492) 2024-03-22T13:01:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.452+0000 7fca2fe00640 1 -- 172.21.15.6:0/2920923098 learned_addr learned my addr 172.21.15.6:0/2920923098 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:09.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.455+0000 7fca2fe00640 1 -- 172.21.15.6:0/2920923098 >> v2:172.21.15.6:3300/0 conn(0x7fca380cc380 msgr2=0x7fca3806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:09.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.455+0000 7fca2fe00640 1 --2- 172.21.15.6:0/2920923098 >> v2:172.21.15.6:3300/0 conn(0x7fca380cc380 0x7fca3806c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:09.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.455+0000 7fca2fe00640 1 -- 172.21.15.6:0/2920923098 >> v1:172.21.15.6:6789/0 conn(0x7fca380cc760 legacy=0x7fca380cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:09.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.455+0000 7fca2fe00640 1 -- 172.21.15.6:0/2920923098 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca380cf680 con 0x7fca380ce8a0 2024-03-22T13:01:09.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.457+0000 7fca2fe00640 1 --2- 172.21.15.6:0/2920923098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380ce8a0 0x7fca3815ae30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca24009870 tx=0x7fca2402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62af5e2e43e470c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:09.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.462+0000 7fca37e00640 1 -- 172.21.15.6:0/2920923098 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24034070 con 0x7fca380ce8a0 2024-03-22T13:01:09.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.462+0000 7fca37e00640 1 -- 172.21.15.6:0/2920923098 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca24030840 con 0x7fca380ce8a0 2024-03-22T13:01:09.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.462+0000 7fca37e00640 1 -- 172.21.15.6:0/2920923098 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24030c10 con 0x7fca380ce8a0 2024-03-22T13:01:09.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.462+0000 7fca3e600640 1 -- 172.21.15.6:0/2920923098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380ce8a0 msgr2=0x7fca3815ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.462+0000 7fca3e600640 1 --2- 172.21.15.6:0/2920923098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380ce8a0 0x7fca3815ae30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca24009870 tx=0x7fca2402b980 comp rx=0 tx=0).stop 2024-03-22T13:01:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.463+0000 7fca3e600640 1 -- 172.21.15.6:0/2920923098 shutdown_connections 2024-03-22T13:01:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.463+0000 7fca3e600640 1 --2- 172.21.15.6:0/2920923098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380ce8a0 0x7fca3815ae30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.463+0000 7fca3e600640 1 --2- 172.21.15.6:0/2920923098 >> v2:172.21.15.6:3300/0 conn(0x7fca380cc380 0x7fca3806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.463+0000 7fca3e600640 1 -- 172.21.15.6:0/2920923098 >> 172.21.15.6:0/2920923098 conn(0x7fca380b1480 msgr2=0x7fca380b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.463+0000 7fca3e600640 1 -- 172.21.15.6:0/2920923098 shutdown_connections 2024-03-22T13:01:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.463+0000 7fca3e600640 1 -- 172.21.15.6:0/2920923098 wait complete. 2024-03-22T13:01:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.465+0000 7fca3e600640 1 Processor -- start 2024-03-22T13:01:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.465+0000 7fca3e600640 1 -- start start 2024-03-22T13:01:09.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.466+0000 7fca3e600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380cc380 0x7fca38148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.466+0000 7fca3e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca380cc760 0x7fca38155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.466+0000 7fca3e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380ce8a0 0x7fca381560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.466+0000 7fca3e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca381549d0 con 0x7fca380ce8a0 2024-03-22T13:01:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.466+0000 7fca3e600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3815a910 con 0x7fca380cc760 2024-03-22T13:01:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.466+0000 7fca3e600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca380ccb90 con 0x7fca380cc380 2024-03-22T13:01:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.467+0000 7fca3d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380cc380 0x7fca38148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.467+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca380cc760 0x7fca38155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.467+0000 7fca3dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380ce8a0 0x7fca381560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.467+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca380cc760 0x7fca38155bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:39240/0 (socket says 172.21.15.6:39240) 2024-03-22T13:01:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.467+0000 7fca2fe00640 1 -- 172.21.15.6:0/605520298 learned_addr learned my addr 172.21.15.6:0/605520298 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:09.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.471+0000 7fca2fe00640 1 -- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380cc380 msgr2=0x7fca38148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.471+0000 7fca2fe00640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380cc380 0x7fca38148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.471+0000 7fca2fe00640 1 -- 172.21.15.6:0/605520298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380ce8a0 msgr2=0x7fca381560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.471+0000 7fca2fe00640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380ce8a0 0x7fca381560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.471+0000 7fca2fe00640 1 -- 172.21.15.6:0/605520298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38153450 con 0x7fca380cc760 2024-03-22T13:01:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.473+0000 7fca2fe00640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca380cc760 0x7fca38155bb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca24000c00 tx=0x7fca24004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:09.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.478+0000 7fca35600640 1 -- 172.21.15.6:0/605520298 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24048070 con 0x7fca380cc760 2024-03-22T13:01:09.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.478+0000 7fca35600640 1 -- 172.21.15.6:0/605520298 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca24030640 con 0x7fca380cc760 2024-03-22T13:01:09.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.479+0000 7fca35600640 1 -- 172.21.15.6:0/605520298 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24034040 con 0x7fca380cc760 2024-03-22T13:01:09.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.479+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca38154df0 con 0x7fca380cc760 2024-03-22T13:01:09.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.479+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca381553b0 con 0x7fca380cc760 2024-03-22T13:01:09.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.480+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca38075650 con 0x7fca380cc760 2024-03-22T13:01:09.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.511+0000 7fca35600640 1 -- 172.21.15.6:0/605520298 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fca2404b070 con 0x7fca380cc760 2024-03-22T13:01:09.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.513+0000 7fca35600640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fca1806bc20 0x7fca1806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:09.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.514+0000 7fca3d200640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fca1806bc20 0x7fca1806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:09.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.514+0000 7fca3d200640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fca1806bc20 0x7fca1806e0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca28011340 tx=0x7fca2800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:09.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.515+0000 7fca35600640 1 -- 172.21.15.6:0/605520298 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fca24079350 con 0x7fca380cc760 2024-03-22T13:01:09.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:09.780+0000 7fca35600640 1 -- 172.21.15.6:0/605520298 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fca24052020 con 0x7fca380cc760 2024-03-22T13:01:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.036+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca3806de60 con 0x7fca380cc760 2024-03-22T13:01:10.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.192+0000 7fca35600640 1 -- 172.21.15.6:0/605520298 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fca24030810 con 0x7fca380cc760 2024-03-22T13:01:10.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:10.192 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:10.192 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:10.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fca1806bc20 msgr2=0x7fca1806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:10.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fca1806bc20 0x7fca1806e0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca28011340 tx=0x7fca2800b040 comp rx=0 tx=0).stop 2024-03-22T13:01:10.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca380cc760 msgr2=0x7fca38155bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:10.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca380cc760 0x7fca38155bb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca24000c00 tx=0x7fca24004730 comp rx=0 tx=0).stop 2024-03-22T13:01:10.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 shutdown_connections 2024-03-22T13:01:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380ce8a0 0x7fca381560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fca1806bc20 0x7fca1806e0e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca380cc760 0x7fca38155bb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 --2- 172.21.15.6:0/605520298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca380cc380 0x7fca38148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 >> 172.21.15.6:0/605520298 conn(0x7fca380b1480 msgr2=0x7fca38159800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.195+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 shutdown_connections 2024-03-22T13:01:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:10.196+0000 7fca3e600640 1 -- 172.21.15.6:0/605520298 wait complete. 2024-03-22T13:01:10.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:10.816 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:11.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:12.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.012+0000 7fd06c400640 1 Processor -- start 2024-03-22T13:01:12.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.012+0000 7fd06c400640 1 -- start start 2024-03-22T13:01:12.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.014+0000 7fd06c400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0640c9630 0x7fd0640c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:12.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.014+0000 7fd06c400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0640c2e50 0x7fd0640c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:12.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.014+0000 7fd06c400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd064158010 con 0x7fd06415b850 2024-03-22T13:01:12.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.014+0000 7fd06c400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd064158ee0 con 0x7fd0640c9630 2024-03-22T13:01:12.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.014+0000 7fd06c400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0641582d0 con 0x7fd0640c2e50 2024-03-22T13:01:12.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.015+0000 7fd069a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0640c9630 0x7fd0640c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:12.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.015+0000 7fd069000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0640c2e50 0x7fd0640c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:12.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.015+0000 7fd06a400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd06415b850 0x7fd06415bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:56044/0 (socket says 172.21.15.6:56044) 2024-03-22T13:01:12.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.015+0000 7fd06a400640 1 -- 172.21.15.6:0/3707195238 learned_addr learned my addr 172.21.15.6:0/3707195238 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:12.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.017+0000 7fd069000640 1 -- 172.21.15.6:0/3707195238 >> v2:172.21.15.6:3300/0 conn(0x7fd0640c9630 msgr2=0x7fd0640c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:12.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.017+0000 7fd069000640 1 --2- 172.21.15.6:0/3707195238 >> v2:172.21.15.6:3300/0 conn(0x7fd0640c9630 0x7fd0640c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.017+0000 7fd069000640 1 -- 172.21.15.6:0/3707195238 >> v1:172.21.15.92:6789/0 conn(0x7fd06415b850 legacy=0x7fd06415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:12.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.017+0000 7fd069000640 1 -- 172.21.15.6:0/3707195238 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0641585b0 con 0x7fd0640c2e50 2024-03-22T13:01:12.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.020+0000 7fd069000640 1 --2- 172.21.15.6:0/3707195238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0640c2e50 0x7fd0640c8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd0500097a0 tx=0x7fd05002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c8e76f0e80d7cac server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:12.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.024+0000 7fd063e00640 1 -- 172.21.15.6:0/3707195238 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050034070 con 0x7fd0640c2e50 2024-03-22T13:01:12.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.024+0000 7fd063e00640 1 -- 172.21.15.6:0/3707195238 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd050030610 con 0x7fd0640c2e50 2024-03-22T13:01:12.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.024+0000 7fd063e00640 1 -- 172.21.15.6:0/3707195238 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500309e0 con 0x7fd0640c2e50 2024-03-22T13:01:12.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.025+0000 7fd06c400640 1 -- 172.21.15.6:0/3707195238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0640c2e50 msgr2=0x7fd0640c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:12.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.025+0000 7fd06c400640 1 --2- 172.21.15.6:0/3707195238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0640c2e50 0x7fd0640c8a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd0500097a0 tx=0x7fd05002b960 comp rx=0 tx=0).stop 2024-03-22T13:01:12.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.025+0000 7fd06c400640 1 -- 172.21.15.6:0/3707195238 shutdown_connections 2024-03-22T13:01:12.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.025+0000 7fd06c400640 1 --2- 172.21.15.6:0/3707195238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0640c2e50 0x7fd0640c8a70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.025+0000 7fd06c400640 1 --2- 172.21.15.6:0/3707195238 >> v2:172.21.15.6:3300/0 conn(0x7fd0640c9630 0x7fd0640c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.025+0000 7fd06c400640 1 -- 172.21.15.6:0/3707195238 >> 172.21.15.6:0/3707195238 conn(0x7fd0640714f0 msgr2=0x7fd064071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:12.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.026+0000 7fd06c400640 1 -- 172.21.15.6:0/3707195238 shutdown_connections 2024-03-22T13:01:12.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.026+0000 7fd06c400640 1 -- 172.21.15.6:0/3707195238 wait complete. 2024-03-22T13:01:12.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.028+0000 7fd06c400640 1 Processor -- start 2024-03-22T13:01:12.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.028+0000 7fd06c400640 1 -- start start 2024-03-22T13:01:12.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.029+0000 7fd06c400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0640c2e50 0x7fd0641e3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:12.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.029+0000 7fd06c400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0640c9630 0x7fd0640cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:12.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.029+0000 7fd06c400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd06415b850 0x7fd0640cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:12.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.029+0000 7fd06c400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0640ca330 con 0x7fd0640c9630 2024-03-22T13:01:12.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.029+0000 7fd06c400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd064158b70 con 0x7fd0640c2e50 2024-03-22T13:01:12.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.029+0000 7fd06c400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0640c8fb0 con 0x7fd06415b850 2024-03-22T13:01:12.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.030+0000 7fd06a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd06415b850 0x7fd0640cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:12.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.030+0000 7fd069000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0640c9630 0x7fd0640cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:12.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.031+0000 7fd06a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd06415b850 0x7fd0640cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:58514/0 (socket says 172.21.15.6:58514) 2024-03-22T13:01:12.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.031+0000 7fd06a400640 1 -- 172.21.15.6:0/3795545322 learned_addr learned my addr 172.21.15.6:0/3795545322 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:12.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.033+0000 7fd069a00640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0640c2e50 0x7fd0641e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:12.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.037+0000 7fd06a400640 1 -- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0640c2e50 msgr2=0x7fd0641e3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:12.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.037+0000 7fd06a400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0640c2e50 0x7fd0641e3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.037+0000 7fd06a400640 1 -- 172.21.15.6:0/3795545322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0640c9630 msgr2=0x7fd0640cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:12.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.037+0000 7fd06a400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0640c9630 0x7fd0640cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.037+0000 7fd06a400640 1 -- 172.21.15.6:0/3795545322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0641f2550 con 0x7fd06415b850 2024-03-22T13:01:12.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.042+0000 7fd06a400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd06415b850 0x7fd0640cc6a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00cbf0 tx=0x7fd05c00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:12.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.045+0000 7fd061600640 1 -- 172.21.15.6:0/3795545322 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c017070 con 0x7fd06415b850 2024-03-22T13:01:12.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.045+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0641e1530 con 0x7fd06415b850 2024-03-22T13:01:12.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.045+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0641e1af0 con 0x7fd06415b850 2024-03-22T13:01:12.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.047+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0640b3fc0 con 0x7fd06415b850 2024-03-22T13:01:12.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.047+0000 7fd061600640 1 -- 172.21.15.6:0/3795545322 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c00ad40 con 0x7fd06415b850 2024-03-22T13:01:12.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.047+0000 7fd061600640 1 -- 172.21.15.6:0/3795545322 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c021060 con 0x7fd06415b850 2024-03-22T13:01:12.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.073+0000 7fd061600640 1 -- 172.21.15.6:0/3795545322 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd05c01a070 con 0x7fd06415b850 2024-03-22T13:01:12.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.075+0000 7fd061600640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd04406bbd0 0x7fd04406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:12.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.075+0000 7fd061600640 1 -- 172.21.15.6:0/3795545322 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd05c057550 con 0x7fd06415b850 2024-03-22T13:01:12.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.075+0000 7fd069a00640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd04406bbd0 0x7fd04406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.076+0000 7fd069a00640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd04406bbd0 0x7fd04406e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd058009b40 tx=0x7fd058009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.348+0000 7fd061600640 1 -- 172.21.15.6:0/3795545322 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd0640b3fc0 con 0x7fd06415b850 2024-03-22T13:01:12.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.608+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0640c4540 con 0x7fd06415b850 2024-03-22T13:01:12.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.628+0000 7fd061600640 1 -- 172.21.15.6:0/3795545322 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7fd0640c4540 con 0x7fd06415b850 2024-03-22T13:01:12.629 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:12.629 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:12.629 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:12.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd04406bbd0 msgr2=0x7fd04406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:12.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd04406bbd0 0x7fd04406e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd058009b40 tx=0x7fd058009190 comp rx=0 tx=0).stop 2024-03-22T13:01:12.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd06415b850 msgr2=0x7fd0640cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:12.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd06415b850 0x7fd0640cc6a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00cbf0 tx=0x7fd05c00a430 comp rx=0 tx=0).stop 2024-03-22T13:01:12.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 shutdown_connections 2024-03-22T13:01:12.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd06415b850 0x7fd0640cc6a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0640c9630 0x7fd0640cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd04406bbd0 0x7fd04406e090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 --2- 172.21.15.6:0/3795545322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0640c2e50 0x7fd0641e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:12.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 >> 172.21.15.6:0/3795545322 conn(0x7fd0640714f0 msgr2=0x7fd0640ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:12.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 shutdown_connections 2024-03-22T13:01:12.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:12.631+0000 7fd06c400640 1 -- 172.21.15.6:0/3795545322 wait complete. 2024-03-22T13:01:13.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:13.254 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:14.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:14.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.453+0000 7f74f6000640 1 Processor -- start 2024-03-22T13:01:14.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.453+0000 7f74f6000640 1 -- start start 2024-03-22T13:01:14.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.454+0000 7f74f6000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74f00c5210 0x7f74f0158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:14.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.455+0000 7f74f6000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f00c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:14.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.455+0000 7f74f6000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f006e0f0 con 0x7f74f00c67d0 2024-03-22T13:01:14.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.455+0000 7f74f6000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f00c6f90 con 0x7f74f00c5210 2024-03-22T13:01:14.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.455+0000 7f74f6000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f006d350 con 0x7f74f00c55f0 2024-03-22T13:01:14.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.456+0000 7f74efe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f00c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:14.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.456+0000 7f74f4c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74f00c5210 0x7f74f0158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:14.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.456+0000 7f74efe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f00c6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:39320/0 (socket says 172.21.15.6:39320) 2024-03-22T13:01:14.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.456+0000 7f74efe00640 1 -- 172.21.15.6:0/3993350393 learned_addr learned my addr 172.21.15.6:0/3993350393 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:14.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.459+0000 7f74efe00640 1 -- 172.21.15.6:0/3993350393 >> v2:172.21.15.6:3300/0 conn(0x7f74f00c5210 msgr2=0x7f74f0158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:14.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.459+0000 7f74efe00640 1 --2- 172.21.15.6:0/3993350393 >> v2:172.21.15.6:3300/0 conn(0x7f74f00c5210 0x7f74f0158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:14.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.459+0000 7f74efe00640 1 -- 172.21.15.6:0/3993350393 >> v1:172.21.15.6:6789/0 conn(0x7f74f00c67d0 legacy=0x7f74f006ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:14.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.459+0000 7f74efe00640 1 -- 172.21.15.6:0/3993350393 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f00c6cb0 con 0x7f74f00c55f0 2024-03-22T13:01:14.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.463+0000 7f74efe00640 1 --2- 172.21.15.6:0/3993350393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f00c6290 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009870 tx=0x7f74dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=106b679eb51261cf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:14.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.465+0000 7f74ef400640 1 -- 172.21.15.6:0/3993350393 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc034070 con 0x7f74f00c55f0 2024-03-22T13:01:14.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74ef400640 1 -- 172.21.15.6:0/3993350393 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74dc030840 con 0x7f74f00c55f0 2024-03-22T13:01:14.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74ef400640 1 -- 172.21.15.6:0/3993350393 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc030c10 con 0x7f74f00c55f0 2024-03-22T13:01:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74f6000640 1 -- 172.21.15.6:0/3993350393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 msgr2=0x7f74f00c6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74f6000640 1 --2- 172.21.15.6:0/3993350393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f00c6290 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009870 tx=0x7f74dc02b980 comp rx=0 tx=0).stop 2024-03-22T13:01:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74f6000640 1 -- 172.21.15.6:0/3993350393 shutdown_connections 2024-03-22T13:01:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74f6000640 1 --2- 172.21.15.6:0/3993350393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f00c6290 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:14.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74f6000640 1 --2- 172.21.15.6:0/3993350393 >> v2:172.21.15.6:3300/0 conn(0x7f74f00c5210 0x7f74f0158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:14.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74f6000640 1 -- 172.21.15.6:0/3993350393 >> 172.21.15.6:0/3993350393 conn(0x7f74f00cb890 msgr2=0x7f74f00cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:14.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.468+0000 7f74f6000640 1 -- 172.21.15.6:0/3993350393 shutdown_connections 2024-03-22T13:01:14.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.469+0000 7f74f6000640 1 -- 172.21.15.6:0/3993350393 wait complete. 2024-03-22T13:01:14.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.470+0000 7f74f6000640 1 Processor -- start 2024-03-22T13:01:14.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.470+0000 7f74f6000640 1 -- start start 2024-03-22T13:01:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.471+0000 7f74f6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00c5210 0x7f74f00cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.471+0000 7f74f6000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f01f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.471+0000 7f74f6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f00c67d0 0x7f74f01f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.471+0000 7f74f6000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f006dbd0 con 0x7f74f00c67d0 2024-03-22T13:01:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.471+0000 7f74f6000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f006d8f0 con 0x7f74f00c5210 2024-03-22T13:01:14.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.471+0000 7f74f6000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f006f4f0 con 0x7f74f00c55f0 2024-03-22T13:01:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.472+0000 7f74efe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f01f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.472+0000 7f74f4c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00c5210 0x7f74f00cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.472+0000 7f74f5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f00c67d0 0x7f74f01f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.472+0000 7f74f5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f00c67d0 0x7f74f01f2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35042/0 (socket says 172.21.15.6:35042) 2024-03-22T13:01:14.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.472+0000 7f74f5600640 1 -- 172.21.15.6:0/360862303 learned_addr learned my addr 172.21.15.6:0/360862303 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:14.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.476+0000 7f74f5600640 1 -- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 msgr2=0x7f74f01f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:14.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.476+0000 7f74f5600640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f01f1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:14.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.476+0000 7f74f5600640 1 -- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00c5210 msgr2=0x7f74f00cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:14.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.476+0000 7f74f5600640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00c5210 0x7f74f00cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:14.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.476+0000 7f74f5600640 1 -- 172.21.15.6:0/360862303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f0156b00 con 0x7f74f00c67d0 2024-03-22T13:01:14.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.479+0000 7f74f5600640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f00c67d0 0x7f74f01f2140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f74e800c680 tx=0x7f74e800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:14.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.485+0000 7f74ecc00640 1 -- 172.21.15.6:0/360862303 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8003b70 con 0x7f74f00c67d0 2024-03-22T13:01:14.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.485+0000 7f74ecc00640 1 -- 172.21.15.6:0/360862303 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e8003d10 con 0x7f74f00c67d0 2024-03-22T13:01:14.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.485+0000 7f74ecc00640 1 -- 172.21.15.6:0/360862303 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8005310 con 0x7f74f00c67d0 2024-03-22T13:01:14.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.485+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f015b230 con 0x7f74f00c67d0 2024-03-22T13:01:14.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.485+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f01f1120 con 0x7f74f00c67d0 2024-03-22T13:01:14.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.486+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f0075650 con 0x7f74f00c67d0 2024-03-22T13:01:14.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.509+0000 7f74ecc00640 1 -- 172.21.15.6:0/360862303 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74e80054b0 con 0x7f74f00c67d0 2024-03-22T13:01:14.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.511+0000 7f74ecc00640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74d006bc20 0x7f74d006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:14.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.511+0000 7f74f4c00640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74d006bc20 0x7f74d006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:14.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.512+0000 7f74f4c00640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74d006bc20 0x7f74d006e0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009b40 tx=0x7f74e4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.824+0000 7f74ecc00640 1 -- 172.21.15.6:0/360862303 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f74e80057a0 con 0x7f74f00c67d0 2024-03-22T13:01:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:14.824+0000 7f74ecc00640 1 -- 172.21.15.6:0/360862303 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f74e80596d0 con 0x7f74f00c67d0 2024-03-22T13:01:15.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.079+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74f014c470 con 0x7f74f00c67d0 2024-03-22T13:01:15.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.106+0000 7f74ecc00640 1 -- 172.21.15.6:0/360862303 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f74e805d770 con 0x7f74f00c67d0 2024-03-22T13:01:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:15.106 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:15.106 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.109+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74d006bc20 msgr2=0x7f74d006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.109+0000 7f74f6000640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74d006bc20 0x7f74d006e0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009b40 tx=0x7f74e4009190 comp rx=0 tx=0).stop 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.109+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f00c67d0 msgr2=0x7f74f01f2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.109+0000 7f74f6000640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f00c67d0 0x7f74f01f2140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f74e800c680 tx=0x7f74e800cb40 comp rx=0 tx=0).stop 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 shutdown_connections 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f00c67d0 0x7f74f01f2140 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74f00c55f0 0x7f74f01f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74d006bc20 0x7f74d006e0e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 --2- 172.21.15.6:0/360862303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f00c5210 0x7f74f00cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 >> 172.21.15.6:0/360862303 conn(0x7f74f00cb890 msgr2=0x7f74f00cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 shutdown_connections 2024-03-22T13:01:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:15.110+0000 7f74f6000640 1 -- 172.21.15.6:0/360862303 wait complete. 2024-03-22T13:01:15.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:15.735 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:16.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.946+0000 7f3f2b200640 1 Processor -- start 2024-03-22T13:01:16.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.946+0000 7f3f2b200640 1 -- start start 2024-03-22T13:01:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.948+0000 7f3f2b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f2415b0c0 0x7f3f240ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.948+0000 7f3f2b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f240cc380 con 0x7f3f24154980 2024-03-22T13:01:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.949+0000 7f3f2b200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f240cc660 con 0x7f3f240cd300 2024-03-22T13:01:16.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.949+0000 7f3f2b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2415a520 con 0x7f3f2415b0c0 2024-03-22T13:01:16.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.949+0000 7f3f23400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3f240cd300 0x7f3f2415a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60604/0 (socket says 172.21.15.6:60604) 2024-03-22T13:01:16.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.949+0000 7f3f23400640 1 -- 172.21.15.6:0/208563352 learned_addr learned my addr 172.21.15.6:0/208563352 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:16.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.949+0000 7f3f29200640 1 --2- 172.21.15.6:0/208563352 >> v2:172.21.15.6:3300/0 conn(0x7f3f2415b0c0 0x7f3f240ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:16.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.952+0000 7f3f22a00640 1 -- 172.21.15.6:0/208563352 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126773680 0 0) 0x7f3f240cc660 con 0x7f3f240cd300 2024-03-22T13:01:16.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.952+0000 7f3f22a00640 1 -- 172.21.15.6:0/208563352 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f08003520 con 0x7f3f240cd300 2024-03-22T13:01:16.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.954+0000 7f3f29200640 1 -- 172.21.15.6:0/208563352 >> v1:172.21.15.92:6789/0 conn(0x7f3f240cd300 legacy=0x7f3f2415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:16.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.954+0000 7f3f29200640 1 -- 172.21.15.6:0/208563352 >> v1:172.21.15.6:6789/0 conn(0x7f3f24154980 legacy=0x7f3f240ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:16.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.954+0000 7f3f29200640 1 -- 172.21.15.6:0/208563352 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f240cc940 con 0x7f3f2415b0c0 2024-03-22T13:01:16.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.959+0000 7f3f29200640 1 --2- 172.21.15.6:0/208563352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f2415b0c0 0x7f3f240ceea0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0097a0 tx=0x7f3f1c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d6912a744f09bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:16.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.963+0000 7f3f22a00640 1 -- 172.21.15.6:0/208563352 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c034070 con 0x7f3f2415b0c0 2024-03-22T13:01:16.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.964+0000 7f3f22a00640 1 -- 172.21.15.6:0/208563352 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c030610 con 0x7f3f2415b0c0 2024-03-22T13:01:16.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.964+0000 7f3f22a00640 1 -- 172.21.15.6:0/208563352 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c0309e0 con 0x7f3f2415b0c0 2024-03-22T13:01:16.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.964+0000 7f3f2b200640 1 -- 172.21.15.6:0/208563352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f2415b0c0 msgr2=0x7f3f240ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:16.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.964+0000 7f3f2b200640 1 --2- 172.21.15.6:0/208563352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f2415b0c0 0x7f3f240ceea0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0097a0 tx=0x7f3f1c02b960 comp rx=0 tx=0).stop 2024-03-22T13:01:16.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.964+0000 7f3f2b200640 1 -- 172.21.15.6:0/208563352 shutdown_connections 2024-03-22T13:01:16.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.964+0000 7f3f2b200640 1 --2- 172.21.15.6:0/208563352 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f2415b0c0 0x7f3f240ceea0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:16.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.964+0000 7f3f2b200640 1 -- 172.21.15.6:0/208563352 >> 172.21.15.6:0/208563352 conn(0x7f3f240b16c0 msgr2=0x7f3f240b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:16.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.965+0000 7f3f2b200640 1 -- 172.21.15.6:0/208563352 shutdown_connections 2024-03-22T13:01:16.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.965+0000 7f3f2b200640 1 -- 172.21.15.6:0/208563352 wait complete. 2024-03-22T13:01:16.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.966+0000 7f3f2b200640 1 Processor -- start 2024-03-22T13:01:16.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.966+0000 7f3f2b200640 1 -- start start 2024-03-22T13:01:16.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.967+0000 7f3f2b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f240cd300 0x7f3f240c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:16.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.967+0000 7f3f2b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f24154980 0x7f3f240c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:16.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.967+0000 7f3f2b200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f2415b0c0 0x7f3f24159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:16.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.967+0000 7f3f2b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240cea10 con 0x7f3f24154980 2024-03-22T13:01:16.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.967+0000 7f3f2b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240ccf00 con 0x7f3f240cd300 2024-03-22T13:01:16.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.967+0000 7f3f2b200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240ceb90 con 0x7f3f2415b0c0 2024-03-22T13:01:16.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.968+0000 7f3f23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f24154980 0x7f3f240c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:16.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.968+0000 7f3f29200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f2415b0c0 0x7f3f24159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:16.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.968+0000 7f3f23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f24154980 0x7f3f240c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35060/0 (socket says 172.21.15.6:35060) 2024-03-22T13:01:16.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.968+0000 7f3f23400640 1 -- 172.21.15.6:0/2164683955 learned_addr learned my addr 172.21.15.6:0/2164683955 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:16.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.972+0000 7f3f23400640 1 -- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f2415b0c0 msgr2=0x7f3f24159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:16.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.972+0000 7f3f23400640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f2415b0c0 0x7f3f24159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:16.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.972+0000 7f3f23400640 1 -- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f240cd300 msgr2=0x7f3f240c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:16.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.972+0000 7f3f23400640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f240cd300 0x7f3f240c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:16.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.972+0000 7f3f23400640 1 -- 172.21.15.6:0/2164683955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2415c0f0 con 0x7f3f24154980 2024-03-22T13:01:16.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.975+0000 7f3f23400640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f24154980 0x7f3f240c9150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00c500 tx=0x7f3f0c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:16.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.980+0000 7f3f1be00640 1 -- 172.21.15.6:0/2164683955 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c017070 con 0x7f3f24154980 2024-03-22T13:01:16.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.980+0000 7f3f1be00640 1 -- 172.21.15.6:0/2164683955 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c005ce0 con 0x7f3f24154980 2024-03-22T13:01:16.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.980+0000 7f3f1be00640 1 -- 172.21.15.6:0/2164683955 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c0072f0 con 0x7f3f24154980 2024-03-22T13:01:16.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.980+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2415b660 con 0x7f3f24154980 2024-03-22T13:01:16.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.981+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f241f3d90 con 0x7f3f24154980 2024-03-22T13:01:16.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:16.982+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f240756c0 con 0x7f3f24154980 2024-03-22T13:01:17.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.278+0000 7f3f1be00640 1 -- 172.21.15.6:0/2164683955 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f0c01a070 con 0x7f3f24154980 2024-03-22T13:01:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.282+0000 7f3f1be00640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f0006bc20 0x7f3f0006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.283+0000 7f3f23e00640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f0006bc20 0x7f3f0006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.284+0000 7f3f23e00640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f0006bc20 0x7f3f0006e0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f140012e0 tx=0x7f3f14007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.310+0000 7f3f1be00640 1 -- 172.21.15.6:0/2164683955 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3f0c056bc0 con 0x7f3f24154980 2024-03-22T13:01:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.310+0000 7f3f1be00640 1 -- 172.21.15.6:0/2164683955 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f2415c0f0 con 0x7f3f24154980 2024-03-22T13:01:17.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.596+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f241f4070 con 0x7f3f24154980 2024-03-22T13:01:17.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.617+0000 7f3f1be00640 1 -- 172.21.15.6:0/2164683955 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5980 (secure 0 0 0) 0x7f3f0c058b20 con 0x7f3f24154980 2024-03-22T13:01:17.617 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:17.618 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T12:58:20.444157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:01:17.618 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-03-22T13:01:17.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f0006bc20 msgr2=0x7f3f0006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:17.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f0006bc20 0x7f3f0006e0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f140012e0 tx=0x7f3f14007d50 comp rx=0 tx=0).stop 2024-03-22T13:01:17.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f24154980 msgr2=0x7f3f240c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f24154980 0x7f3f240c9150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00c500 tx=0x7f3f0c00c9c0 comp rx=0 tx=0).stop 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 shutdown_connections 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f2415b0c0 0x7f3f24159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f24154980 0x7f3f240c9150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f0006bc20 0x7f3f0006e0e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 --2- 172.21.15.6:0/2164683955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f240cd300 0x7f3f240c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 >> 172.21.15.6:0/2164683955 conn(0x7f3f240b16c0 msgr2=0x7f3f24153a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 shutdown_connections 2024-03-22T13:01:17.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:17.620+0000 7f3f2b200640 1 -- 172.21.15.6:0/2164683955 wait complete. 2024-03-22T13:01:18.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T12:58:20.444157+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:01:18.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:19.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:19.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.445+0000 7fe457200640 1 Processor -- start 2024-03-22T13:01:19.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.445+0000 7fe457200640 1 -- start start 2024-03-22T13:01:19.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.446+0000 7fe457200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe45806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:19.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.446+0000 7fe457200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe45815ae30 0x7fe4580cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:19.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.446+0000 7fe457200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45815b260 con 0x7fe4580ce8a0 2024-03-22T13:01:19.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.446+0000 7fe457200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4580cee00 con 0x7fe45815ae30 2024-03-22T13:01:19.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.446+0000 7fe457200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4580cf0c0 con 0x7fe4580cc380 2024-03-22T13:01:19.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.447+0000 7fe44d400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe4580ce8a0 0x7fe4580cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60620/0 (socket says 172.21.15.6:60620) 2024-03-22T13:01:19.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.447+0000 7fe44d400640 1 -- 172.21.15.6:0/2987000983 learned_addr learned my addr 172.21.15.6:0/2987000983 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:19.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.447+0000 7fe455e00640 1 --2- 172.21.15.6:0/2987000983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe45806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:19.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.447+0000 7fe456800640 1 --2- 172.21.15.6:0/2987000983 >> v2:172.21.15.6:3300/0 conn(0x7fe45815ae30 0x7fe4580cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.450+0000 7fe455e00640 1 -- 172.21.15.6:0/2987000983 >> v2:172.21.15.6:3300/0 conn(0x7fe45815ae30 msgr2=0x7fe4580cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.450+0000 7fe455e00640 1 --2- 172.21.15.6:0/2987000983 >> v2:172.21.15.6:3300/0 conn(0x7fe45815ae30 0x7fe4580cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.450+0000 7fe455e00640 1 -- 172.21.15.6:0/2987000983 >> v1:172.21.15.92:6789/0 conn(0x7fe4580ce8a0 legacy=0x7fe4580cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.450+0000 7fe455e00640 1 -- 172.21.15.6:0/2987000983 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4580cf3a0 con 0x7fe4580cc380 2024-03-22T13:01:19.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.453+0000 7fe455e00640 1 --2- 172.21.15.6:0/2987000983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe45806c230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe448009d90 tx=0x7fe44802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=978c4ee09aee1563 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.457+0000 7fe455400640 1 -- 172.21.15.6:0/2987000983 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448039070 con 0x7fe4580cc380 2024-03-22T13:01:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.457+0000 7fe455400640 1 -- 172.21.15.6:0/2987000983 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44802cc50 con 0x7fe4580cc380 2024-03-22T13:01:19.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.457+0000 7fe455400640 1 -- 172.21.15.6:0/2987000983 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448034440 con 0x7fe4580cc380 2024-03-22T13:01:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 -- 172.21.15.6:0/2987000983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 msgr2=0x7fe45806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 --2- 172.21.15.6:0/2987000983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe45806c230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe448009d90 tx=0x7fe44802bea0 comp rx=0 tx=0).stop 2024-03-22T13:01:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 -- 172.21.15.6:0/2987000983 shutdown_connections 2024-03-22T13:01:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 --2- 172.21.15.6:0/2987000983 >> v2:172.21.15.6:3300/0 conn(0x7fe45815ae30 0x7fe4580cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 --2- 172.21.15.6:0/2987000983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe45806c230 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 -- 172.21.15.6:0/2987000983 >> 172.21.15.6:0/2987000983 conn(0x7fe4580b1480 msgr2=0x7fe4580b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:19.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 -- 172.21.15.6:0/2987000983 shutdown_connections 2024-03-22T13:01:19.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.458+0000 7fe457200640 1 -- 172.21.15.6:0/2987000983 wait complete. 2024-03-22T13:01:19.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.461+0000 7fe457200640 1 Processor -- start 2024-03-22T13:01:19.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.461+0000 7fe457200640 1 -- start start 2024-03-22T13:01:19.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe457200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe458153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:19.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe457200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4580ce8a0 0x7fe4581e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:19.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe457200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe45815ae30 0x7fe4581e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:19.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe457200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45815a910 con 0x7fe4580ce8a0 2024-03-22T13:01:19.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe457200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45815a630 con 0x7fe45815ae30 2024-03-22T13:01:19.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe457200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4581549d0 con 0x7fe4580cc380 2024-03-22T13:01:19.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe455e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe458153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:19.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.463+0000 7fe44d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4580ce8a0 0x7fe4581e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:19.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.464+0000 7fe456800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe45815ae30 0x7fe4581e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:19.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.464+0000 7fe456800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe45815ae30 0x7fe4581e5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50892/0 (socket says 172.21.15.6:50892) 2024-03-22T13:01:19.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.464+0000 7fe456800640 1 -- 172.21.15.6:0/143689889 learned_addr learned my addr 172.21.15.6:0/143689889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:19.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.464+0000 7fe455e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe458153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:39362/0 (socket says 172.21.15.6:39362) 2024-03-22T13:01:19.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.469+0000 7fe455e00640 1 -- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe45815ae30 msgr2=0x7fe4581e5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:19.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.469+0000 7fe455e00640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe45815ae30 0x7fe4581e5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:19.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.469+0000 7fe455e00640 1 -- 172.21.15.6:0/143689889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4580ce8a0 msgr2=0x7fe4581e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:19.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.469+0000 7fe455e00640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4580ce8a0 0x7fe4581e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:19.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.469+0000 7fe455e00640 1 -- 172.21.15.6:0/143689889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45815de80 con 0x7fe4580cc380 2024-03-22T13:01:19.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.469+0000 7fe456800640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe45815ae30 0x7fe4581e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:01:19.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.473+0000 7fe455e00640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe458153a60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe4580cf240 tx=0x7fe448004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:19.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.476+0000 7fe44ea00640 1 -- 172.21.15.6:0/143689889 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe448039050 con 0x7fe4580cc380 2024-03-22T13:01:19.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.477+0000 7fe44ea00640 1 -- 172.21.15.6:0/143689889 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe448045070 con 0x7fe4580cc380 2024-03-22T13:01:19.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.477+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4581e3650 con 0x7fe4580cc380 2024-03-22T13:01:19.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.477+0000 7fe44ea00640 1 -- 172.21.15.6:0/143689889 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44803e610 con 0x7fe4580cc380 2024-03-22T13:01:19.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.477+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45815d300 con 0x7fe4580cc380 2024-03-22T13:01:19.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.478+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe458075650 con 0x7fe4580cc380 2024-03-22T13:01:19.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.628+0000 7fe44ea00640 1 -- 172.21.15.6:0/143689889 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe44803e7b0 con 0x7fe4580cc380 2024-03-22T13:01:19.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.629+0000 7fe44ea00640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe43406bc20 0x7fe43406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:19.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.630+0000 7fe44ea00640 1 -- 172.21.15.6:0/143689889 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe44807a300 con 0x7fe4580cc380 2024-03-22T13:01:19.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.630+0000 7fe44d400640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe43406bc20 0x7fe43406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:19.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.631+0000 7fe44d400640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe43406bc20 0x7fe43406e0e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe4580b57c0 tx=0x7fe440007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:19.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:19.773+0000 7fe44ea00640 1 -- 172.21.15.6:0/143689889 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe44807c260 con 0x7fe4580cc380 2024-03-22T13:01:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.057+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe45815e680 con 0x7fe4580cc380 2024-03-22T13:01:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.137+0000 7fe44ea00640 1 -- 172.21.15.6:0/143689889 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fe448080300 con 0x7fe4580cc380 2024-03-22T13:01:20.137 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:20.137 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:20.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe43406bc20 msgr2=0x7fe43406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:20.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe43406bc20 0x7fe43406e0e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe4580b57c0 tx=0x7fe440007e80 comp rx=0 tx=0).stop 2024-03-22T13:01:20.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 msgr2=0x7fe458153a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:20.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe458153a60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe4580cf240 tx=0x7fe448004a40 comp rx=0 tx=0).stop 2024-03-22T13:01:20.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 shutdown_connections 2024-03-22T13:01:20.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe43406bc20 0x7fe43406e0e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:20.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe45815ae30 0x7fe4581e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:20.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4580ce8a0 0x7fe4581e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:20.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 --2- 172.21.15.6:0/143689889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4580cc380 0x7fe458153a60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:20.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 >> 172.21.15.6:0/143689889 conn(0x7fe4580b1480 msgr2=0x7fe4580b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:20.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 shutdown_connections 2024-03-22T13:01:20.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:20.140+0000 7fe457200640 1 -- 172.21.15.6:0/143689889 wait complete. 2024-03-22T13:01:20.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:20.765 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:21.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:21.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.957+0000 7eff9ee00640 1 Processor -- start 2024-03-22T13:01:21.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.958+0000 7eff9ee00640 1 -- start start 2024-03-22T13:01:21.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.959+0000 7eff9ee00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7eff980c68c0 0x7eff9815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:21.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.959+0000 7eff9ee00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980c4740 con 0x7eff980c7ee0 2024-03-22T13:01:21.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.959+0000 7eff9ee00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff98154620 con 0x7eff98154240 2024-03-22T13:01:21.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.959+0000 7eff9ee00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9815e130 con 0x7eff980c68c0 2024-03-22T13:01:21.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.960+0000 7eff97400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7eff980c7ee0 0x7eff9806b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55576/0 (socket says 172.21.15.6:55576) 2024-03-22T13:01:21.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.960+0000 7eff97400640 1 -- 172.21.15.6:0/2715306474 learned_addr learned my addr 172.21.15.6:0/2715306474 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:21.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.960+0000 7eff9ce00640 1 --2- 172.21.15.6:0/2715306474 >> v2:172.21.15.92:3300/0 conn(0x7eff980c68c0 0x7eff9815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:21.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.964+0000 7eff96a00640 1 -- 172.21.15.6:0/2715306474 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105251068 0 0) 0x7eff980c4740 con 0x7eff980c7ee0 2024-03-22T13:01:21.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.964+0000 7eff96a00640 1 -- 172.21.15.6:0/2715306474 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff7c003520 con 0x7eff980c7ee0 2024-03-22T13:01:21.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.964+0000 7eff9ce00640 1 -- 172.21.15.6:0/2715306474 >> v1:172.21.15.92:6789/0 conn(0x7eff98154240 legacy=0x7eff980c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:21.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.964+0000 7eff9ce00640 1 -- 172.21.15.6:0/2715306474 >> v1:172.21.15.6:6789/0 conn(0x7eff980c7ee0 legacy=0x7eff9806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:21.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.964+0000 7eff9ce00640 1 -- 172.21.15.6:0/2715306474 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff980c6e00 con 0x7eff980c68c0 2024-03-22T13:01:21.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.969+0000 7eff9ce00640 1 --2- 172.21.15.6:0/2715306474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c68c0 0x7eff9815ac90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7eff90009d90 tx=0x7eff9002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2991d9f7a6b355fb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:21.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.972+0000 7eff96a00640 1 -- 172.21.15.6:0/2715306474 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90039070 con 0x7eff980c68c0 2024-03-22T13:01:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.974+0000 7eff96a00640 1 -- 172.21.15.6:0/2715306474 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9002cd80 con 0x7eff980c68c0 2024-03-22T13:01:21.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.974+0000 7eff96a00640 1 -- 172.21.15.6:0/2715306474 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90034640 con 0x7eff980c68c0 2024-03-22T13:01:21.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.975+0000 7eff9ee00640 1 -- 172.21.15.6:0/2715306474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c68c0 msgr2=0x7eff9815ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:21.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.975+0000 7eff9ee00640 1 --2- 172.21.15.6:0/2715306474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c68c0 0x7eff9815ac90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7eff90009d90 tx=0x7eff9002bfd0 comp rx=0 tx=0).stop 2024-03-22T13:01:21.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.975+0000 7eff9ee00640 1 -- 172.21.15.6:0/2715306474 shutdown_connections 2024-03-22T13:01:21.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.975+0000 7eff9ee00640 1 --2- 172.21.15.6:0/2715306474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c68c0 0x7eff9815ac90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:21.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.975+0000 7eff9ee00640 1 -- 172.21.15.6:0/2715306474 >> 172.21.15.6:0/2715306474 conn(0x7eff980cb930 msgr2=0x7eff980cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:21.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.975+0000 7eff9ee00640 1 -- 172.21.15.6:0/2715306474 shutdown_connections 2024-03-22T13:01:21.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.975+0000 7eff9ee00640 1 -- 172.21.15.6:0/2715306474 wait complete. 2024-03-22T13:01:21.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.977+0000 7eff9ee00640 1 Processor -- start 2024-03-22T13:01:21.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.977+0000 7eff9ee00640 1 -- start start 2024-03-22T13:01:21.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.978+0000 7eff9ee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff980c68c0 0x7eff98153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:21.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.978+0000 7eff9ee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c7ee0 0x7eff981f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:21.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.978+0000 7eff9ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff98154240 0x7eff981f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:21.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.978+0000 7eff9ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9815dae0 con 0x7eff98154240 2024-03-22T13:01:21.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.978+0000 7eff9ee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9815d800 con 0x7eff980c7ee0 2024-03-22T13:01:21.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.978+0000 7eff9ee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9815ddc0 con 0x7eff980c68c0 2024-03-22T13:01:21.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.979+0000 7eff9ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff98154240 0x7eff981f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:21.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.979+0000 7eff97400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c7ee0 0x7eff981f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:21.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.979+0000 7eff97e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff980c68c0 0x7eff98153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:21.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.979+0000 7eff9ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff98154240 0x7eff981f09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35098/0 (socket says 172.21.15.6:35098) 2024-03-22T13:01:21.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.979+0000 7eff9ce00640 1 -- 172.21.15.6:0/167000670 learned_addr learned my addr 172.21.15.6:0/167000670 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:21.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.983+0000 7eff97400640 1 -- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff980c68c0 msgr2=0x7eff98153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:21.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.983+0000 7eff97400640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff980c68c0 0x7eff98153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:21.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.983+0000 7eff97400640 1 -- 172.21.15.6:0/167000670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff98154240 msgr2=0x7eff981f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:21.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.983+0000 7eff97400640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff98154240 0x7eff981f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:21.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.983+0000 7eff97400640 1 -- 172.21.15.6:0/167000670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98157080 con 0x7eff980c7ee0 2024-03-22T13:01:21.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.985+0000 7eff97400640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c7ee0 0x7eff981f0490 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c430 tx=0x7eff8400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:21.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.987+0000 7eff8fe00640 1 -- 172.21.15.6:0/167000670 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84017070 con 0x7eff980c7ee0 2024-03-22T13:01:21.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.988+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff98157360 con 0x7eff980c7ee0 2024-03-22T13:01:21.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.988+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff98157920 con 0x7eff980c7ee0 2024-03-22T13:01:21.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:21.989+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff98159230 con 0x7eff980c7ee0 2024-03-22T13:01:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.193+0000 7eff8fe00640 1 -- 172.21.15.6:0/167000670 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff84005ce0 con 0x7eff980c7ee0 2024-03-22T13:01:22.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.273+0000 7eff8fe00640 1 -- 172.21.15.6:0/167000670 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84021060 con 0x7eff980c7ee0 2024-03-22T13:01:22.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.275+0000 7eff8fe00640 1 -- 172.21.15.6:0/167000670 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7eff8401a070 con 0x7eff980c7ee0 2024-03-22T13:01:22.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.277+0000 7eff8fe00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7eff7806bbd0 0x7eff7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:22.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.278+0000 7eff8fe00640 1 -- 172.21.15.6:0/167000670 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7eff8408ab00 con 0x7eff980c7ee0 2024-03-22T13:01:22.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.278+0000 7eff97e00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7eff7806bbd0 0x7eff7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:22.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.279+0000 7eff97e00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7eff7806bbd0 0x7eff7806e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7eff880016e0 tx=0x7eff88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:22.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.288+0000 7eff8fe00640 1 -- 172.21.15.6:0/167000670 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7eff84058a80 con 0x7eff980c7ee0 2024-03-22T13:01:22.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.533+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff980c7ab0 con 0x7eff980c7ee0 2024-03-22T13:01:22.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.548+0000 7eff8fe00640 1 -- 172.21.15.6:0/167000670 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7eff8405cb20 con 0x7eff980c7ee0 2024-03-22T13:01:22.548 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:22.548 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:22.548 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.551+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7eff7806bbd0 msgr2=0x7eff7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.551+0000 7eff9ee00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7eff7806bbd0 0x7eff7806e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7eff880016e0 tx=0x7eff88009040 comp rx=0 tx=0).stop 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.551+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c7ee0 msgr2=0x7eff981f0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.551+0000 7eff9ee00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c7ee0 0x7eff981f0490 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c430 tx=0x7eff8400c8f0 comp rx=0 tx=0).stop 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 shutdown_connections 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7eff98154240 0x7eff981f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7eff7806bbd0 0x7eff7806e090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff980c7ee0 0x7eff981f0490 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 --2- 172.21.15.6:0/167000670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7eff980c68c0 0x7eff98153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 >> 172.21.15.6:0/167000670 conn(0x7eff980cb930 msgr2=0x7eff980cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 shutdown_connections 2024-03-22T13:01:22.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:22.552+0000 7eff9ee00640 1 -- 172.21.15.6:0/167000670 wait complete. 2024-03-22T13:01:23.183 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:23.183 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:24.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:24.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.401+0000 7fdfb7200640 1 Processor -- start 2024-03-22T13:01:24.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.402+0000 7fdfb7200640 1 -- start start 2024-03-22T13:01:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.403+0000 7fdfb7200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdfb0152400 0x7fdfb015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.403+0000 7fdfb7200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb00716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.403+0000 7fdfb7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdfb006f8c0 0x7fdfb0071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.403+0000 7fdfb7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb00724d0 con 0x7fdfb006f8c0 2024-03-22T13:01:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.403+0000 7fdfb7200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb01590b0 con 0x7fdfb0152400 2024-03-22T13:01:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.403+0000 7fdfb7200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb00721f0 con 0x7fdfb015bc80 2024-03-22T13:01:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.404+0000 7fdfaf400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb00716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.404+0000 7fdfafe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdfb0152400 0x7fdfb015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.405+0000 7fdfaf400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb00716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53504/0 (socket says 172.21.15.6:53504) 2024-03-22T13:01:24.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.405+0000 7fdfaf400640 1 -- 172.21.15.6:0/1363134592 learned_addr learned my addr 172.21.15.6:0/1363134592 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.407+0000 7fdfafe00640 1 -- 172.21.15.6:0/1363134592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 msgr2=0x7fdfb00716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.407+0000 7fdfafe00640 1 --2- 172.21.15.6:0/1363134592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb00716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.407+0000 7fdfafe00640 1 -- 172.21.15.6:0/1363134592 >> v2:172.21.15.6:3300/0 conn(0x7fdfb006f8c0 msgr2=0x7fdfb0071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.407+0000 7fdfafe00640 1 --2- 172.21.15.6:0/1363134592 >> v2:172.21.15.6:3300/0 conn(0x7fdfb006f8c0 0x7fdfb0071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.407+0000 7fdfafe00640 1 -- 172.21.15.6:0/1363134592 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb01582f0 con 0x7fdfb0152400 2024-03-22T13:01:24.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.411+0000 7fdfafe00640 1 --2- 172.21.15.6:0/1363134592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb0152400 0x7fdfb015b740 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0071390 tx=0x7fdfa002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49258850599ba8c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:24.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.415+0000 7fdfaea00640 1 -- 172.21.15.6:0/1363134592 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0035020 con 0x7fdfb0152400 2024-03-22T13:01:24.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.477+0000 7fdfaea00640 1 -- 172.21.15.6:0/1363134592 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa0031830 con 0x7fdfb0152400 2024-03-22T13:01:24.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.477+0000 7fdfaea00640 1 -- 172.21.15.6:0/1363134592 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0031c00 con 0x7fdfb0152400 2024-03-22T13:01:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.478+0000 7fdfb7200640 1 -- 172.21.15.6:0/1363134592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb0152400 msgr2=0x7fdfb015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.478+0000 7fdfb7200640 1 --2- 172.21.15.6:0/1363134592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb0152400 0x7fdfb015b740 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0071390 tx=0x7fdfa002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.478+0000 7fdfb7200640 1 -- 172.21.15.6:0/1363134592 shutdown_connections 2024-03-22T13:01:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.478+0000 7fdfb7200640 1 --2- 172.21.15.6:0/1363134592 >> v2:172.21.15.6:3300/0 conn(0x7fdfb006f8c0 0x7fdfb0071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.478+0000 7fdfb7200640 1 --2- 172.21.15.6:0/1363134592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb00716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.478+0000 7fdfb7200640 1 --2- 172.21.15.6:0/1363134592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb0152400 0x7fdfb015b740 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.478+0000 7fdfb7200640 1 -- 172.21.15.6:0/1363134592 >> 172.21.15.6:0/1363134592 conn(0x7fdfb00cb930 msgr2=0x7fdfb00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.479+0000 7fdfb7200640 1 -- 172.21.15.6:0/1363134592 shutdown_connections 2024-03-22T13:01:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.479+0000 7fdfb7200640 1 -- 172.21.15.6:0/1363134592 wait complete. 2024-03-22T13:01:24.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.481+0000 7fdfb7200640 1 Processor -- start 2024-03-22T13:01:24.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.481+0000 7fdfb7200640 1 -- start start 2024-03-22T13:01:24.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.482+0000 7fdfb7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb006f8c0 0x7fdfb01f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:24.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.482+0000 7fdfb7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfb0152400 0x7fdfb015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:24.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.482+0000 7fdfb7200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb01482f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:24.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.482+0000 7fdfb7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0158b90 con 0x7fdfb0152400 2024-03-22T13:01:24.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.482+0000 7fdfb7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb01588b0 con 0x7fdfb006f8c0 2024-03-22T13:01:24.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.482+0000 7fdfb7200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb01527e0 con 0x7fdfb015bc80 2024-03-22T13:01:24.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.483+0000 7fdfafe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb006f8c0 0x7fdfb01f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:24.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.483+0000 7fdfaf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfb0152400 0x7fdfb015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:24.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.483+0000 7fdfafe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb006f8c0 0x7fdfb01f0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51480/0 (socket says 172.21.15.6:51480) 2024-03-22T13:01:24.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.483+0000 7fdfafe00640 1 -- 172.21.15.6:0/679762757 learned_addr learned my addr 172.21.15.6:0/679762757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:24.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.487+0000 7fdfafe00640 1 -- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 msgr2=0x7fdfb01482f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:24.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.487+0000 7fdfafe00640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb01482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.487+0000 7fdfafe00640 1 -- 172.21.15.6:0/679762757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfb0152400 msgr2=0x7fdfb015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:24.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.487+0000 7fdfafe00640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfb0152400 0x7fdfb015a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:24.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.487+0000 7fdfafe00640 1 -- 172.21.15.6:0/679762757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb014b9d0 con 0x7fdfb006f8c0 2024-03-22T13:01:24.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.490+0000 7fdfafe00640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb006f8c0 0x7fdfb01f0cd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdfa002bee0 tx=0x7fdfa00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:24.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.494+0000 7fdfa7e00640 1 -- 172.21.15.6:0/679762757 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0041070 con 0x7fdfb006f8c0 2024-03-22T13:01:24.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.494+0000 7fdfa7e00640 1 -- 172.21.15.6:0/679762757 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa003f070 con 0x7fdfb006f8c0 2024-03-22T13:01:24.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.494+0000 7fdfa7e00640 1 -- 172.21.15.6:0/679762757 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0035040 con 0x7fdfb006f8c0 2024-03-22T13:01:24.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.494+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb01f1210 con 0x7fdfb006f8c0 2024-03-22T13:01:24.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.494+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb014bc40 con 0x7fdfb006f8c0 2024-03-22T13:01:24.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.495+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb00756c0 con 0x7fdfb006f8c0 2024-03-22T13:01:24.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.756+0000 7fdfa7e00640 1 -- 172.21.15.6:0/679762757 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdfa0038020 con 0x7fdfb006f8c0 2024-03-22T13:01:24.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.758+0000 7fdfa7e00640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf9006bc20 0x7fdf9006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:24.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.759+0000 7fdfa7e00640 1 -- 172.21.15.6:0/679762757 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdfa00ac8c0 con 0x7fdfb006f8c0 2024-03-22T13:01:24.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.759+0000 7fdfaf400640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf9006bc20 0x7fdf9006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:24.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.760+0000 7fdfaf400640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf9006bc20 0x7fdf9006e0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0150c30 tx=0x7fdf9c009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:24.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:24.770+0000 7fdfa7e00640 1 -- 172.21.15.6:0/679762757 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdfb01f1210 con 0x7fdfb006f8c0 2024-03-22T13:01:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.021+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfb00c6400 con 0x7fdfb006f8c0 2024-03-22T13:01:25.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.047+0000 7fdfa7e00640 1 -- 172.21.15.6:0/679762757 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fdfa0031800 con 0x7fdfb006f8c0 2024-03-22T13:01:25.047 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:25.047 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:25.047 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:25.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf9006bc20 msgr2=0x7fdf9006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:25.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf9006bc20 0x7fdf9006e0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0150c30 tx=0x7fdf9c009240 comp rx=0 tx=0).stop 2024-03-22T13:01:25.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb006f8c0 msgr2=0x7fdfb01f0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:25.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb006f8c0 0x7fdfb01f0cd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdfa002bee0 tx=0x7fdfa00046c0 comp rx=0 tx=0).stop 2024-03-22T13:01:25.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 shutdown_connections 2024-03-22T13:01:25.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdfb015bc80 0x7fdfb01482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:25.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdfb0152400 0x7fdfb015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:25.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf9006bc20 0x7fdf9006e0e0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0150c30 tx=0x7fdf9c009240 comp rx=0 tx=0).stop 2024-03-22T13:01:25.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 --2- 172.21.15.6:0/679762757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfb006f8c0 0x7fdfb01f0cd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:25.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.051+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 >> 172.21.15.6:0/679762757 conn(0x7fdfb00cb930 msgr2=0x7fdfb00c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:25.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.053+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 shutdown_connections 2024-03-22T13:01:25.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:25.053+0000 7fdfb7200640 1 -- 172.21.15.6:0/679762757 wait complete. 2024-03-22T13:01:25.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:25.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:26.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:26.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.880+0000 7fade3200640 1 Processor -- start 2024-03-22T13:01:26.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.881+0000 7fade3200640 1 -- start start 2024-03-22T13:01:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.882+0000 7fade3200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fade415af60 0x7fade40cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.882+0000 7fade3200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade40cc860 0x7fade406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.882+0000 7fade3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fade40ccca0 0x7fade40cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.882+0000 7fade3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade40cc3d0 con 0x7fade40ccca0 2024-03-22T13:01:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.882+0000 7fade3200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade40ce380 con 0x7fade415af60 2024-03-22T13:01:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.882+0000 7fade3200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade40ce660 con 0x7fade40cc860 2024-03-22T13:01:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.883+0000 7fade1400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade40cc860 0x7fade406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.883+0000 7fade1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fade415af60 0x7fade40cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.883+0000 7fade1400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade40cc860 0x7fade406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53508/0 (socket says 172.21.15.6:53508) 2024-03-22T13:01:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.883+0000 7fade1400640 1 -- 172.21.15.6:0/1779393823 learned_addr learned my addr 172.21.15.6:0/1779393823 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.886+0000 7fade1e00640 1 -- 172.21.15.6:0/1779393823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade40cc860 msgr2=0x7fade406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.886+0000 7fade1e00640 1 --2- 172.21.15.6:0/1779393823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade40cc860 0x7fade406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.886+0000 7fade1e00640 1 -- 172.21.15.6:0/1779393823 >> v2:172.21.15.6:3300/0 conn(0x7fade40ccca0 msgr2=0x7fade40cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.886+0000 7fade1e00640 1 --2- 172.21.15.6:0/1779393823 >> v2:172.21.15.6:3300/0 conn(0x7fade40ccca0 0x7fade40cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.886+0000 7fade1e00640 1 -- 172.21.15.6:0/1779393823 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade40ce940 con 0x7fade415af60 2024-03-22T13:01:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.889+0000 7fade1e00640 1 --2- 172.21.15.6:0/1779393823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade415af60 0x7fade40cdb00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fade408cbc0 tx=0x7fadd402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f777f49aeadbd5c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.892+0000 7fade0a00640 1 -- 172.21.15.6:0/1779393823 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4035020 con 0x7fade415af60 2024-03-22T13:01:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.894+0000 7fade0a00640 1 -- 172.21.15.6:0/1779393823 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd4031640 con 0x7fade415af60 2024-03-22T13:01:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.894+0000 7fade0a00640 1 -- 172.21.15.6:0/1779393823 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd402d040 con 0x7fade415af60 2024-03-22T13:01:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 -- 172.21.15.6:0/1779393823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade415af60 msgr2=0x7fade40cdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 --2- 172.21.15.6:0/1779393823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade415af60 0x7fade40cdb00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fade408cbc0 tx=0x7fadd402b8f0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 -- 172.21.15.6:0/1779393823 shutdown_connections 2024-03-22T13:01:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 --2- 172.21.15.6:0/1779393823 >> v2:172.21.15.6:3300/0 conn(0x7fade40ccca0 0x7fade40cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 --2- 172.21.15.6:0/1779393823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade40cc860 0x7fade406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 --2- 172.21.15.6:0/1779393823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade415af60 0x7fade40cdb00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 -- 172.21.15.6:0/1779393823 >> 172.21.15.6:0/1779393823 conn(0x7fade40cf9d0 msgr2=0x7fade40cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 -- 172.21.15.6:0/1779393823 shutdown_connections 2024-03-22T13:01:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.895+0000 7fade3200640 1 -- 172.21.15.6:0/1779393823 wait complete. 2024-03-22T13:01:26.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.898+0000 7fade3200640 1 Processor -- start 2024-03-22T13:01:26.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.898+0000 7fade3200640 1 -- start start 2024-03-22T13:01:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.900+0000 7fade3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade40cc860 0x7fade415b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.900+0000 7fade3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fade40ccca0 0x7fade414dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.900+0000 7fade3200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade415af60 0x7fade40b1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.900+0000 7fade3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade406dd60 con 0x7fade40ccca0 2024-03-22T13:01:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.900+0000 7fade3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade415a550 con 0x7fade40cc860 2024-03-22T13:01:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.900+0000 7fade3200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade406dee0 con 0x7fade415af60 2024-03-22T13:01:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.900+0000 7fade1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade40cc860 0x7fade415b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.901+0000 7fade2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade415af60 0x7fade40b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.901+0000 7fade1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade40cc860 0x7fade415b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51508/0 (socket says 172.21.15.6:51508) 2024-03-22T13:01:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.901+0000 7fade1e00640 1 -- 172.21.15.6:0/775869554 learned_addr learned my addr 172.21.15.6:0/775869554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.905+0000 7fade1400640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fade40ccca0 0x7fade414dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.906+0000 7fade1e00640 1 -- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade415af60 msgr2=0x7fade40b1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.906+0000 7fade1e00640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade415af60 0x7fade40b1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.906+0000 7fade1e00640 1 -- 172.21.15.6:0/775869554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fade40ccca0 msgr2=0x7fade414dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.906+0000 7fade1e00640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fade40ccca0 0x7fade414dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.906+0000 7fade1e00640 1 -- 172.21.15.6:0/775869554 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade415b9e0 con 0x7fade40cc860 2024-03-22T13:01:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.911+0000 7fade1e00640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade40cc860 0x7fade415b4a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009890 tx=0x7fadd40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.918+0000 7fadda000640 1 -- 172.21.15.6:0/775869554 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd403aa60 con 0x7fade40cc860 2024-03-22T13:01:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.918+0000 7fadda000640 1 -- 172.21.15.6:0/775869554 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd403ac00 con 0x7fade40cc860 2024-03-22T13:01:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.918+0000 7fadda000640 1 -- 172.21.15.6:0/775869554 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd40443f0 con 0x7fade40cc860 2024-03-22T13:01:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.918+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade415c5a0 con 0x7fade40cc860 2024-03-22T13:01:26.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.918+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade415cb60 con 0x7fade40cc860 2024-03-22T13:01:26.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:26.919+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade4075650 con 0x7fade40cc860 2024-03-22T13:01:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.159+0000 7fadda000640 1 -- 172.21.15.6:0/775869554 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fadd4038020 con 0x7fade40cc860 2024-03-22T13:01:27.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.161+0000 7fadda000640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadc006bc20 0x7fadc006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:27.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.161+0000 7fade1400640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadc006bc20 0x7fadc006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:27.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.162+0000 7fade1400640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadc006bc20 0x7fadc006e0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fade414ec40 tx=0x7fadcc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:27.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.183+0000 7fadda000640 1 -- 172.21.15.6:0/775869554 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fadd40793e0 con 0x7fade40cc860 2024-03-22T13:01:27.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.217+0000 7fadda000640 1 -- 172.21.15.6:0/775869554 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fadd403f070 con 0x7fade40cc860 2024-03-22T13:01:27.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.475+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fade40b5950 con 0x7fade40cc860 2024-03-22T13:01:27.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.511+0000 7fadda000640 1 -- 172.21.15.6:0/775869554 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fadd407b340 con 0x7fade40cc860 2024-03-22T13:01:27.512 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:27.512 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:27.512 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:27.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadc006bc20 msgr2=0x7fadc006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:27.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadc006bc20 0x7fadc006e0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fade414ec40 tx=0x7fadcc008040 comp rx=0 tx=0).stop 2024-03-22T13:01:27.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade40cc860 msgr2=0x7fade415b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade40cc860 0x7fade415b4a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009890 tx=0x7fadd40046c0 comp rx=0 tx=0).stop 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 shutdown_connections 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade415af60 0x7fade40b1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fade40ccca0 0x7fade414dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fadc006bc20 0x7fadc006e0e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 --2- 172.21.15.6:0/775869554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade40cc860 0x7fade415b4a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.514+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 >> 172.21.15.6:0/775869554 conn(0x7fade40cf9d0 msgr2=0x7fade40ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.515+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 shutdown_connections 2024-03-22T13:01:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:27.515+0000 7fade3200640 1 -- 172.21.15.6:0/775869554 wait complete. 2024-03-22T13:01:28.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:28.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:29.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:29.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.340+0000 7f79afe00640 1 Processor -- start 2024-03-22T13:01:29.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.340+0000 7f79afe00640 1 -- start start 2024-03-22T13:01:29.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.342+0000 7f79afe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f79b00cc380 0x7f79b006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:29.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.342+0000 7f79afe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f79b00ce8a0 0x7f79b015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:29.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.342+0000 7f79afe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00cee00 con 0x7f79b00cc760 2024-03-22T13:01:29.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.342+0000 7f79afe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b00cf0c0 con 0x7f79b00cc380 2024-03-22T13:01:29.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.342+0000 7f79afe00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b00cf3a0 con 0x7f79b00ce8a0 2024-03-22T13:01:29.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.343+0000 7f79aea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f79b00cc380 0x7f79b006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:29.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.343+0000 7f79ae000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f79b00ce8a0 0x7f79b015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:29.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.343+0000 7f79aea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f79b00cc380 0x7f79b006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34280/0 (socket says 172.21.15.6:34280) 2024-03-22T13:01:29.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.343+0000 7f79aea00640 1 -- 172.21.15.6:0/816199633 learned_addr learned my addr 172.21.15.6:0/816199633 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:29.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.346+0000 7f79aea00640 1 -- 172.21.15.6:0/816199633 >> v2:172.21.15.92:3300/0 conn(0x7f79b00ce8a0 msgr2=0x7f79b015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.346+0000 7f79aea00640 1 --2- 172.21.15.6:0/816199633 >> v2:172.21.15.92:3300/0 conn(0x7f79b00ce8a0 0x7f79b015ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.346+0000 7f79aea00640 1 -- 172.21.15.6:0/816199633 >> v1:172.21.15.92:6789/0 conn(0x7f79b00cc760 legacy=0x7f79b00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:29.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.346+0000 7f79aea00640 1 -- 172.21.15.6:0/816199633 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b00cf680 con 0x7f79b00cc380 2024-03-22T13:01:29.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.348+0000 7f79aea00640 1 --2- 172.21.15.6:0/816199633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00cc380 0x7f79b006c230 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f79b00cb7f0 tx=0x7f79a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a570ca908a453629 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.353+0000 7f79ad600640 1 -- 172.21.15.6:0/816199633 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0034070 con 0x7f79b00cc380 2024-03-22T13:01:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.353+0000 7f79ad600640 1 -- 172.21.15.6:0/816199633 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0030860 con 0x7f79b00cc380 2024-03-22T13:01:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.353+0000 7f79ad600640 1 -- 172.21.15.6:0/816199633 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0030c30 con 0x7f79b00cc380 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 -- 172.21.15.6:0/816199633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00cc380 msgr2=0x7f79b006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 --2- 172.21.15.6:0/816199633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00cc380 0x7f79b006c230 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f79b00cb7f0 tx=0x7f79a002b9a0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 -- 172.21.15.6:0/816199633 shutdown_connections 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 --2- 172.21.15.6:0/816199633 >> v2:172.21.15.92:3300/0 conn(0x7f79b00ce8a0 0x7f79b015ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 --2- 172.21.15.6:0/816199633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00cc380 0x7f79b006c230 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 -- 172.21.15.6:0/816199633 >> 172.21.15.6:0/816199633 conn(0x7f79b00b1480 msgr2=0x7f79b00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 -- 172.21.15.6:0/816199633 shutdown_connections 2024-03-22T13:01:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.354+0000 7f79afe00640 1 -- 172.21.15.6:0/816199633 wait complete. 2024-03-22T13:01:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.357+0000 7f79afe00640 1 Processor -- start 2024-03-22T13:01:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.357+0000 7f79afe00640 1 -- start start 2024-03-22T13:01:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.358+0000 7f79afe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79b00cc380 0x7f79b00b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.358+0000 7f79afe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79b00cc760 0x7f79b00b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.358+0000 7f79afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00ce8a0 0x7f79b00b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.358+0000 7f79afe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b01549d0 con 0x7f79b00ce8a0 2024-03-22T13:01:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.358+0000 7f79afe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b015a910 con 0x7f79b00cc380 2024-03-22T13:01:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.358+0000 7f79afe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00ccb70 con 0x7f79b00cc760 2024-03-22T13:01:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.359+0000 7f79aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79b00cc380 0x7f79b00b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.359+0000 7f79af400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00ce8a0 0x7f79b00b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.359+0000 7f79ae000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79b00cc760 0x7f79b00b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.360+0000 7f79aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79b00cc380 0x7f79b00b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51522/0 (socket says 172.21.15.6:51522) 2024-03-22T13:01:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.360+0000 7f79aea00640 1 -- 172.21.15.6:0/2388817115 learned_addr learned my addr 172.21.15.6:0/2388817115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.365+0000 7f79aea00640 1 -- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79b00cc760 msgr2=0x7f79b00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.365+0000 7f79aea00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79b00cc760 0x7f79b00b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.365+0000 7f79aea00640 1 -- 172.21.15.6:0/2388817115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00ce8a0 msgr2=0x7f79b00b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.365+0000 7f79aea00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00ce8a0 0x7f79b00b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.365+0000 7f79aea00640 1 -- 172.21.15.6:0/2388817115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b015d720 con 0x7f79b00cc380 2024-03-22T13:01:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.365+0000 7f79af400640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00ce8a0 0x7f79b00b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:01:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.369+0000 7f79aea00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79b00cc380 0x7f79b00b4ba0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f79a002bfb0 tx=0x7f79a0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:29.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.373+0000 7f79a6a00640 1 -- 172.21.15.6:0/2388817115 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0048070 con 0x7f79b00cc380 2024-03-22T13:01:29.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.373+0000 7f79a6a00640 1 -- 172.21.15.6:0/2388817115 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0030660 con 0x7f79b00cc380 2024-03-22T13:01:29.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.373+0000 7f79a6a00640 1 -- 172.21.15.6:0/2388817115 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0034040 con 0x7f79b00cc380 2024-03-22T13:01:29.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.373+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b00b3320 con 0x7f79b00cc380 2024-03-22T13:01:29.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.374+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b00b38e0 con 0x7f79b00cc380 2024-03-22T13:01:29.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.375+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b0075650 con 0x7f79b00cc380 2024-03-22T13:01:29.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.407+0000 7f79a6a00640 1 -- 172.21.15.6:0/2388817115 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f79a004b070 con 0x7f79b00cc380 2024-03-22T13:01:29.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.409+0000 7f79a6a00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f799006bc20 0x7f799006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:29.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.410+0000 7f79ae000640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f799006bc20 0x7f799006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:29.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.410+0000 7f79ae000640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f799006bc20 0x7f799006e0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f79b00c9f20 tx=0x7f799c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:29.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.412+0000 7f79a6a00640 1 -- 172.21.15.6:0/2388817115 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f79a0030810 con 0x7f79b00cc380 2024-03-22T13:01:29.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.672+0000 7f79a6a00640 1 -- 172.21.15.6:0/2388817115 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f79a0052020 con 0x7f79b00cc380 2024-03-22T13:01:29.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.945+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b015e680 con 0x7f79b00cc380 2024-03-22T13:01:29.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.960+0000 7f79a6a00640 1 -- 172.21.15.6:0/2388817115 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f79b015d720 con 0x7f79b00cc380 2024-03-22T13:01:29.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:29.960 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:29.961 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:29.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f799006bc20 msgr2=0x7f799006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:29.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f799006bc20 0x7f799006e0e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f79b00c9f20 tx=0x7f799c009190 comp rx=0 tx=0).stop 2024-03-22T13:01:29.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79b00cc380 msgr2=0x7f79b00b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:29.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79b00cc380 0x7f79b00b4ba0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f79a002bfb0 tx=0x7f79a0004730 comp rx=0 tx=0).stop 2024-03-22T13:01:29.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 shutdown_connections 2024-03-22T13:01:29.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79b00ce8a0 0x7f79b00b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79b00cc760 0x7f79b00b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f799006bc20 0x7f799006e0e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 --2- 172.21.15.6:0/2388817115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79b00cc380 0x7f79b00b4ba0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:29.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 >> 172.21.15.6:0/2388817115 conn(0x7f79b00b1480 msgr2=0x7f79b0157e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:29.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 shutdown_connections 2024-03-22T13:01:29.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:29.963+0000 7f79afe00640 1 -- 172.21.15.6:0/2388817115 wait complete. 2024-03-22T13:01:30.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:30.583 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:30.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:01:30.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:01:31.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.808+0000 7f9415c00640 1 Processor -- start 2024-03-22T13:01:31.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.808+0000 7f9415c00640 1 -- start start 2024-03-22T13:01:31.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.810+0000 7f9415c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f94100ca520 0x7f941006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:31.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.810+0000 7f9415c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f941015b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:31.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.810+0000 7f9415c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f94100ca900 0x7f94100cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:31.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.810+0000 7f9415c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94100cd000 con 0x7f94100ca900 2024-03-22T13:01:31.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.810+0000 7f9415c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941006c7a0 con 0x7f94100ca520 2024-03-22T13:01:31.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.810+0000 7f9415c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941006ca80 con 0x7f941015b0e0 2024-03-22T13:01:31.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.811+0000 7f940e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f941015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:31.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.811+0000 7f940e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f941015b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53544/0 (socket says 172.21.15.6:53544) 2024-03-22T13:01:31.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.811+0000 7f940e000640 1 -- 172.21.15.6:0/166957281 learned_addr learned my addr 172.21.15.6:0/166957281 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:31.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.811+0000 7f940f400640 1 --2- 172.21.15.6:0/166957281 >> v2:172.21.15.6:3300/0 conn(0x7f94100ca900 0x7f94100cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:31.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.814+0000 7f940f400640 1 -- 172.21.15.6:0/166957281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 msgr2=0x7f941015b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:31.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.814+0000 7f940f400640 1 --2- 172.21.15.6:0/166957281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f941015b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:31.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.814+0000 7f940f400640 1 -- 172.21.15.6:0/166957281 >> v2:172.21.15.92:3300/0 conn(0x7f94100ca520 msgr2=0x7f941006d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:31.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.814+0000 7f940f400640 1 --2- 172.21.15.6:0/166957281 >> v2:172.21.15.92:3300/0 conn(0x7f94100ca520 0x7f941006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:31.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.814+0000 7f940f400640 1 -- 172.21.15.6:0/166957281 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941006cd60 con 0x7f94100ca900 2024-03-22T13:01:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.817+0000 7f940f400640 1 --2- 172.21.15.6:0/166957281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 0x7f94100cc3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94100d0ca0 tx=0x7f940802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3927fe6e7cc2d55e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:31.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.826+0000 7f940d600640 1 -- 172.21.15.6:0/166957281 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408039070 con 0x7f94100ca900 2024-03-22T13:01:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.826+0000 7f940d600640 1 -- 172.21.15.6:0/166957281 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940802cbd0 con 0x7f94100ca900 2024-03-22T13:01:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.826+0000 7f940d600640 1 -- 172.21.15.6:0/166957281 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408034440 con 0x7f94100ca900 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 -- 172.21.15.6:0/166957281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 msgr2=0x7f94100cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 --2- 172.21.15.6:0/166957281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 0x7f94100cc3c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94100d0ca0 tx=0x7f940802be20 comp rx=0 tx=0).stop 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 -- 172.21.15.6:0/166957281 shutdown_connections 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 --2- 172.21.15.6:0/166957281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 0x7f94100cc3c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 --2- 172.21.15.6:0/166957281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f941015b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 --2- 172.21.15.6:0/166957281 >> v2:172.21.15.92:3300/0 conn(0x7f94100ca520 0x7f941006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 -- 172.21.15.6:0/166957281 >> 172.21.15.6:0/166957281 conn(0x7f94100b16b0 msgr2=0x7f94100b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 -- 172.21.15.6:0/166957281 shutdown_connections 2024-03-22T13:01:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.827+0000 7f9415c00640 1 -- 172.21.15.6:0/166957281 wait complete. 2024-03-22T13:01:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.829+0000 7f9415c00640 1 Processor -- start 2024-03-22T13:01:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.829+0000 7f9415c00640 1 -- start start 2024-03-22T13:01:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.830+0000 7f9415c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94100ca520 0x7f94101578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.830+0000 7f9415c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 0x7f94100c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.830+0000 7f9415c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f94101f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.830+0000 7f9415c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94100cad60 con 0x7f94100ca900 2024-03-22T13:01:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.830+0000 7f9415c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941006d320 con 0x7f94100ca520 2024-03-22T13:01:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.830+0000 7f9415c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9410154c20 con 0x7f941015b0e0 2024-03-22T13:01:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.831+0000 7f940f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f94101f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.831+0000 7f940f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f94101f3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53558/0 (socket says 172.21.15.6:53558) 2024-03-22T13:01:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.831+0000 7f940f400640 1 -- 172.21.15.6:0/745620027 learned_addr learned my addr 172.21.15.6:0/745620027 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.832+0000 7f940ea00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94100ca520 0x7f94101578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.837+0000 7f940f400640 1 -- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94100ca520 msgr2=0x7f94101578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.837+0000 7f940f400640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94100ca520 0x7f94101578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.837+0000 7f940f400640 1 -- 172.21.15.6:0/745620027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 msgr2=0x7f94100c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.837+0000 7f940f400640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 0x7f94100c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.837+0000 7f940f400640 1 -- 172.21.15.6:0/745620027 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410154f20 con 0x7f941015b0e0 2024-03-22T13:01:31.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.840+0000 7f940f400640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f94101f3de0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9408009d60 tx=0x7f9408004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.846+0000 7f9406a00640 1 -- 172.21.15.6:0/745620027 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408044030 con 0x7f941015b0e0 2024-03-22T13:01:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.846+0000 7f9406a00640 1 -- 172.21.15.6:0/745620027 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940802c9d0 con 0x7f941015b0e0 2024-03-22T13:01:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.846+0000 7f9406a00640 1 -- 172.21.15.6:0/745620027 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408039040 con 0x7f941015b0e0 2024-03-22T13:01:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.846+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9410158290 con 0x7f941015b0e0 2024-03-22T13:01:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.846+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9410155470 con 0x7f941015b0e0 2024-03-22T13:01:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.848+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94100cbbe0 con 0x7f941015b0e0 2024-03-22T13:01:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.886+0000 7f9406a00640 1 -- 172.21.15.6:0/745620027 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9408047070 con 0x7f941015b0e0 2024-03-22T13:01:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.887+0000 7f9406a00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f93f006bc20 0x7f93f006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.888+0000 7f940ea00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f93f006bc20 0x7f93f006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.889+0000 7f940ea00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f93f006bc20 0x7f93f006e0e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94000098e0 tx=0x7f9400009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:31.895+0000 7f9406a00640 1 -- 172.21.15.6:0/745620027 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f940802cb80 con 0x7f941015b0e0 2024-03-22T13:01:32.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.349+0000 7f9406a00640 1 -- 172.21.15.6:0/745620027 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f94100cbbe0 con 0x7f941015b0e0 2024-03-22T13:01:32.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.597+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941015d370 con 0x7f941015b0e0 2024-03-22T13:01:32.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.620+0000 7f9406a00640 1 -- 172.21.15.6:0/745620027 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f940804e070 con 0x7f941015b0e0 2024-03-22T13:01:32.620 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:32.621 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f93f006bc20 msgr2=0x7f93f006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f93f006bc20 0x7f93f006e0e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f94000098e0 tx=0x7f9400009060 comp rx=0 tx=0).stop 2024-03-22T13:01:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 msgr2=0x7f94101f3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f94101f3de0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9408009d60 tx=0x7f9408004a40 comp rx=0 tx=0).stop 2024-03-22T13:01:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 shutdown_connections 2024-03-22T13:01:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f941015b0e0 0x7f94101f3de0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94100ca900 0x7f94100c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f93f006bc20 0x7f93f006e0e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 --2- 172.21.15.6:0/745620027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94100ca520 0x7f94101578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 >> 172.21.15.6:0/745620027 conn(0x7f94100b16b0 msgr2=0x7f94100afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 shutdown_connections 2024-03-22T13:01:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:32.623+0000 7f9415c00640 1 -- 172.21.15.6:0/745620027 wait complete. 2024-03-22T13:01:33.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:33.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:34.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:34.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.449+0000 7fa192c00640 1 Processor -- start 2024-03-22T13:01:34.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.449+0000 7fa192c00640 1 -- start start 2024-03-22T13:01:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.451+0000 7fa192c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa18c0ca520 0x7fa18c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.451+0000 7fa192c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa18c0ccad0 0x7fa18c15b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.451+0000 7fa192c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18c06c7a0 con 0x7fa18c15b0e0 2024-03-22T13:01:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.451+0000 7fa192c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c0cd000 con 0x7fa18c0ccad0 2024-03-22T13:01:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.451+0000 7fa192c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c06ca80 con 0x7fa18c0ca520 2024-03-22T13:01:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.452+0000 7fa18be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa18c0ca520 0x7fa18c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.452+0000 7fa190c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa18c0ccad0 0x7fa18c15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.452+0000 7fa190c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa18c0ccad0 0x7fa18c15b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49362/0 (socket says 172.21.15.6:49362) 2024-03-22T13:01:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.452+0000 7fa190c00640 1 -- 172.21.15.6:0/3118892320 learned_addr learned my addr 172.21.15.6:0/3118892320 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:34.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.456+0000 7fa190c00640 1 -- 172.21.15.6:0/3118892320 >> v2:172.21.15.92:3300/0 conn(0x7fa18c0ca520 msgr2=0x7fa18c06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:34.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.456+0000 7fa190c00640 1 --2- 172.21.15.6:0/3118892320 >> v2:172.21.15.92:3300/0 conn(0x7fa18c0ca520 0x7fa18c06d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:34.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.456+0000 7fa190c00640 1 -- 172.21.15.6:0/3118892320 >> v1:172.21.15.92:6789/0 conn(0x7fa18c15b0e0 legacy=0x7fa18c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:34.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.456+0000 7fa190c00640 1 -- 172.21.15.6:0/3118892320 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c06cd60 con 0x7fa18c0ccad0 2024-03-22T13:01:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.459+0000 7fa190c00640 1 --2- 172.21.15.6:0/3118892320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c0ccad0 0x7fa18c15b980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0d0ca0 tx=0x7fa18402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2680398a3e82eaae server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:34.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.465+0000 7fa18aa00640 1 -- 172.21.15.6:0/3118892320 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18402cb20 con 0x7fa18c0ccad0 2024-03-22T13:01:34.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.465+0000 7fa18aa00640 1 -- 172.21.15.6:0/3118892320 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18402ccc0 con 0x7fa18c0ccad0 2024-03-22T13:01:34.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.465+0000 7fa18aa00640 1 -- 172.21.15.6:0/3118892320 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184034610 con 0x7fa18c0ccad0 2024-03-22T13:01:34.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 -- 172.21.15.6:0/3118892320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c0ccad0 msgr2=0x7fa18c15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 --2- 172.21.15.6:0/3118892320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c0ccad0 0x7fa18c15b980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0d0ca0 tx=0x7fa18402bf80 comp rx=0 tx=0).stop 2024-03-22T13:01:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 -- 172.21.15.6:0/3118892320 shutdown_connections 2024-03-22T13:01:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 --2- 172.21.15.6:0/3118892320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c0ccad0 0x7fa18c15b980 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 --2- 172.21.15.6:0/3118892320 >> v2:172.21.15.92:3300/0 conn(0x7fa18c0ca520 0x7fa18c06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 -- 172.21.15.6:0/3118892320 >> 172.21.15.6:0/3118892320 conn(0x7fa18c0b16b0 msgr2=0x7fa18c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 -- 172.21.15.6:0/3118892320 shutdown_connections 2024-03-22T13:01:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.466+0000 7fa192c00640 1 -- 172.21.15.6:0/3118892320 wait complete. 2024-03-22T13:01:34.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.469+0000 7fa192c00640 1 Processor -- start 2024-03-22T13:01:34.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.469+0000 7fa192c00640 1 -- start start 2024-03-22T13:01:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.470+0000 7fa192c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa18c0ca520 0x7fa18c1f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.470+0000 7fa192c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa18c0ccad0 0x7fa18c155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.470+0000 7fa192c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c15b0e0 0x7fa18c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.470+0000 7fa192c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c154c20 con 0x7fa18c15b0e0 2024-03-22T13:01:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.470+0000 7fa192c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c06d320 con 0x7fa18c0ca520 2024-03-22T13:01:34.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.470+0000 7fa192c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0ca900 con 0x7fa18c0ccad0 2024-03-22T13:01:34.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.471+0000 7fa18be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa18c0ca520 0x7fa18c1f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:34.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.471+0000 7fa18b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa18c0ccad0 0x7fa18c155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:34.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.471+0000 7fa190c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c15b0e0 0x7fa18c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:34.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.471+0000 7fa18be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa18c0ca520 0x7fa18c1f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60510/0 (socket says 172.21.15.6:60510) 2024-03-22T13:01:34.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.472+0000 7fa18be00640 1 -- 172.21.15.6:0/2267694249 learned_addr learned my addr 172.21.15.6:0/2267694249 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:34.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.475+0000 7fa18be00640 1 -- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa18c0ccad0 msgr2=0x7fa18c155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:34.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.475+0000 7fa18be00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa18c0ccad0 0x7fa18c155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:34.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.475+0000 7fa18be00640 1 -- 172.21.15.6:0/2267694249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c15b0e0 msgr2=0x7fa18c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:34.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.475+0000 7fa18be00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c15b0e0 0x7fa18c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:34.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.475+0000 7fa18be00640 1 -- 172.21.15.6:0/2267694249 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c1579f0 con 0x7fa18c0ca520 2024-03-22T13:01:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.478+0000 7fa18be00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa18c0ca520 0x7fa18c1f3d70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9b0 tx=0x7fa17c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.483+0000 7fa183e00640 1 -- 172.21.15.6:0/2267694249 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c017070 con 0x7fa18c0ca520 2024-03-22T13:01:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.483+0000 7fa183e00640 1 -- 172.21.15.6:0/2267694249 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c00c9a0 con 0x7fa18c0ca520 2024-03-22T13:01:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.483+0000 7fa183e00640 1 -- 172.21.15.6:0/2267694249 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c00cd90 con 0x7fa18c0ca520 2024-03-22T13:01:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.483+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18c157c60 con 0x7fa18c0ca520 2024-03-22T13:01:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.483+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c1eff00 con 0x7fa18c0ca520 2024-03-22T13:01:34.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.484+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c0756c0 con 0x7fa18c0ca520 2024-03-22T13:01:34.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.510+0000 7fa183e00640 1 -- 172.21.15.6:0/2267694249 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa17c01a070 con 0x7fa18c0ca520 2024-03-22T13:01:34.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.512+0000 7fa183e00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa16c06bc20 0x7fa16c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:34.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.512+0000 7fa18b400640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa16c06bc20 0x7fa16c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:34.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.513+0000 7fa183e00640 1 -- 172.21.15.6:0/2267694249 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa17c057720 con 0x7fa18c0ca520 2024-03-22T13:01:34.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.513+0000 7fa18b400640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa16c06bc20 0x7fa16c06e0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0b0000 tx=0x7fa178009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:34.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:34.785+0000 7fa183e00640 1 -- 172.21.15.6:0/2267694249 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa17c059680 con 0x7fa18c0ca520 2024-03-22T13:01:35.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.042+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa18c0b5b20 con 0x7fa18c0ca520 2024-03-22T13:01:35.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.064+0000 7fa183e00640 1 -- 172.21.15.6:0/2267694249 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fa17c05d720 con 0x7fa18c0ca520 2024-03-22T13:01:35.064 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:35.064 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:35.065 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa16c06bc20 msgr2=0x7fa16c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa16c06bc20 0x7fa16c06e0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0b0000 tx=0x7fa178009040 comp rx=0 tx=0).stop 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa18c0ca520 msgr2=0x7fa18c1f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa18c0ca520 0x7fa18c1f3d70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9b0 tx=0x7fa17c00ee70 comp rx=0 tx=0).stop 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 shutdown_connections 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa18c15b0e0 0x7fa18c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa18c0ccad0 0x7fa18c155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:35.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa16c06bc20 0x7fa16c06e0e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:35.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 --2- 172.21.15.6:0/2267694249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa18c0ca520 0x7fa18c1f3d70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:35.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 >> 172.21.15.6:0/2267694249 conn(0x7fa18c0b16b0 msgr2=0x7fa18c0cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:35.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 shutdown_connections 2024-03-22T13:01:35.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:35.067+0000 7fa192c00640 1 -- 172.21.15.6:0/2267694249 wait complete. 2024-03-22T13:01:35.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:35.690 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:36.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:36.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.887+0000 7fc1daa00640 1 Processor -- start 2024-03-22T13:01:36.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.887+0000 7fc1daa00640 1 -- start start 2024-03-22T13:01:36.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.888+0000 7fc1daa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1d41589e0 0x7fc1d415f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:36.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.888+0000 7fc1daa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc1d40c6880 0x7fc1d406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:36.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.888+0000 7fc1daa00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40715b0 con 0x7fc1d4072c10 2024-03-22T13:01:36.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.889+0000 7fc1daa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d40c6dd0 con 0x7fc1d41589e0 2024-03-22T13:01:36.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.889+0000 7fc1daa00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d4161f20 con 0x7fc1d40c6880 2024-03-22T13:01:36.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.889+0000 7fc1da000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc1d40c6880 0x7fc1d406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:36.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.890+0000 7fc1d8c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc1d4072c10 0x7fc1d41623b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:38750/0 (socket says 172.21.15.6:38750) 2024-03-22T13:01:36.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.890+0000 7fc1d8c00640 1 -- 172.21.15.6:0/3421586621 learned_addr learned my addr 172.21.15.6:0/3421586621 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:36.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.894+0000 7fc1da000640 1 -- 172.21.15.6:0/3421586621 >> v2:172.21.15.6:3300/0 conn(0x7fc1d41589e0 msgr2=0x7fc1d415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:36.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.894+0000 7fc1da000640 1 --2- 172.21.15.6:0/3421586621 >> v2:172.21.15.6:3300/0 conn(0x7fc1d41589e0 0x7fc1d415f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:36.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.894+0000 7fc1da000640 1 -- 172.21.15.6:0/3421586621 >> v1:172.21.15.92:6789/0 conn(0x7fc1d4072c10 legacy=0x7fc1d41623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:36.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.894+0000 7fc1da000640 1 -- 172.21.15.6:0/3421586621 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d4071890 con 0x7fc1d40c6880 2024-03-22T13:01:36.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.900+0000 7fc1da000640 1 --2- 172.21.15.6:0/3421586621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d406d1d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc1d40c2c40 tx=0x7fc1cc02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce9c570039b98a6b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:36.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.906+0000 7fc1d3e00640 1 -- 172.21.15.6:0/3421586621 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc039070 con 0x7fc1d40c6880 2024-03-22T13:01:36.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.906+0000 7fc1d3e00640 1 -- 172.21.15.6:0/3421586621 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1cc02cc80 con 0x7fc1d40c6880 2024-03-22T13:01:36.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.906+0000 7fc1d3e00640 1 -- 172.21.15.6:0/3421586621 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc034660 con 0x7fc1d40c6880 2024-03-22T13:01:36.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 -- 172.21.15.6:0/3421586621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 msgr2=0x7fc1d406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:36.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 --2- 172.21.15.6:0/3421586621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d406d1d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc1d40c2c40 tx=0x7fc1cc02bed0 comp rx=0 tx=0).stop 2024-03-22T13:01:36.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 -- 172.21.15.6:0/3421586621 shutdown_connections 2024-03-22T13:01:36.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 --2- 172.21.15.6:0/3421586621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d406d1d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:36.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 --2- 172.21.15.6:0/3421586621 >> v2:172.21.15.6:3300/0 conn(0x7fc1d41589e0 0x7fc1d415f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:36.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 -- 172.21.15.6:0/3421586621 >> 172.21.15.6:0/3421586621 conn(0x7fc1d40cb890 msgr2=0x7fc1d40cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:36.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 -- 172.21.15.6:0/3421586621 shutdown_connections 2024-03-22T13:01:36.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.907+0000 7fc1daa00640 1 -- 172.21.15.6:0/3421586621 wait complete. 2024-03-22T13:01:36.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.910+0000 7fc1daa00640 1 Processor -- start 2024-03-22T13:01:36.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.910+0000 7fc1daa00640 1 -- start start 2024-03-22T13:01:36.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.911+0000 7fc1daa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1d4072c10 0x7fc1d414fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:36.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.911+0000 7fc1daa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:36.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.911+0000 7fc1daa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1d41589e0 0x7fc1d4159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:36.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.911+0000 7fc1daa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d4072450 con 0x7fc1d4072c10 2024-03-22T13:01:36.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.911+0000 7fc1daa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d415f670 con 0x7fc1d40c6880 2024-03-22T13:01:36.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.911+0000 7fc1daa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d4158dc0 con 0x7fc1d41589e0 2024-03-22T13:01:36.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.912+0000 7fc1d9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1d4072c10 0x7fc1d414fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:36.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.912+0000 7fc1d8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:36.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.912+0000 7fc1d8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d415d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60528/0 (socket says 172.21.15.6:60528) 2024-03-22T13:01:36.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.913+0000 7fc1d8c00640 1 -- 172.21.15.6:0/1948279220 learned_addr learned my addr 172.21.15.6:0/1948279220 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:36.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.914+0000 7fc1da000640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1d41589e0 0x7fc1d4159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:36.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.918+0000 7fc1d8c00640 1 -- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1d41589e0 msgr2=0x7fc1d4159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:36.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.918+0000 7fc1d8c00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1d41589e0 0x7fc1d4159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:36.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.918+0000 7fc1d8c00640 1 -- 172.21.15.6:0/1948279220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1d4072c10 msgr2=0x7fc1d414fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:36.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.918+0000 7fc1d8c00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1d4072c10 0x7fc1d414fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:36.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.918+0000 7fc1d8c00640 1 -- 172.21.15.6:0/1948279220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d415d8f0 con 0x7fc1d40c6880 2024-03-22T13:01:36.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.921+0000 7fc1d8c00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d415d3b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800c420 tx=0x7fc1c800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.926+0000 7fc1d1600640 1 -- 172.21.15.6:0/1948279220 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8017070 con 0x7fc1d40c6880 2024-03-22T13:01:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.926+0000 7fc1d1600640 1 -- 172.21.15.6:0/1948279220 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c8005ce0 con 0x7fc1d40c6880 2024-03-22T13:01:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.926+0000 7fc1d1600640 1 -- 172.21.15.6:0/1948279220 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80072f0 con 0x7fc1d40c6880 2024-03-22T13:01:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.926+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d41500b0 con 0x7fc1d40c6880 2024-03-22T13:01:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.926+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d41f4080 con 0x7fc1d40c6880 2024-03-22T13:01:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.927+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d4075650 con 0x7fc1d40c6880 2024-03-22T13:01:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.952+0000 7fc1d1600640 1 -- 172.21.15.6:0/1948279220 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc1c801a070 con 0x7fc1d40c6880 2024-03-22T13:01:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.953+0000 7fc1d1600640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1b406bc20 0x7fc1b406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.953+0000 7fc1d9600640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1b406bc20 0x7fc1b406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:36.954+0000 7fc1d9600640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1b406bc20 0x7fc1b406e0e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009990 tx=0x7fc1c0009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:37.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.221+0000 7fc1d1600640 1 -- 172.21.15.6:0/1948279220 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc1c80566f0 con 0x7fc1d40c6880 2024-03-22T13:01:37.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.239+0000 7fc1d1600640 1 -- 172.21.15.6:0/1948279220 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc1c8021020 con 0x7fc1d40c6880 2024-03-22T13:01:37.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.488+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1d4071f50 con 0x7fc1d40c6880 2024-03-22T13:01:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.503+0000 7fc1d1600640 1 -- 172.21.15.6:0/1948279220 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fc1c8058650 con 0x7fc1d40c6880 2024-03-22T13:01:37.503 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:37.503 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1b406bc20 msgr2=0x7fc1b406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1b406bc20 0x7fc1b406e0e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009990 tx=0x7fc1c0009110 comp rx=0 tx=0).stop 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 msgr2=0x7fc1d415d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d415d3b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800c420 tx=0x7fc1c800c820 comp rx=0 tx=0).stop 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 shutdown_connections 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1d41589e0 0x7fc1d4159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc1b406bc20 0x7fc1b406e0e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1d40c6880 0x7fc1d415d3b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 --2- 172.21.15.6:0/1948279220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1d4072c10 0x7fc1d414fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.505+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 >> 172.21.15.6:0/1948279220 conn(0x7fc1d40cb890 msgr2=0x7fc1d415ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.506+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 shutdown_connections 2024-03-22T13:01:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:37.506+0000 7fc1daa00640 1 -- 172.21.15.6:0/1948279220 wait complete. 2024-03-22T13:01:38.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:38.125 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:39.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.314+0000 7fcaf5a00640 1 Processor -- start 2024-03-22T13:01:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.314+0000 7fcaf5a00640 1 -- start start 2024-03-22T13:01:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.316+0000 7fcaf5a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cc3b0 0x7fcaf00cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.316+0000 7fcaf5a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf00ce230 con 0x7fcaf00cf770 2024-03-22T13:01:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.316+0000 7fcaf5a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf00ce510 con 0x7fcaf00ccea0 2024-03-22T13:01:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.316+0000 7fcaf5a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00ce7f0 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.317+0000 7fcaeea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cc3b0 0x7fcaf00cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.317+0000 7fcaee000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcaf00cf770 0x7fcaf00cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35880/0 (socket says 172.21.15.6:35880) 2024-03-22T13:01:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.317+0000 7fcaee000640 1 -- 172.21.15.6:0/1081140336 learned_addr learned my addr 172.21.15.6:0/1081140336 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.321+0000 7fcaeea00640 1 -- 172.21.15.6:0/1081140336 >> v1:172.21.15.92:6789/0 conn(0x7fcaf00ccea0 legacy=0x7fcaf015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.321+0000 7fcaeea00640 1 -- 172.21.15.6:0/1081140336 >> v1:172.21.15.6:6789/0 conn(0x7fcaf00cf770 legacy=0x7fcaf00cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.321+0000 7fcaeea00640 1 -- 172.21.15.6:0/1081140336 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00cead0 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.324+0000 7fcaeea00640 1 --2- 172.21.15.6:0/1081140336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cc3b0 0x7fcaf00cf230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcae00097d0 tx=0x7fcae002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95b817167b45139 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.329+0000 7fcaed600640 1 -- 172.21.15.6:0/1081140336 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0035020 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.329+0000 7fcaed600640 1 -- 172.21.15.6:0/1081140336 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae0031810 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.329+0000 7fcaed600640 1 -- 172.21.15.6:0/1081140336 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0031be0 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.330+0000 7fcaf5a00640 1 -- 172.21.15.6:0/1081140336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cc3b0 msgr2=0x7fcaf00cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.330+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/1081140336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cc3b0 0x7fcaf00cf230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcae00097d0 tx=0x7fcae002b8b0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.330+0000 7fcaf5a00640 1 -- 172.21.15.6:0/1081140336 shutdown_connections 2024-03-22T13:01:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.330+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/1081140336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cc3b0 0x7fcaf00cf230 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.330+0000 7fcaf5a00640 1 -- 172.21.15.6:0/1081140336 >> 172.21.15.6:0/1081140336 conn(0x7fcaf00b16d0 msgr2=0x7fcaf00b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.330+0000 7fcaf5a00640 1 -- 172.21.15.6:0/1081140336 shutdown_connections 2024-03-22T13:01:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.330+0000 7fcaf5a00640 1 -- 172.21.15.6:0/1081140336 wait complete. 2024-03-22T13:01:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.332+0000 7fcaf5a00640 1 Processor -- start 2024-03-22T13:01:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.332+0000 7fcaf5a00640 1 -- start start 2024-03-22T13:01:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaf5a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf00cc3b0 0x7fcaf00c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaf5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcaf00ccea0 0x7fcaf01f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaf5a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cf770 0x7fcaf00c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaf5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf015a5d0 con 0x7fcaf00ccea0 2024-03-22T13:01:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaf5a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf006dc10 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaf5a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0154bb0 con 0x7fcaf00cf770 2024-03-22T13:01:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaef400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cf770 0x7fcaf00c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.333+0000 7fcaeea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf00cc3b0 0x7fcaf00c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.334+0000 7fcaef400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cf770 0x7fcaf00c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46026/0 (socket says 172.21.15.6:46026) 2024-03-22T13:01:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.334+0000 7fcaef400640 1 -- 172.21.15.6:0/736269014 learned_addr learned my addr 172.21.15.6:0/736269014 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.334+0000 7fcaee000640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcaf00ccea0 0x7fcaf01f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.337+0000 7fcaeea00640 1 -- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cf770 msgr2=0x7fcaf00c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.337+0000 7fcaeea00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cf770 0x7fcaf00c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.337+0000 7fcaeea00640 1 -- 172.21.15.6:0/736269014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcaf00ccea0 msgr2=0x7fcaf01f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.337+0000 7fcaeea00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcaf00ccea0 0x7fcaf01f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.337+0000 7fcaeea00640 1 -- 172.21.15.6:0/736269014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00c7f70 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.340+0000 7fcaeea00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf00cc3b0 0x7fcaf00c5bb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcaf008cec0 tx=0x7fcae00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.345+0000 7fcae6a00640 1 -- 172.21.15.6:0/736269014 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0041070 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.345+0000 7fcae6a00640 1 -- 172.21.15.6:0/736269014 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae003b430 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.345+0000 7fcae6a00640 1 -- 172.21.15.6:0/736269014 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0035040 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.345+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf01f48c0 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.345+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf00c9a00 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.347+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf00756c0 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.371+0000 7fcae6a00640 1 -- 172.21.15.6:0/736269014 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcae0038020 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.372+0000 7fcae6a00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcad006bc20 0x7fcad006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:39.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.372+0000 7fcaee000640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcad006bc20 0x7fcad006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:39.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.373+0000 7fcaee000640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcad006bc20 0x7fcad006e0e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0157490 tx=0x7fcadc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:39.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.615+0000 7fcae6a00640 1 -- 172.21.15.6:0/736269014 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fcae0079000 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.627+0000 7fcae6a00640 1 -- 172.21.15.6:0/736269014 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcae007af60 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.877+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaf00ccc80 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.898+0000 7fcae6a00640 1 -- 172.21.15.6:0/736269014 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fcae007f000 con 0x7fcaf00cc3b0 2024-03-22T13:01:39.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:39.899 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:39.899 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:39.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcad006bc20 msgr2=0x7fcad006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:39.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcad006bc20 0x7fcad006e0e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0157490 tx=0x7fcadc009040 comp rx=0 tx=0).stop 2024-03-22T13:01:39.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf00cc3b0 msgr2=0x7fcaf00c5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:39.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf00cc3b0 0x7fcaf00c5bb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcaf008cec0 tx=0x7fcae00046c0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 shutdown_connections 2024-03-22T13:01:39.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcaf00cf770 0x7fcaf00c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcaf00ccea0 0x7fcaf01f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcad006bc20 0x7fcad006e0e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 --2- 172.21.15.6:0/736269014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf00cc3b0 0x7fcaf00c5bb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 >> 172.21.15.6:0/736269014 conn(0x7fcaf00b16d0 msgr2=0x7fcaf00b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 shutdown_connections 2024-03-22T13:01:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:39.901+0000 7fcaf5a00640 1 -- 172.21.15.6:0/736269014 wait complete. 2024-03-22T13:01:40.525 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:40.545 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:41.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.739+0000 7f64e1c00640 1 Processor -- start 2024-03-22T13:01:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.739+0000 7f64e1c00640 1 -- start start 2024-03-22T13:01:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.740+0000 7f64e1c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64dc152400 0x7f64dc15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.740+0000 7f64e1c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.740+0000 7f64e1c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc1590b0 con 0x7f64dc15bc80 2024-03-22T13:01:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.740+0000 7f64e1c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64dc0724d0 con 0x7f64dc152400 2024-03-22T13:01:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.740+0000 7f64e1c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc0721f0 con 0x7f64dc06f8c0 2024-03-22T13:01:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.741+0000 7f64db400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.741+0000 7f64da000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f64dc15bc80 0x7f64dc0716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:38766/0 (socket says 172.21.15.6:38766) 2024-03-22T13:01:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.741+0000 7f64da000640 1 -- 172.21.15.6:0/3946742882 learned_addr learned my addr 172.21.15.6:0/3946742882 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.741+0000 7f64daa00640 1 --2- 172.21.15.6:0/3946742882 >> v2:172.21.15.6:3300/0 conn(0x7f64dc152400 0x7f64dc15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:41.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.744+0000 7f64db400640 1 -- 172.21.15.6:0/3946742882 >> v2:172.21.15.6:3300/0 conn(0x7f64dc152400 msgr2=0x7f64dc15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:41.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.744+0000 7f64db400640 1 --2- 172.21.15.6:0/3946742882 >> v2:172.21.15.6:3300/0 conn(0x7f64dc152400 0x7f64dc15b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:41.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.744+0000 7f64db400640 1 -- 172.21.15.6:0/3946742882 >> v1:172.21.15.92:6789/0 conn(0x7f64dc15bc80 legacy=0x7f64dc0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:41.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.745+0000 7f64db400640 1 -- 172.21.15.6:0/3946742882 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc1582f0 con 0x7f64dc06f8c0 2024-03-22T13:01:41.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.747+0000 7f64db400640 1 --2- 172.21.15.6:0/3946742882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc0c5040 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009d90 tx=0x7f64d402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e2bae18f2dcdf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:41.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.749+0000 7f64d9600640 1 -- 172.21.15.6:0/3946742882 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4039070 con 0x7f64dc06f8c0 2024-03-22T13:01:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.752+0000 7f64d9600640 1 -- 172.21.15.6:0/3946742882 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d402cce0 con 0x7f64dc06f8c0 2024-03-22T13:01:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.752+0000 7f64d9600640 1 -- 172.21.15.6:0/3946742882 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4034660 con 0x7f64dc06f8c0 2024-03-22T13:01:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.752+0000 7f64e1c00640 1 -- 172.21.15.6:0/3946742882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 msgr2=0x7f64dc0c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.752+0000 7f64e1c00640 1 --2- 172.21.15.6:0/3946742882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc0c5040 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009d90 tx=0x7f64d402bf30 comp rx=0 tx=0).stop 2024-03-22T13:01:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.753+0000 7f64e1c00640 1 -- 172.21.15.6:0/3946742882 shutdown_connections 2024-03-22T13:01:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.753+0000 7f64e1c00640 1 --2- 172.21.15.6:0/3946742882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc0c5040 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.753+0000 7f64e1c00640 1 --2- 172.21.15.6:0/3946742882 >> v2:172.21.15.6:3300/0 conn(0x7f64dc152400 0x7f64dc15b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.753+0000 7f64e1c00640 1 -- 172.21.15.6:0/3946742882 >> 172.21.15.6:0/3946742882 conn(0x7f64dc0cb930 msgr2=0x7f64dc0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.753+0000 7f64e1c00640 1 -- 172.21.15.6:0/3946742882 shutdown_connections 2024-03-22T13:01:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.753+0000 7f64e1c00640 1 -- 172.21.15.6:0/3946742882 wait complete. 2024-03-22T13:01:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.755+0000 7f64e1c00640 1 Processor -- start 2024-03-22T13:01:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.755+0000 7f64e1c00640 1 -- start start 2024-03-22T13:01:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.756+0000 7f64e1c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc1f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.756+0000 7f64e1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64dc152400 0x7f64dc1485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.756+0000 7f64e1c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64dc15bc80 0x7f64dc1f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.756+0000 7f64e1c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc158b90 con 0x7f64dc152400 2024-03-22T13:01:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.756+0000 7f64e1c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc1588b0 con 0x7f64dc15bc80 2024-03-22T13:01:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.756+0000 7f64e1c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc1527e0 con 0x7f64dc06f8c0 2024-03-22T13:01:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.756+0000 7f64da000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64dc152400 0x7f64dc1485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.757+0000 7f64daa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc1f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.757+0000 7f64db400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64dc15bc80 0x7f64dc1f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.757+0000 7f64da000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64dc152400 0x7f64dc1485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49408/0 (socket says 172.21.15.6:49408) 2024-03-22T13:01:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.757+0000 7f64da000640 1 -- 172.21.15.6:0/1344488241 learned_addr learned my addr 172.21.15.6:0/1344488241 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.761+0000 7f64da000640 1 -- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 msgr2=0x7f64dc1f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.761+0000 7f64da000640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc1f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.761+0000 7f64da000640 1 -- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64dc15bc80 msgr2=0x7f64dc1f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.761+0000 7f64da000640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64dc15bc80 0x7f64dc1f1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.761+0000 7f64da000640 1 -- 172.21.15.6:0/1344488241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc0c4070 con 0x7f64dc152400 2024-03-22T13:01:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.763+0000 7f64da000640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64dc152400 0x7f64dc1485b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f64c800c3f0 tx=0x7f64c800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:41.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.767+0000 7f64d2a00640 1 -- 172.21.15.6:0/1344488241 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8017070 con 0x7f64dc152400 2024-03-22T13:01:41.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.767+0000 7f64d2a00640 1 -- 172.21.15.6:0/1344488241 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c8005ce0 con 0x7f64dc152400 2024-03-22T13:01:41.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.767+0000 7f64d2a00640 1 -- 172.21.15.6:0/1344488241 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c80072f0 con 0x7f64dc152400 2024-03-22T13:01:41.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.768+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc0c4350 con 0x7f64dc152400 2024-03-22T13:01:41.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.768+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc0c4910 con 0x7f64dc152400 2024-03-22T13:01:41.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.769+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64dc0756c0 con 0x7f64dc152400 2024-03-22T13:01:41.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.798+0000 7f64d2a00640 1 -- 172.21.15.6:0/1344488241 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f64c801a070 con 0x7f64dc152400 2024-03-22T13:01:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.799+0000 7f64d2a00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64bc06bc20 0x7f64bc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:41.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.800+0000 7f64daa00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64bc06bc20 0x7f64bc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:41.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.801+0000 7f64daa00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64bc06bc20 0x7f64bc06e0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009b40 tx=0x7f64cc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:41.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:41.803+0000 7f64d2a00640 1 -- 172.21.15.6:0/1344488241 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f64c80566a0 con 0x7f64dc152400 2024-03-22T13:01:42.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.124+0000 7f64d2a00640 1 -- 172.21.15.6:0/1344488241 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f64c8021020 con 0x7f64dc152400 2024-03-22T13:01:42.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.440+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64dc0c6400 con 0x7f64dc152400 2024-03-22T13:01:42.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.462+0000 7f64d2a00640 1 -- 172.21.15.6:0/1344488241 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f64c8058600 con 0x7f64dc152400 2024-03-22T13:01:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:42.462 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:42.462 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64bc06bc20 msgr2=0x7f64bc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64bc06bc20 0x7f64bc06e0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009b40 tx=0x7f64cc009190 comp rx=0 tx=0).stop 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64dc152400 msgr2=0x7f64dc1485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64dc152400 0x7f64dc1485b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f64c800c3f0 tx=0x7f64c800c7f0 comp rx=0 tx=0).stop 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 shutdown_connections 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64bc06bc20 0x7f64bc06e0e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64dc15bc80 0x7f64dc1f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64dc152400 0x7f64dc1485b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 --2- 172.21.15.6:0/1344488241 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64dc06f8c0 0x7f64dc1f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 >> 172.21.15.6:0/1344488241 conn(0x7f64dc0cb930 msgr2=0x7f64dc0744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.465+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 shutdown_connections 2024-03-22T13:01:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:42.466+0000 7f64e1c00640 1 -- 172.21.15.6:0/1344488241 wait complete. 2024-03-22T13:01:43.094 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:43.095 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:44.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:44.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.282+0000 7f7ee4a00640 1 Processor -- start 2024-03-22T13:01:44.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.283+0000 7f7ee4a00640 1 -- start start 2024-03-22T13:01:44.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.284+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee00ca520 0x7f7ee006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:44.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.284+0000 7f7ee4a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7ee015b0e0 0x7f7ee015b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:44.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.284+0000 7f7ee4a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ee00ca900 0x7f7ee00cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:44.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.284+0000 7f7ee4a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee00cd000 con 0x7f7ee00ca900 2024-03-22T13:01:44.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.284+0000 7f7ee4a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee006c7a0 con 0x7f7ee015b0e0 2024-03-22T13:01:44.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.284+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee006ca80 con 0x7f7ee00ca520 2024-03-22T13:01:44.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.285+0000 7f7ede800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ee00ca900 0x7f7ee00cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:44.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.285+0000 7f7edd400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7ee015b0e0 0x7f7ee015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:44.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.285+0000 7f7edde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee00ca520 0x7f7ee006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:44.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.285+0000 7f7ede800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ee00ca900 0x7f7ee00cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50520/0 (socket says 172.21.15.6:50520) 2024-03-22T13:01:44.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.285+0000 7f7ede800640 1 -- 172.21.15.6:0/2029981369 learned_addr learned my addr 172.21.15.6:0/2029981369 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:44.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.287+0000 7f7edd400640 1 -- 172.21.15.6:0/2029981369 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee00ca520 msgr2=0x7f7ee006d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:44.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.287+0000 7f7edd400640 1 --2- 172.21.15.6:0/2029981369 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee00ca520 0x7f7ee006d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.287+0000 7f7edd400640 1 -- 172.21.15.6:0/2029981369 >> v2:172.21.15.6:3300/0 conn(0x7f7ee00ca900 msgr2=0x7f7ee00cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:44.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.287+0000 7f7edd400640 1 --2- 172.21.15.6:0/2029981369 >> v2:172.21.15.6:3300/0 conn(0x7f7ee00ca900 0x7f7ee00cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.287+0000 7f7edd400640 1 -- 172.21.15.6:0/2029981369 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee006cd60 con 0x7f7ee015b0e0 2024-03-22T13:01:44.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.289+0000 7f7edd400640 1 --2- 172.21.15.6:0/2029981369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee015b0e0 0x7f7ee015b980 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7ee008d040 tx=0x7f7ecc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66df3790f5f6f4ae server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:44.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.294+0000 7f7edca00640 1 -- 172.21.15.6:0/2029981369 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc0315a0 con 0x7f7ee015b0e0 2024-03-22T13:01:44.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.294+0000 7f7edca00640 1 -- 172.21.15.6:0/2029981369 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc037070 con 0x7f7ee015b0e0 2024-03-22T13:01:44.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.294+0000 7f7edca00640 1 -- 172.21.15.6:0/2029981369 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc031960 con 0x7f7ee015b0e0 2024-03-22T13:01:44.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 -- 172.21.15.6:0/2029981369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee015b0e0 msgr2=0x7f7ee015b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:44.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/2029981369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee015b0e0 0x7f7ee015b980 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7ee008d040 tx=0x7f7ecc02b840 comp rx=0 tx=0).stop 2024-03-22T13:01:44.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 -- 172.21.15.6:0/2029981369 shutdown_connections 2024-03-22T13:01:44.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/2029981369 >> v2:172.21.15.6:3300/0 conn(0x7f7ee00ca900 0x7f7ee00cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/2029981369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee015b0e0 0x7f7ee015b980 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/2029981369 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee00ca520 0x7f7ee006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 -- 172.21.15.6:0/2029981369 >> 172.21.15.6:0/2029981369 conn(0x7f7ee00b16b0 msgr2=0x7f7ee00b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:44.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 -- 172.21.15.6:0/2029981369 shutdown_connections 2024-03-22T13:01:44.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.295+0000 7f7ee4a00640 1 -- 172.21.15.6:0/2029981369 wait complete. 2024-03-22T13:01:44.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.298+0000 7f7ee4a00640 1 Processor -- start 2024-03-22T13:01:44.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.298+0000 7f7ee4a00640 1 -- start start 2024-03-22T13:01:44.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.300+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ee00ca520 0x7f7ee015a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.300+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee00ca900 0x7f7ee01f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.300+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee015b0e0 0x7f7ee00c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.300+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee00cad60 con 0x7f7ee00ca520 2024-03-22T13:01:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.300+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee006d320 con 0x7f7ee00ca900 2024-03-22T13:01:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.300+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee0154c20 con 0x7f7ee015b0e0 2024-03-22T13:01:44.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.300+0000 7f7ede800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee015b0e0 0x7f7ee00c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.301+0000 7f7edd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee00ca900 0x7f7ee01f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.301+0000 7f7ede800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee015b0e0 0x7f7ee00c7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46762/0 (socket says 172.21.15.6:46762) 2024-03-22T13:01:44.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.301+0000 7f7ede800640 1 -- 172.21.15.6:0/3224579779 learned_addr learned my addr 172.21.15.6:0/3224579779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:44.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.305+0000 7f7edde00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ee00ca520 0x7f7ee015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:44.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.305+0000 7f7edd400640 1 -- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee015b0e0 msgr2=0x7f7ee00c7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:44.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.305+0000 7f7edd400640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee015b0e0 0x7f7ee00c7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.305+0000 7f7edd400640 1 -- 172.21.15.6:0/3224579779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ee00ca520 msgr2=0x7f7ee015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:44.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.305+0000 7f7edd400640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ee00ca520 0x7f7ee015a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.305+0000 7f7edd400640 1 -- 172.21.15.6:0/3224579779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee0155a60 con 0x7f7ee00ca900 2024-03-22T13:01:44.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.307+0000 7f7edd400640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee00ca900 0x7f7ee01f01c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc033000 tx=0x7f7ecc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:44.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.311+0000 7f7ed6000640 1 -- 172.21.15.6:0/3224579779 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc032630 con 0x7f7ee00ca900 2024-03-22T13:01:44.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.311+0000 7f7ed6000640 1 -- 172.21.15.6:0/3224579779 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc0327f0 con 0x7f7ee00ca900 2024-03-22T13:01:44.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.311+0000 7f7ed6000640 1 -- 172.21.15.6:0/3224579779 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc032be0 con 0x7f7ee00ca900 2024-03-22T13:01:44.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.311+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee01551c0 con 0x7f7ee00ca900 2024-03-22T13:01:44.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.311+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee0155780 con 0x7f7ee00ca900 2024-03-22T13:01:44.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.313+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee00756c0 con 0x7f7ee00ca900 2024-03-22T13:01:44.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.333+0000 7f7ed6000640 1 -- 172.21.15.6:0/3224579779 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7ecc04e3f0 con 0x7f7ee00ca900 2024-03-22T13:01:44.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.334+0000 7f7ed6000640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ec006bc20 0x7f7ec006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:44.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.334+0000 7f7ed6000640 1 -- 172.21.15.6:0/3224579779 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f7ecc02f070 con 0x7f7ee00ca900 2024-03-22T13:01:44.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.334+0000 7f7edde00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ec006bc20 0x7f7ec006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:44.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.335+0000 7f7edde00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ec006bc20 0x7f7ec006e0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00098f0 tx=0x7f7ed0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:44.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.652+0000 7f7ed6000640 1 -- 172.21.15.6:0/3224579779 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7ecc040070 con 0x7f7ee00ca900 2024-03-22T13:01:44.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.942+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ee00cb080 con 0x7f7ee00ca900 2024-03-22T13:01:44.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.955+0000 7f7ed6000640 1 -- 172.21.15.6:0/3224579779 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f7ecc07b4a0 con 0x7f7ee00ca900 2024-03-22T13:01:44.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:44.956 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:44.956 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ec006bc20 msgr2=0x7f7ec006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ec006bc20 0x7f7ec006e0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00098f0 tx=0x7f7ed0008040 comp rx=0 tx=0).stop 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee00ca900 msgr2=0x7f7ee01f01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee00ca900 0x7f7ee01f01c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc033000 tx=0x7f7ecc0046c0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 shutdown_connections 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7ee015b0e0 0x7f7ee00c7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ec006bc20 0x7f7ec006e0e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ee00ca900 0x7f7ee01f01c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 --2- 172.21.15.6:0/3224579779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ee00ca520 0x7f7ee015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 >> 172.21.15.6:0/3224579779 conn(0x7f7ee00b16b0 msgr2=0x7f7ee015cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.957+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 shutdown_connections 2024-03-22T13:01:44.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:44.958+0000 7f7ee4a00640 1 -- 172.21.15.6:0/3224579779 wait complete. 2024-03-22T13:01:45.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:45.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:46.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.774+0000 7fe7e1400640 1 Processor -- start 2024-03-22T13:01:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.774+0000 7fe7e1400640 1 -- start start 2024-03-22T13:01:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.775+0000 7fe7e1400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.776+0000 7fe7e1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7dc0ccea0 0x7fe7dc15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.776+0000 7fe7e1400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc0ce230 con 0x7fe7dc0cf770 2024-03-22T13:01:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.776+0000 7fe7e1400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7dc0ce510 con 0x7fe7dc0ccea0 2024-03-22T13:01:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.776+0000 7fe7e1400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc0ce7f0 con 0x7fe7dc0cc3b0 2024-03-22T13:01:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.776+0000 7fe7da800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.777+0000 7fe7db200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7dc0ccea0 0x7fe7dc15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.777+0000 7fe7d9e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe7dc0cf770 0x7fe7dc0cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:41762/0 (socket says 172.21.15.6:41762) 2024-03-22T13:01:46.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.777+0000 7fe7d9e00640 1 -- 172.21.15.6:0/3635959041 learned_addr learned my addr 172.21.15.6:0/3635959041 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.779+0000 7fe7da800640 1 -- 172.21.15.6:0/3635959041 >> v2:172.21.15.6:3300/0 conn(0x7fe7dc0ccea0 msgr2=0x7fe7dc15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.779+0000 7fe7da800640 1 --2- 172.21.15.6:0/3635959041 >> v2:172.21.15.6:3300/0 conn(0x7fe7dc0ccea0 0x7fe7dc15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.779+0000 7fe7da800640 1 -- 172.21.15.6:0/3635959041 >> v1:172.21.15.92:6789/0 conn(0x7fe7dc0cf770 legacy=0x7fe7dc0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.779+0000 7fe7da800640 1 -- 172.21.15.6:0/3635959041 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc0cead0 con 0x7fe7dc0cc3b0 2024-03-22T13:01:46.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.783+0000 7fe7da800640 1 --2- 172.21.15.6:0/3635959041 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0cf230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0097a0 tx=0x7fe7cc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68c4268c286e7c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:46.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7d9400640 1 -- 172.21.15.6:0/3635959041 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0315a0 con 0x7fe7dc0cc3b0 2024-03-22T13:01:46.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7d9400640 1 -- 172.21.15.6:0/3635959041 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc037070 con 0x7fe7dc0cc3b0 2024-03-22T13:01:46.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7d9400640 1 -- 172.21.15.6:0/3635959041 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc031960 con 0x7fe7dc0cc3b0 2024-03-22T13:01:46.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7e1400640 1 -- 172.21.15.6:0/3635959041 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cc3b0 msgr2=0x7fe7dc0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:46.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3635959041 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0cf230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0097a0 tx=0x7fe7cc02b840 comp rx=0 tx=0).stop 2024-03-22T13:01:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7e1400640 1 -- 172.21.15.6:0/3635959041 shutdown_connections 2024-03-22T13:01:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3635959041 >> v2:172.21.15.6:3300/0 conn(0x7fe7dc0ccea0 0x7fe7dc15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3635959041 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0cf230 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.787+0000 7fe7e1400640 1 -- 172.21.15.6:0/3635959041 >> 172.21.15.6:0/3635959041 conn(0x7fe7dc0b16d0 msgr2=0x7fe7dc0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.788+0000 7fe7e1400640 1 -- 172.21.15.6:0/3635959041 shutdown_connections 2024-03-22T13:01:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.788+0000 7fe7e1400640 1 -- 172.21.15.6:0/3635959041 wait complete. 2024-03-22T13:01:46.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.790+0000 7fe7e1400640 1 Processor -- start 2024-03-22T13:01:46.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.790+0000 7fe7e1400640 1 -- start start 2024-03-22T13:01:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.791+0000 7fe7e1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.791+0000 7fe7e1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7dc0ccea0 0x7fe7dc1f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.791+0000 7fe7e1400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cf770 0x7fe7dc1f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.791+0000 7fe7e1400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc15a5d0 con 0x7fe7dc0ccea0 2024-03-22T13:01:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.791+0000 7fe7e1400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc06dc10 con 0x7fe7dc0cc3b0 2024-03-22T13:01:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.791+0000 7fe7e1400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7dc154bb0 con 0x7fe7dc0cf770 2024-03-22T13:01:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.792+0000 7fe7d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7dc0ccea0 0x7fe7dc1f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.792+0000 7fe7db200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cf770 0x7fe7dc1f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.792+0000 7fe7da800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.792+0000 7fe7d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7dc0ccea0 0x7fe7dc1f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50540/0 (socket says 172.21.15.6:50540) 2024-03-22T13:01:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.792+0000 7fe7d9e00640 1 -- 172.21.15.6:0/3522577002 learned_addr learned my addr 172.21.15.6:0/3522577002 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:46.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.796+0000 7fe7db200640 1 -- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7dc0cc3b0 msgr2=0x7fe7dc0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:46.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.796+0000 7fe7db200640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:46.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.796+0000 7fe7db200640 1 -- 172.21.15.6:0/3522577002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7dc0ccea0 msgr2=0x7fe7dc1f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:46.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.796+0000 7fe7db200640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7dc0ccea0 0x7fe7dc1f2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:46.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.796+0000 7fe7db200640 1 -- 172.21.15.6:0/3522577002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc0c6060 con 0x7fe7dc0cf770 2024-03-22T13:01:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.799+0000 7fe7db200640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cf770 0x7fe7dc1f2ab0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400ef90 tx=0x7fe7d400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:46.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.804+0000 7fe7d2a00640 1 -- 172.21.15.6:0/3522577002 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4018020 con 0x7fe7dc0cf770 2024-03-22T13:01:46.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.804+0000 7fe7d2a00640 1 -- 172.21.15.6:0/3522577002 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d4014480 con 0x7fe7dc0cf770 2024-03-22T13:01:46.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.804+0000 7fe7d2a00640 1 -- 172.21.15.6:0/3522577002 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4014870 con 0x7fe7dc0cf770 2024-03-22T13:01:46.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.804+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7dc15e440 con 0x7fe7dc0cf770 2024-03-22T13:01:46.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.804+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7dc0c9a60 con 0x7fe7dc0cf770 2024-03-22T13:01:46.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.806+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7dc157f00 con 0x7fe7dc0cf770 2024-03-22T13:01:46.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.843+0000 7fe7d2a00640 1 -- 172.21.15.6:0/3522577002 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe7d4005d10 con 0x7fe7dc0cf770 2024-03-22T13:01:46.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.844+0000 7fe7d2a00640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe7bc06bc20 0x7fe7bc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:46.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.845+0000 7fe7da800640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe7bc06bc20 0x7fe7bc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:46.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:46.846+0000 7fe7da800640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe7bc06bc20 0x7fe7bc06e0e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc036000 tx=0x7fe7cc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:47.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.109+0000 7fe7d2a00640 1 -- 172.21.15.6:0/3522577002 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe7d4058250 con 0x7fe7dc0cf770 2024-03-22T13:01:47.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.121+0000 7fe7d2a00640 1 -- 172.21.15.6:0/3522577002 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe7dc157f00 con 0x7fe7dc0cf770 2024-03-22T13:01:47.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.368+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7dc0756c0 con 0x7fe7dc0cf770 2024-03-22T13:01:47.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.479+0000 7fe7d2a00640 1 -- 172.21.15.6:0/3522577002 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fe7d4022030 con 0x7fe7dc0cf770 2024-03-22T13:01:47.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:47.480 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:47.480 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.481+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe7bc06bc20 msgr2=0x7fe7bc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.481+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe7bc06bc20 0x7fe7bc06e0e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc036000 tx=0x7fe7cc03d040 comp rx=0 tx=0).stop 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.481+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cf770 msgr2=0x7fe7dc1f2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cf770 0x7fe7dc1f2ab0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400ef90 tx=0x7fe7d400c490 comp rx=0 tx=0).stop 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 shutdown_connections 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7dc0cf770 0x7fe7dc1f2ab0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7dc0ccea0 0x7fe7dc1f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe7bc06bc20 0x7fe7bc06e0e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 --2- 172.21.15.6:0/3522577002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7dc0cc3b0 0x7fe7dc0c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 >> 172.21.15.6:0/3522577002 conn(0x7fe7dc0b16d0 msgr2=0x7fe7dc1577f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 shutdown_connections 2024-03-22T13:01:47.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:47.482+0000 7fe7e1400640 1 -- 172.21.15.6:0/3522577002 wait complete. 2024-03-22T13:01:48.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:48.105 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:49.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.307+0000 7f2943200640 1 Processor -- start 2024-03-22T13:01:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.307+0000 7f2943200640 1 -- start start 2024-03-22T13:01:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.310+0000 7f2943200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f293c06b080 0x7f293c06b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.310+0000 7f2943200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c0c80f0 con 0x7f293c0c5c20 2024-03-22T13:01:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.310+0000 7f2943200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c0c3bf0 con 0x7f293c150f50 2024-03-22T13:01:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.310+0000 7f2943200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293c0c3760 con 0x7f293c06b080 2024-03-22T13:01:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.311+0000 7f293be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f293c150f50 0x7f293c0c79e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:41778/0 (socket says 172.21.15.6:41778) 2024-03-22T13:01:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.311+0000 7f293be00640 1 -- 172.21.15.6:0/2125821896 learned_addr learned my addr 172.21.15.6:0/2125821896 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.311+0000 7f2941200640 1 --2- 172.21.15.6:0/2125821896 >> v2:172.21.15.6:3300/0 conn(0x7f293c06b080 0x7f293c06b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.315+0000 7f293aa00640 1 -- 172.21.15.6:0/2125821896 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92674903 0 0) 0x7f293c0c3bf0 con 0x7f293c150f50 2024-03-22T13:01:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.315+0000 7f293aa00640 1 -- 172.21.15.6:0/2125821896 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2920003520 con 0x7f293c150f50 2024-03-22T13:01:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.318+0000 7f2941200640 1 -- 172.21.15.6:0/2125821896 >> v1:172.21.15.92:6789/0 conn(0x7f293c150f50 legacy=0x7f293c0c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.318+0000 7f2941200640 1 -- 172.21.15.6:0/2125821896 >> v1:172.21.15.6:6789/0 conn(0x7f293c0c5c20 legacy=0x7f293c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.318+0000 7f2941200640 1 -- 172.21.15.6:0/2125821896 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c071e70 con 0x7f293c06b080 2024-03-22T13:01:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.323+0000 7f2941200640 1 --2- 172.21.15.6:0/2125821896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c06b080 0x7f293c06b510 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2934009d90 tx=0x7f293402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0b926ea74aa3b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:49.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.328+0000 7f293aa00640 1 -- 172.21.15.6:0/2125821896 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934039070 con 0x7f293c06b080 2024-03-22T13:01:49.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.328+0000 7f293aa00640 1 -- 172.21.15.6:0/2125821896 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293402cd80 con 0x7f293c06b080 2024-03-22T13:01:49.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.329+0000 7f293aa00640 1 -- 172.21.15.6:0/2125821896 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934034660 con 0x7f293c06b080 2024-03-22T13:01:49.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.329+0000 7f2943200640 1 -- 172.21.15.6:0/2125821896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c06b080 msgr2=0x7f293c06b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:49.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.329+0000 7f2943200640 1 --2- 172.21.15.6:0/2125821896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c06b080 0x7f293c06b510 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2934009d90 tx=0x7f293402bfd0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.330+0000 7f2943200640 1 -- 172.21.15.6:0/2125821896 shutdown_connections 2024-03-22T13:01:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.330+0000 7f2943200640 1 --2- 172.21.15.6:0/2125821896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c06b080 0x7f293c06b510 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.330+0000 7f2943200640 1 -- 172.21.15.6:0/2125821896 >> 172.21.15.6:0/2125821896 conn(0x7f293c0cb930 msgr2=0x7f293c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.330+0000 7f2943200640 1 -- 172.21.15.6:0/2125821896 shutdown_connections 2024-03-22T13:01:49.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.330+0000 7f2943200640 1 -- 172.21.15.6:0/2125821896 wait complete. 2024-03-22T13:01:49.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.332+0000 7f2943200640 1 Processor -- start 2024-03-22T13:01:49.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.333+0000 7f2943200640 1 -- start start 2024-03-22T13:01:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.334+0000 7f2943200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f293c06b080 0x7f293c0caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.334+0000 7f2943200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f293c0c5c20 0x7f293c139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.334+0000 7f2943200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c150f50 0x7f293c13a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.334+0000 7f2943200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0c3110 con 0x7f293c150f50 2024-03-22T13:01:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.334+0000 7f2943200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0c2e30 con 0x7f293c0c5c20 2024-03-22T13:01:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.334+0000 7f2943200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0c33f0 con 0x7f293c06b080 2024-03-22T13:01:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.335+0000 7f2941200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c150f50 0x7f293c13a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.335+0000 7f293b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f293c0c5c20 0x7f293c139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.335+0000 7f293be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f293c06b080 0x7f293c0caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.335+0000 7f2941200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c150f50 0x7f293c13a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50548/0 (socket says 172.21.15.6:50548) 2024-03-22T13:01:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.335+0000 7f293b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f293c0c5c20 0x7f293c139e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:59056/0 (socket says 172.21.15.6:59056) 2024-03-22T13:01:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.335+0000 7f2941200640 1 -- 172.21.15.6:0/2410989522 learned_addr learned my addr 172.21.15.6:0/2410989522 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.339+0000 7f293b400640 1 -- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f293c06b080 msgr2=0x7f293c0caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.339+0000 7f293b400640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f293c06b080 0x7f293c0caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.339+0000 7f293b400640 1 -- 172.21.15.6:0/2410989522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c150f50 msgr2=0x7f293c13a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.339+0000 7f293b400640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c150f50 0x7f293c13a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.339+0000 7f293b400640 1 -- 172.21.15.6:0/2410989522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c0c9730 con 0x7f293c0c5c20 2024-03-22T13:01:49.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.341+0000 7f293b400640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f293c0c5c20 0x7f293c139e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f292800c340 tx=0x7f292800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.344+0000 7f2933e00640 1 -- 172.21.15.6:0/2410989522 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29280052d0 con 0x7f293c0c5c20 2024-03-22T13:01:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.344+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c0cb520 con 0x7f293c0c5c20 2024-03-22T13:01:49.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.345+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c15aa30 con 0x7f293c0c5c20 2024-03-22T13:01:49.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.346+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c0756c0 con 0x7f293c0c5c20 2024-03-22T13:01:49.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.552+0000 7f2933e00640 1 -- 172.21.15.6:0/2410989522 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2928005470 con 0x7f293c0c5c20 2024-03-22T13:01:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.648+0000 7f2933e00640 1 -- 172.21.15.6:0/2410989522 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928020060 con 0x7f293c0c5c20 2024-03-22T13:01:49.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.651+0000 7f2933e00640 1 -- 172.21.15.6:0/2410989522 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f29280072e0 con 0x7f293c0c5c20 2024-03-22T13:01:49.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.653+0000 7f2933e00640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f291c06bbd0 0x7f291c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:49.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.653+0000 7f293be00640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f291c06bbd0 0x7f291c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:49.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.654+0000 7f293be00640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f291c06bbd0 0x7f291c06e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f292c005880 tx=0x7f292c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:49.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.655+0000 7f2933e00640 1 -- 172.21.15.6:0/2410989522 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f2928056900 con 0x7f293c0c5c20 2024-03-22T13:01:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.670+0000 7f2933e00640 1 -- 172.21.15.6:0/2410989522 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f293c0c80f0 con 0x7f293c0c5c20 2024-03-22T13:01:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.939+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f293c071880 con 0x7f293c0c5c20 2024-03-22T13:01:49.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.953+0000 7f2933e00640 1 -- 172.21.15.6:0/2410989522 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f2928058860 con 0x7f293c0c5c20 2024-03-22T13:01:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:49.955 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:49.955 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:49.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f291c06bbd0 msgr2=0x7f291c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:49.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f291c06bbd0 0x7f291c06e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f292c005880 tx=0x7f292c009040 comp rx=0 tx=0).stop 2024-03-22T13:01:49.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f293c0c5c20 msgr2=0x7f293c139e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f293c0c5c20 0x7f293c139e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f292800c340 tx=0x7f292800c800 comp rx=0 tx=0).stop 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 shutdown_connections 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f293c150f50 0x7f293c13a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f291c06bbd0 0x7f291c06e090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f293c0c5c20 0x7f293c139e10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 --2- 172.21.15.6:0/2410989522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f293c06b080 0x7f293c0caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 >> 172.21.15.6:0/2410989522 conn(0x7f293c0cb930 msgr2=0x7f293c13f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 shutdown_connections 2024-03-22T13:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:49.956+0000 7f2943200640 1 -- 172.21.15.6:0/2410989522 wait complete. 2024-03-22T13:01:50.578 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:50.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:51.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:51.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.772+0000 7f5f23000640 1 Processor -- start 2024-03-22T13:01:51.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.772+0000 7f5f23000640 1 -- start start 2024-03-22T13:01:51.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.773+0000 7f5f23000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c15b7a0 0x7f5f1c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:51.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.773+0000 7f5f23000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c06c440 con 0x7f5f1c0cc640 2024-03-22T13:01:51.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.773+0000 7f5f23000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c06c720 con 0x7f5f1c0cca20 2024-03-22T13:01:51.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.773+0000 7f5f23000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c06d330 con 0x7f5f1c15b7a0 2024-03-22T13:01:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.793+0000 7f5f21200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5f1c0cca20 0x7f5f1c0ca340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:41782/0 (socket says 172.21.15.6:41782) 2024-03-22T13:01:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.793+0000 7f5f21200640 1 -- 172.21.15.6:0/1031341258 learned_addr learned my addr 172.21.15.6:0/1031341258 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.822+0000 7f5f22600640 1 --2- 172.21.15.6:0/1031341258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c15b7a0 0x7f5f1c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.825+0000 7f5f22600640 1 -- 172.21.15.6:0/1031341258 >> v1:172.21.15.92:6789/0 conn(0x7f5f1c0cca20 legacy=0x7f5f1c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.825+0000 7f5f22600640 1 -- 172.21.15.6:0/1031341258 >> v1:172.21.15.6:6789/0 conn(0x7f5f1c0cc640 legacy=0x7f5f1c06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.825+0000 7f5f22600640 1 -- 172.21.15.6:0/1031341258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c06ca00 con 0x7f5f1c15b7a0 2024-03-22T13:01:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.831+0000 7f5f22600640 1 --2- 172.21.15.6:0/1031341258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c15b7a0 0x7f5f1c15e260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009dc0 tx=0x7f5f1402bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=defda43dc060a925 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.834+0000 7f5f1be00640 1 -- 172.21.15.6:0/1031341258 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14039070 con 0x7f5f1c15b7a0 2024-03-22T13:01:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.836+0000 7f5f1be00640 1 -- 172.21.15.6:0/1031341258 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1402cd60 con 0x7f5f1c15b7a0 2024-03-22T13:01:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.836+0000 7f5f1be00640 1 -- 172.21.15.6:0/1031341258 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14034660 con 0x7f5f1c15b7a0 2024-03-22T13:01:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.837+0000 7f5f23000640 1 -- 172.21.15.6:0/1031341258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c15b7a0 msgr2=0x7f5f1c15e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.837+0000 7f5f23000640 1 --2- 172.21.15.6:0/1031341258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c15b7a0 0x7f5f1c15e260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009dc0 tx=0x7f5f1402bfb0 comp rx=0 tx=0).stop 2024-03-22T13:01:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.838+0000 7f5f23000640 1 -- 172.21.15.6:0/1031341258 shutdown_connections 2024-03-22T13:01:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.838+0000 7f5f23000640 1 --2- 172.21.15.6:0/1031341258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c15b7a0 0x7f5f1c15e260 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.838+0000 7f5f23000640 1 -- 172.21.15.6:0/1031341258 >> 172.21.15.6:0/1031341258 conn(0x7f5f1c0cf9c0 msgr2=0x7f5f1c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.838+0000 7f5f23000640 1 -- 172.21.15.6:0/1031341258 shutdown_connections 2024-03-22T13:01:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.838+0000 7f5f23000640 1 -- 172.21.15.6:0/1031341258 wait complete. 2024-03-22T13:01:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.840+0000 7f5f23000640 1 Processor -- start 2024-03-22T13:01:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.841+0000 7f5f23000640 1 -- start start 2024-03-22T13:01:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.842+0000 7f5f23000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f1c0cc640 0x7f5f1c1f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.842+0000 7f5f23000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c0cca20 0x7f5f1c1584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.842+0000 7f5f23000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f1c15b7a0 0x7f5f1c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.842+0000 7f5f23000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c0caa50 con 0x7f5f1c15b7a0 2024-03-22T13:01:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.842+0000 7f5f23000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c06cfc0 con 0x7f5f1c0cc640 2024-03-22T13:01:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.842+0000 7f5f23000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c15b000 con 0x7f5f1c0cca20 2024-03-22T13:01:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.843+0000 7f5f21200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c0cca20 0x7f5f1c1584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.843+0000 7f5f21c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f1c0cc640 0x7f5f1c1f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.843+0000 7f5f21200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c0cca20 0x7f5f1c1584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46812/0 (socket says 172.21.15.6:46812) 2024-03-22T13:01:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.843+0000 7f5f21200640 1 -- 172.21.15.6:0/4269342810 learned_addr learned my addr 172.21.15.6:0/4269342810 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.847+0000 7f5f21200640 1 -- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f1c0cc640 msgr2=0x7f5f1c1f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.847+0000 7f5f21200640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f1c0cc640 0x7f5f1c1f0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.847+0000 7f5f21200640 1 -- 172.21.15.6:0/4269342810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f1c15b7a0 msgr2=0x7f5f1c1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.847+0000 7f5f21200640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f1c15b7a0 0x7f5f1c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.847+0000 7f5f21200640 1 -- 172.21.15.6:0/4269342810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c1f1130 con 0x7f5f1c0cca20 2024-03-22T13:01:51.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.850+0000 7f5f21200640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c0cca20 0x7f5f1c1584a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000c500 tx=0x7f5f1000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.853+0000 7f5f19600640 1 -- 172.21.15.6:0/4269342810 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10017070 con 0x7f5f1c0cca20 2024-03-22T13:01:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.853+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c155cf0 con 0x7f5f1c0cca20 2024-03-22T13:01:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.853+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c1570e0 con 0x7f5f1c0cca20 2024-03-22T13:01:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.854+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1c075650 con 0x7f5f1c0cca20 2024-03-22T13:01:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.856+0000 7f5f19600640 1 -- 172.21.15.6:0/4269342810 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f10005ce0 con 0x7f5f1c0cca20 2024-03-22T13:01:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:51.856+0000 7f5f19600640 1 -- 172.21.15.6:0/4269342810 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10021060 con 0x7f5f1c0cca20 2024-03-22T13:01:52.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.157+0000 7f5f19600640 1 -- 172.21.15.6:0/4269342810 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5f1001a070 con 0x7f5f1c0cca20 2024-03-22T13:01:52.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.158+0000 7f5f19600640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5efc06bbd0 0x7f5efc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:52.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.159+0000 7f5f21c00640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5efc06bbd0 0x7f5efc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.160+0000 7f5f21c00640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5efc06bbd0 0x7f5efc06e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5f08001780 tx=0x7f5f08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.163+0000 7f5f19600640 1 -- 172.21.15.6:0/4269342810 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5f10056770 con 0x7f5f1c0cca20 2024-03-22T13:01:52.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.180+0000 7f5f19600640 1 -- 172.21.15.6:0/4269342810 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5f100586d0 con 0x7f5f1c0cca20 2024-03-22T13:01:52.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.435+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f1c0af100 con 0x7f5f1c0cca20 2024-03-22T13:01:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.457+0000 7f5f19600640 1 -- 172.21.15.6:0/4269342810 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f5f1005c770 con 0x7f5f1c0cca20 2024-03-22T13:01:52.458 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:52.458 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5efc06bbd0 msgr2=0x7f5efc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5efc06bbd0 0x7f5efc06e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5f08001780 tx=0x7f5f08009040 comp rx=0 tx=0).stop 2024-03-22T13:01:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c0cca20 msgr2=0x7f5f1c1584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c0cca20 0x7f5f1c1584a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000c500 tx=0x7f5f1000c9c0 comp rx=0 tx=0).stop 2024-03-22T13:01:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 shutdown_connections 2024-03-22T13:01:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f1c15b7a0 0x7f5f1c1564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f1c0cca20 0x7f5f1c1584a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:52.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5efc06bbd0 0x7f5efc06e090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:52.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 --2- 172.21.15.6:0/4269342810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f1c0cc640 0x7f5f1c1f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:52.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 >> 172.21.15.6:0/4269342810 conn(0x7f5f1c0cf9c0 msgr2=0x7f5f1c06d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:52.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 shutdown_connections 2024-03-22T13:01:52.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:52.461+0000 7f5f23000640 1 -- 172.21.15.6:0/4269342810 wait complete. 2024-03-22T13:01:53.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:53.091 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:54.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.276+0000 7f75c3e00640 1 Processor -- start 2024-03-22T13:01:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.276+0000 7f75c3e00640 1 -- start start 2024-03-22T13:01:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.277+0000 7f75c3e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75c40c5210 0x7f75c4158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.277+0000 7f75c3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75c406ede0 0x7f75c406f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.277+0000 7f75c3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c406e0f0 con 0x7f75c40c55f0 2024-03-22T13:01:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.277+0000 7f75c3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c40c6f90 con 0x7f75c406ede0 2024-03-22T13:01:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.277+0000 7f75c3e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c406d350 con 0x7f75c40c5210 2024-03-22T13:01:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.278+0000 7f75c2a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75c40c5210 0x7f75c4158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.279+0000 7f75c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75c406ede0 0x7f75c406f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.279+0000 7f75c2a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75c40c5210 0x7f75c4158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52696/0 (socket says 172.21.15.6:52696) 2024-03-22T13:01:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.279+0000 7f75c2a00640 1 -- 172.21.15.6:0/364289163 learned_addr learned my addr 172.21.15.6:0/364289163 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.282+0000 7f75c2a00640 1 -- 172.21.15.6:0/364289163 >> v2:172.21.15.6:3300/0 conn(0x7f75c406ede0 msgr2=0x7f75c406f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.282+0000 7f75c2a00640 1 --2- 172.21.15.6:0/364289163 >> v2:172.21.15.6:3300/0 conn(0x7f75c406ede0 0x7f75c406f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.282+0000 7f75c2a00640 1 -- 172.21.15.6:0/364289163 >> v1:172.21.15.6:6789/0 conn(0x7f75c40c55f0 legacy=0x7f75c40c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.282+0000 7f75c2a00640 1 -- 172.21.15.6:0/364289163 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c40c6cb0 con 0x7f75c40c5210 2024-03-22T13:01:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.285+0000 7f75c2a00640 1 --2- 172.21.15.6:0/364289163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c40c5210 0x7f75c4158a60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f75c40c9450 tx=0x7f75b402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5330a2c1f2ea9b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.289+0000 7f75c1600640 1 -- 172.21.15.6:0/364289163 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4035020 con 0x7f75c40c5210 2024-03-22T13:01:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.289+0000 7f75c1600640 1 -- 172.21.15.6:0/364289163 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b4031870 con 0x7f75c40c5210 2024-03-22T13:01:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.289+0000 7f75c1600640 1 -- 172.21.15.6:0/364289163 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4031c40 con 0x7f75c40c5210 2024-03-22T13:01:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.290+0000 7f75c3e00640 1 -- 172.21.15.6:0/364289163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c40c5210 msgr2=0x7f75c4158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.290+0000 7f75c3e00640 1 --2- 172.21.15.6:0/364289163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c40c5210 0x7f75c4158a60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f75c40c9450 tx=0x7f75b402b910 comp rx=0 tx=0).stop 2024-03-22T13:01:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.290+0000 7f75c3e00640 1 -- 172.21.15.6:0/364289163 shutdown_connections 2024-03-22T13:01:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.290+0000 7f75c3e00640 1 --2- 172.21.15.6:0/364289163 >> v2:172.21.15.6:3300/0 conn(0x7f75c406ede0 0x7f75c406f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.290+0000 7f75c3e00640 1 --2- 172.21.15.6:0/364289163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c40c5210 0x7f75c4158a60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.290+0000 7f75c3e00640 1 -- 172.21.15.6:0/364289163 >> 172.21.15.6:0/364289163 conn(0x7f75c40cb890 msgr2=0x7f75c40cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.290+0000 7f75c3e00640 1 -- 172.21.15.6:0/364289163 shutdown_connections 2024-03-22T13:01:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.291+0000 7f75c3e00640 1 -- 172.21.15.6:0/364289163 wait complete. 2024-03-22T13:01:54.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.292+0000 7f75c3e00640 1 Processor -- start 2024-03-22T13:01:54.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.292+0000 7f75c3e00640 1 -- start start 2024-03-22T13:01:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.293+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c406ede0 0x7f75c415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.293+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75c40c5210 0x7f75c415a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.293+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c40c55f0 0x7f75c415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.293+0000 7f75c3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c406dbd0 con 0x7f75c40c55f0 2024-03-22T13:01:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.293+0000 7f75c3e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c406d8f0 con 0x7f75c406ede0 2024-03-22T13:01:54.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.293+0000 7f75c3e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40c69a0 con 0x7f75c40c5210 2024-03-22T13:01:54.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.294+0000 7f75c2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c406ede0 0x7f75c415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:54.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.294+0000 7f75c2000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75c40c5210 0x7f75c415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:54.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.294+0000 7f75c3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c40c55f0 0x7f75c415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:54.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.294+0000 7f75c3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c40c55f0 0x7f75c415b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56488/0 (socket says 172.21.15.6:56488) 2024-03-22T13:01:54.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.294+0000 7f75c3400640 1 -- 172.21.15.6:0/1867373043 learned_addr learned my addr 172.21.15.6:0/1867373043 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:54.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.298+0000 7f75c3400640 1 -- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75c40c5210 msgr2=0x7f75c415a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:54.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.298+0000 7f75c3400640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75c40c5210 0x7f75c415a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.298+0000 7f75c3400640 1 -- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c406ede0 msgr2=0x7f75c415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:54.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.298+0000 7f75c3400640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c406ede0 0x7f75c415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.298+0000 7f75c3400640 1 -- 172.21.15.6:0/1867373043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c41eb920 con 0x7f75c40c55f0 2024-03-22T13:01:54.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.301+0000 7f75c3400640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c40c55f0 0x7f75c415b8c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00efc0 tx=0x7f75bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.303+0000 7f75baa00640 1 -- 172.21.15.6:0/1867373043 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc018020 con 0x7f75c40c55f0 2024-03-22T13:01:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.303+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c4159270 con 0x7f75c40c55f0 2024-03-22T13:01:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.304+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c4159830 con 0x7f75c40c55f0 2024-03-22T13:01:54.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.305+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c40c7cd0 con 0x7f75c40c55f0 2024-03-22T13:01:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.339+0000 7f75baa00640 1 -- 172.21.15.6:0/1867373043 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc014480 con 0x7f75c40c55f0 2024-03-22T13:01:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.339+0000 7f75baa00640 1 -- 172.21.15.6:0/1867373043 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc014850 con 0x7f75c40c55f0 2024-03-22T13:01:54.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.341+0000 7f75baa00640 1 -- 172.21.15.6:0/1867373043 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f75bc005d10 con 0x7f75c40c55f0 2024-03-22T13:01:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.342+0000 7f75baa00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f75a406bc00 0x7f75a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.343+0000 7f75c2a00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f75a406bc00 0x7f75a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:54.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.344+0000 7f75c2a00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f75a406bc00 0x7f75a406e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f75b4036000 tx=0x7f75b403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.346+0000 7f75baa00640 1 -- 172.21.15.6:0/1867373043 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f75bc058070 con 0x7f75c40c55f0 2024-03-22T13:01:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.644+0000 7f75baa00640 1 -- 172.21.15.6:0/1867373043 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f75c40c7cd0 con 0x7f75c40c55f0 2024-03-22T13:01:54.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.893+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75c4075650 con 0x7f75c40c55f0 2024-03-22T13:01:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.944+0000 7f75baa00640 1 -- 172.21.15.6:0/1867373043 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f75bc022070 con 0x7f75c40c55f0 2024-03-22T13:01:54.945 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:54.945 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f75a406bc00 msgr2=0x7f75a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f75a406bc00 0x7f75a406e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f75b4036000 tx=0x7f75b403c040 comp rx=0 tx=0).stop 2024-03-22T13:01:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c40c55f0 msgr2=0x7f75c415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c40c55f0 0x7f75c415b8c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00efc0 tx=0x7f75bc00c490 comp rx=0 tx=0).stop 2024-03-22T13:01:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 shutdown_connections 2024-03-22T13:01:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75c40c55f0 0x7f75c415b8c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75c40c5210 0x7f75c415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f75a406bc00 0x7f75a406e0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 --2- 172.21.15.6:0/1867373043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c406ede0 0x7f75c415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 >> 172.21.15.6:0/1867373043 conn(0x7f75c40cb890 msgr2=0x7f75c4147d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.947+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 shutdown_connections 2024-03-22T13:01:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:54.948+0000 7f75c3e00640 1 -- 172.21.15.6:0/1867373043 wait complete. 2024-03-22T13:01:55.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:55.570 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:56.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:56.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.773+0000 7f9fb1600640 1 Processor -- start 2024-03-22T13:01:56.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.773+0000 7f9fb1600640 1 -- start start 2024-03-22T13:01:56.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.774+0000 7f9fb1600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9fac0c5040 0x7f9fac0c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:56.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.775+0000 7f9fb1600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fac1590b0 con 0x7f9fac06f8c0 2024-03-22T13:01:56.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.775+0000 7f9fb1600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fac0721f0 con 0x7f9fac152400 2024-03-22T13:01:56.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.775+0000 7f9fb1600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fac0724d0 con 0x7f9fac0c5040 2024-03-22T13:01:56.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.776+0000 7f9faaa00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9fac152400 0x7f9fac15b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40914/0 (socket says 172.21.15.6:40914) 2024-03-22T13:01:56.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.776+0000 7f9faaa00640 1 -- 172.21.15.6:0/1462741234 learned_addr learned my addr 172.21.15.6:0/1462741234 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:01:56.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.776+0000 7f9fab400640 1 --2- 172.21.15.6:0/1462741234 >> v2:172.21.15.6:3300/0 conn(0x7f9fac0c5040 0x7f9fac0c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:56.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.779+0000 7f9fa9600640 1 -- 172.21.15.6:0/1462741234 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121910986 0 0) 0x7f9fac0721f0 con 0x7f9fac152400 2024-03-22T13:01:56.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.779+0000 7f9fa9600640 1 -- 172.21.15.6:0/1462741234 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f90003520 con 0x7f9fac152400 2024-03-22T13:01:56.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.783+0000 7f9fab400640 1 -- 172.21.15.6:0/1462741234 >> v1:172.21.15.92:6789/0 conn(0x7f9fac152400 legacy=0x7f9fac15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:56.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.783+0000 7f9fab400640 1 -- 172.21.15.6:0/1462741234 >> v1:172.21.15.6:6789/0 conn(0x7f9fac06f8c0 legacy=0x7f9fac0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:56.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.783+0000 7f9fab400640 1 -- 172.21.15.6:0/1462741234 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac1582f0 con 0x7f9fac0c5040 2024-03-22T13:01:56.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.788+0000 7f9fab400640 1 --2- 172.21.15.6:0/1462741234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac0c5040 0x7f9fac0c5420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009d90 tx=0x7f9fa402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df1d27c469da264 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:56.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.793+0000 7f9fa9600640 1 -- 172.21.15.6:0/1462741234 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4039070 con 0x7f9fac0c5040 2024-03-22T13:01:56.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.793+0000 7f9fa9600640 1 -- 172.21.15.6:0/1462741234 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa402cd80 con 0x7f9fac0c5040 2024-03-22T13:01:56.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.793+0000 7f9fa9600640 1 -- 172.21.15.6:0/1462741234 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4034660 con 0x7f9fac0c5040 2024-03-22T13:01:56.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.794+0000 7f9fb1600640 1 -- 172.21.15.6:0/1462741234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac0c5040 msgr2=0x7f9fac0c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:56.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.794+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1462741234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac0c5040 0x7f9fac0c5420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009d90 tx=0x7f9fa402bfd0 comp rx=0 tx=0).stop 2024-03-22T13:01:56.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.794+0000 7f9fb1600640 1 -- 172.21.15.6:0/1462741234 shutdown_connections 2024-03-22T13:01:56.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.794+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1462741234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac0c5040 0x7f9fac0c5420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:56.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.794+0000 7f9fb1600640 1 -- 172.21.15.6:0/1462741234 >> 172.21.15.6:0/1462741234 conn(0x7f9fac0cb930 msgr2=0x7f9fac0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:56.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.794+0000 7f9fb1600640 1 -- 172.21.15.6:0/1462741234 shutdown_connections 2024-03-22T13:01:56.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.794+0000 7f9fb1600640 1 -- 172.21.15.6:0/1462741234 wait complete. 2024-03-22T13:01:56.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.797+0000 7f9fb1600640 1 Processor -- start 2024-03-22T13:01:56.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.797+0000 7f9fb1600640 1 -- start start 2024-03-22T13:01:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.799+0000 7f9fb1600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fac06f8c0 0x7f9fac156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.799+0000 7f9fb1600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fac0c5040 0x7f9fac0c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.799+0000 7f9fb1600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac152400 0x7f9fac1f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.799+0000 7f9fb1600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac158b90 con 0x7f9fac152400 2024-03-22T13:01:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.799+0000 7f9fb1600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac1588b0 con 0x7f9fac06f8c0 2024-03-22T13:01:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.799+0000 7f9fb1600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac1527e0 con 0x7f9fac0c5040 2024-03-22T13:01:56.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.800+0000 7f9faa000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fac0c5040 0x7f9fac0c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:56.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.800+0000 7f9faaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fac06f8c0 0x7f9fac156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:56.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.800+0000 7f9fab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac152400 0x7f9fac1f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:56.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.800+0000 7f9faaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fac06f8c0 0x7f9fac156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52716/0 (socket says 172.21.15.6:52716) 2024-03-22T13:01:56.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.800+0000 7f9faaa00640 1 -- 172.21.15.6:0/1870240418 learned_addr learned my addr 172.21.15.6:0/1870240418 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.804+0000 7f9fab400640 1 -- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fac0c5040 msgr2=0x7f9fac0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.804+0000 7f9fab400640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fac0c5040 0x7f9fac0c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.804+0000 7f9fab400640 1 -- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fac06f8c0 msgr2=0x7f9fac156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.804+0000 7f9fab400640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fac06f8c0 0x7f9fac156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.804+0000 7f9fab400640 1 -- 172.21.15.6:0/1870240418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac1567e0 con 0x7f9fac152400 2024-03-22T13:01:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.807+0000 7f9fab400640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac152400 0x7f9fac1f1000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4033000 tx=0x7f9fa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:56.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.812+0000 7f9fa2a00640 1 -- 172.21.15.6:0/1870240418 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4044020 con 0x7f9fac152400 2024-03-22T13:01:56.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.812+0000 7f9fa2a00640 1 -- 172.21.15.6:0/1870240418 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa403f470 con 0x7f9fac152400 2024-03-22T13:01:56.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.812+0000 7f9fa2a00640 1 -- 172.21.15.6:0/1870240418 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4039040 con 0x7f9fac152400 2024-03-22T13:01:56.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.812+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fac156ac0 con 0x7f9fac152400 2024-03-22T13:01:56.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.812+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fac157080 con 0x7f9fac152400 2024-03-22T13:01:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.813+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fac073080 con 0x7f9fac152400 2024-03-22T13:01:56.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.844+0000 7f9fa2a00640 1 -- 172.21.15.6:0/1870240418 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9fa4047020 con 0x7f9fac152400 2024-03-22T13:01:56.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.845+0000 7f9fa2a00640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9f8c06bc20 0x7f9f8c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:56.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.845+0000 7f9faaa00640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9f8c06bc20 0x7f9f8c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:56.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:56.846+0000 7f9faaa00640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9f8c06bc20 0x7f9f8c06e0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0011e0 tx=0x7f9f9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:57.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.141+0000 7f9fa2a00640 1 -- 172.21.15.6:0/1870240418 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9fa4079d00 con 0x7f9fac152400 2024-03-22T13:01:57.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.142+0000 7f9fa2a00640 1 -- 172.21.15.6:0/1870240418 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9fac073080 con 0x7f9fac152400 2024-03-22T13:01:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.396+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fac0756c0 con 0x7f9fac152400 2024-03-22T13:01:57.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.413+0000 7f9fa2a00640 1 -- 172.21.15.6:0/1870240418 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f9fa404e030 con 0x7f9fac152400 2024-03-22T13:01:57.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:57.413 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:57.413 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.415+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9f8c06bc20 msgr2=0x7f9f8c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.415+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9f8c06bc20 0x7f9f8c06e0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0011e0 tx=0x7f9f9c009040 comp rx=0 tx=0).stop 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac152400 msgr2=0x7f9fac1f1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac152400 0x7f9fac1f1000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4033000 tx=0x7f9fa4004a40 comp rx=0 tx=0).stop 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 shutdown_connections 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fac152400 0x7f9fac1f1000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fac0c5040 0x7f9fac0c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9f8c06bc20 0x7f9f8c06e0e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 --2- 172.21.15.6:0/1870240418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fac06f8c0 0x7f9fac156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 >> 172.21.15.6:0/1870240418 conn(0x7f9fac0cb930 msgr2=0x7f9fac148660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 shutdown_connections 2024-03-22T13:01:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:57.416+0000 7f9fb1600640 1 -- 172.21.15.6:0/1870240418 wait complete. 2024-03-22T13:01:58.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:01:58.035 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:01:59.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:01:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.237+0000 7ff16a400640 1 Processor -- start 2024-03-22T13:01:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.237+0000 7ff16a400640 1 -- start start 2024-03-22T13:01:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.238+0000 7ff16a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff16406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.238+0000 7ff16a400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff16415b0e0 0x7ff16415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.238+0000 7ff16a400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16406c7a0 con 0x7ff1640ca900 2024-03-22T13:01:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.238+0000 7ff16a400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1640cd000 con 0x7ff16415b0e0 2024-03-22T13:01:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.238+0000 7ff16a400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16406ca80 con 0x7ff1640ca520 2024-03-22T13:01:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.239+0000 7ff163400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff16406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.239+0000 7ff163400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff16406d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57298/0 (socket says 172.21.15.6:57298) 2024-03-22T13:01:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.239+0000 7ff163400640 1 -- 172.21.15.6:0/316382806 learned_addr learned my addr 172.21.15.6:0/316382806 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:59.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.242+0000 7ff163400640 1 -- 172.21.15.6:0/316382806 >> v2:172.21.15.92:3300/0 conn(0x7ff16415b0e0 msgr2=0x7ff16415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.242+0000 7ff163400640 1 --2- 172.21.15.6:0/316382806 >> v2:172.21.15.92:3300/0 conn(0x7ff16415b0e0 0x7ff16415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.242+0000 7ff163400640 1 -- 172.21.15.6:0/316382806 >> v1:172.21.15.92:6789/0 conn(0x7ff1640ca900 legacy=0x7ff1640cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:01:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.242+0000 7ff163400640 1 -- 172.21.15.6:0/316382806 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16406cd60 con 0x7ff1640ca520 2024-03-22T13:01:59.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.244+0000 7ff163400640 1 --2- 172.21.15.6:0/316382806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff16406d840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff154009870 tx=0x7ff15402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36f4f0c9ba05b410 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:59.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.248+0000 7ff162000640 1 -- 172.21.15.6:0/316382806 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154035020 con 0x7ff1640ca520 2024-03-22T13:01:59.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.248+0000 7ff162000640 1 -- 172.21.15.6:0/316382806 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff154031640 con 0x7ff1640ca520 2024-03-22T13:01:59.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.248+0000 7ff162000640 1 -- 172.21.15.6:0/316382806 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15402d040 con 0x7ff1640ca520 2024-03-22T13:01:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 -- 172.21.15.6:0/316382806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 msgr2=0x7ff16406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 --2- 172.21.15.6:0/316382806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff16406d840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff154009870 tx=0x7ff15402b8f0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 -- 172.21.15.6:0/316382806 shutdown_connections 2024-03-22T13:01:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 --2- 172.21.15.6:0/316382806 >> v2:172.21.15.92:3300/0 conn(0x7ff16415b0e0 0x7ff16415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 --2- 172.21.15.6:0/316382806 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff16406d840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 -- 172.21.15.6:0/316382806 >> 172.21.15.6:0/316382806 conn(0x7ff1640b16b0 msgr2=0x7ff1640b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:59.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 -- 172.21.15.6:0/316382806 shutdown_connections 2024-03-22T13:01:59.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.249+0000 7ff16a400640 1 -- 172.21.15.6:0/316382806 wait complete. 2024-03-22T13:01:59.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.252+0000 7ff16a400640 1 Processor -- start 2024-03-22T13:01:59.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.252+0000 7ff16a400640 1 -- start start 2024-03-22T13:01:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff16a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff1641e4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:59.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff16a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1640ca900 0x7ff1640b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:59.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff16a400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16415b0e0 0x7ff1641f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:59.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff16a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1640cad60 con 0x7ff1640ca900 2024-03-22T13:01:59.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff16a400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16406d320 con 0x7ff16415b0e0 2024-03-22T13:01:59.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff16a400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff164154c20 con 0x7ff1640ca520 2024-03-22T13:01:59.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff163e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16415b0e0 0x7ff1641f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:59.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.254+0000 7ff163400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff1641e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:59.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.255+0000 7ff163e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16415b0e0 0x7ff1641f3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52742/0 (socket says 172.21.15.6:52742) 2024-03-22T13:01:59.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.255+0000 7ff163e00640 1 -- 172.21.15.6:0/615408862 learned_addr learned my addr 172.21.15.6:0/615408862 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:01:59.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.259+0000 7ff163e00640 1 -- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 msgr2=0x7ff1641e4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:59.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.259+0000 7ff163e00640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff1641e4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.259+0000 7ff163e00640 1 -- 172.21.15.6:0/615408862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1640ca900 msgr2=0x7ff1640b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:59.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.259+0000 7ff163e00640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1640ca900 0x7ff1640b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.259+0000 7ff163e00640 1 -- 172.21.15.6:0/615408862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1640b5540 con 0x7ff16415b0e0 2024-03-22T13:01:59.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.261+0000 7ff163e00640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16415b0e0 0x7ff1641f3ec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00b9c0 tx=0x7ff15c00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:59.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.264+0000 7ff15b400640 1 -- 172.21.15.6:0/615408862 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c019070 con 0x7ff16415b0e0 2024-03-22T13:01:59.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.264+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1640c8870 con 0x7ff16415b0e0 2024-03-22T13:01:59.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.264+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1640c8e30 con 0x7ff16415b0e0 2024-03-22T13:01:59.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.266+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1640756c0 con 0x7ff16415b0e0 2024-03-22T13:01:59.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.267+0000 7ff15b400640 1 -- 172.21.15.6:0/615408862 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15c00cba0 con 0x7ff16415b0e0 2024-03-22T13:01:59.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.267+0000 7ff15b400640 1 -- 172.21.15.6:0/615408862 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c023060 con 0x7ff16415b0e0 2024-03-22T13:01:59.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.544+0000 7ff15b400640 1 -- 172.21.15.6:0/615408862 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff15c01c070 con 0x7ff16415b0e0 2024-03-22T13:01:59.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.547+0000 7ff15b400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff14406bbd0 0x7ff14406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:01:59.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.547+0000 7ff15b400640 1 -- 172.21.15.6:0/615408862 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff15c08b3e0 con 0x7ff16415b0e0 2024-03-22T13:01:59.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.547+0000 7ff163400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff14406bbd0 0x7ff14406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:01:59.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.548+0000 7ff163400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff14406bbd0 0x7ff14406e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff154036000 tx=0x7ff15403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:01:59.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.563+0000 7ff15b400640 1 -- 172.21.15.6:0/615408862 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff15c0594d0 con 0x7ff16415b0e0 2024-03-22T13:01:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.820+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1640cb080 con 0x7ff16415b0e0 2024-03-22T13:01:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.833+0000 7ff15b400640 1 -- 172.21.15.6:0/615408862 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7ff15c05d570 con 0x7ff16415b0e0 2024-03-22T13:01:59.833 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:01:59.834 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:01:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff14406bbd0 msgr2=0x7ff14406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff14406bbd0 0x7ff14406e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff154036000 tx=0x7ff15403c040 comp rx=0 tx=0).stop 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16415b0e0 msgr2=0x7ff1641f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16415b0e0 0x7ff1641f3ec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00b9c0 tx=0x7ff15c00be80 comp rx=0 tx=0).stop 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 shutdown_connections 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff14406bbd0 0x7ff14406e090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff16415b0e0 0x7ff1641f3ec0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1640ca900 0x7ff1640b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 --2- 172.21.15.6:0/615408862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1640ca520 0x7ff1641e4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:01:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 >> 172.21.15.6:0/615408862 conn(0x7ff1640b16b0 msgr2=0x7ff1640affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:01:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 shutdown_connections 2024-03-22T13:01:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:01:59.836+0000 7ff16a400640 1 -- 172.21.15.6:0/615408862 wait complete. 2024-03-22T13:02:00.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:00.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:00.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:02:00.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:02:01.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:01.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.651+0000 7fedc5000640 1 Processor -- start 2024-03-22T13:02:01.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.651+0000 7fedc5000640 1 -- start start 2024-03-22T13:02:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.653+0000 7fedc5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fedc00cc380 0x7fedc006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.653+0000 7fedc5000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc015b240 con 0x7fedc015ae30 2024-03-22T13:02:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.653+0000 7fedc5000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00cee00 con 0x7fedc00ce8a0 2024-03-22T13:02:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.653+0000 7fedc5000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc00cf0c0 con 0x7fedc00cc380 2024-03-22T13:02:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.654+0000 7fedb7e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fedc00ce8a0 0x7fedc00cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40940/0 (socket says 172.21.15.6:40940) 2024-03-22T13:02:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.654+0000 7fedb7e00640 1 -- 172.21.15.6:0/382539918 learned_addr learned my addr 172.21.15.6:0/382539918 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.654+0000 7fedbf400640 1 --2- 172.21.15.6:0/382539918 >> v2:172.21.15.6:3300/0 conn(0x7fedc00cc380 0x7fedc006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.657+0000 7fedbea00640 1 -- 172.21.15.6:0/382539918 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135419546 0 0) 0x7fedc00cee00 con 0x7fedc00ce8a0 2024-03-22T13:02:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.657+0000 7fedbea00640 1 -- 172.21.15.6:0/382539918 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4003520 con 0x7fedc00ce8a0 2024-03-22T13:02:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.658+0000 7fedbf400640 1 -- 172.21.15.6:0/382539918 >> v1:172.21.15.92:6789/0 conn(0x7fedc00ce8a0 legacy=0x7fedc00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.658+0000 7fedbf400640 1 -- 172.21.15.6:0/382539918 >> v1:172.21.15.6:6789/0 conn(0x7fedc015ae30 legacy=0x7fedc0164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.658+0000 7fedbf400640 1 -- 172.21.15.6:0/382539918 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc00cf3a0 con 0x7fedc00cc380 2024-03-22T13:02:01.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.664+0000 7fedbf400640 1 --2- 172.21.15.6:0/382539918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc006c230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fedb00097a0 tx=0x7fedb002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2c98f5ed2ad91b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.668+0000 7fedbea00640 1 -- 172.21.15.6:0/382539918 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0034070 con 0x7fedc00cc380 2024-03-22T13:02:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.671+0000 7fedbea00640 1 -- 172.21.15.6:0/382539918 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb0030610 con 0x7fedc00cc380 2024-03-22T13:02:01.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.671+0000 7fedbea00640 1 -- 172.21.15.6:0/382539918 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb00309c0 con 0x7fedc00cc380 2024-03-22T13:02:01.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.671+0000 7fedc5000640 1 -- 172.21.15.6:0/382539918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 msgr2=0x7fedc006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:01.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.671+0000 7fedc5000640 1 --2- 172.21.15.6:0/382539918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc006c230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fedb00097a0 tx=0x7fedb002b960 comp rx=0 tx=0).stop 2024-03-22T13:02:01.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.672+0000 7fedc5000640 1 -- 172.21.15.6:0/382539918 shutdown_connections 2024-03-22T13:02:01.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.672+0000 7fedc5000640 1 --2- 172.21.15.6:0/382539918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc006c230 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:01.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.672+0000 7fedc5000640 1 -- 172.21.15.6:0/382539918 >> 172.21.15.6:0/382539918 conn(0x7fedc00b1480 msgr2=0x7fedc00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:01.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.672+0000 7fedc5000640 1 -- 172.21.15.6:0/382539918 shutdown_connections 2024-03-22T13:02:01.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.672+0000 7fedc5000640 1 -- 172.21.15.6:0/382539918 wait complete. 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.674+0000 7fedc5000640 1 Processor -- start 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.674+0000 7fedc5000640 1 -- start start 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedc5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc01501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedc5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedc00ce8a0 0x7fedc01506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedc5000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedc015ae30 0x7fedc00c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedc5000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc015a910 con 0x7fedc00cc380 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedc5000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc015a630 con 0x7fedc00ce8a0 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedc5000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc01549d0 con 0x7fedc015ae30 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedbf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc01501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.675+0000 7fedb7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedc00ce8a0 0x7fedc01506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.676+0000 7fedbfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedc015ae30 0x7fedc00c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.676+0000 7fedbf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc01501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56536/0 (socket says 172.21.15.6:56536) 2024-03-22T13:02:01.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.676+0000 7fedbf400640 1 -- 172.21.15.6:0/986166711 learned_addr learned my addr 172.21.15.6:0/986166711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.679+0000 7fedbf400640 1 -- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedc015ae30 msgr2=0x7fedc00c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.679+0000 7fedbf400640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedc015ae30 0x7fedc00c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.679+0000 7fedbf400640 1 -- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedc00ce8a0 msgr2=0x7fedc01506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.679+0000 7fedbf400640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedc00ce8a0 0x7fedc01506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.679+0000 7fedbf400640 1 -- 172.21.15.6:0/986166711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc014fcd0 con 0x7fedc00cc380 2024-03-22T13:02:01.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.682+0000 7fedbf400640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc01501b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedb002b9a0 tx=0x7fedb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.687+0000 7fedb7400640 1 -- 172.21.15.6:0/986166711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0034050 con 0x7fedc00cc380 2024-03-22T13:02:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.687+0000 7fedb7400640 1 -- 172.21.15.6:0/986166711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb0003820 con 0x7fedc00cc380 2024-03-22T13:02:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.687+0000 7fedb7400640 1 -- 172.21.15.6:0/986166711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0003bf0 con 0x7fedc00cc380 2024-03-22T13:02:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.687+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc01517e0 con 0x7fedc00cc380 2024-03-22T13:02:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.687+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc00c9a10 con 0x7fedc00cc380 2024-03-22T13:02:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.689+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc014fea0 con 0x7fedc00cc380 2024-03-22T13:02:01.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.984+0000 7fedb7400640 1 -- 172.21.15.6:0/986166711 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fedb0037070 con 0x7fedc00cc380 2024-03-22T13:02:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.984+0000 7fedb7400640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed9c06bc20 0x7fed9c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:01.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.985+0000 7fedb7e00640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed9c06bc20 0x7fed9c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:01.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:01.986+0000 7fedb7e00640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed9c06bc20 0x7fed9c06e0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fedc00b0d70 tx=0x7feda8007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.023+0000 7fedb7400640 1 -- 172.21.15.6:0/986166711 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fedb00791b0 con 0x7fedc00cc380 2024-03-22T13:02:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.024+0000 7fedb7400640 1 -- 172.21.15.6:0/986166711 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fedb007b110 con 0x7fedc00cc380 2024-03-22T13:02:02.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.284+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedc015a2e0 con 0x7fedc00cc380 2024-03-22T13:02:02.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.312+0000 7fedb7400640 1 -- 172.21.15.6:0/986166711 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fedc015a2e0 con 0x7fedc00cc380 2024-03-22T13:02:02.312 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:02.313 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:02.313 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed9c06bc20 msgr2=0x7fed9c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed9c06bc20 0x7fed9c06e0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fedc00b0d70 tx=0x7feda8007d30 comp rx=0 tx=0).stop 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 msgr2=0x7fedc01501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc01501b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedb002b9a0 tx=0x7fedb00046c0 comp rx=0 tx=0).stop 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 shutdown_connections 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedc015ae30 0x7fedc00c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed9c06bc20 0x7fed9c06e0e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedc00ce8a0 0x7fedc01506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 --2- 172.21.15.6:0/986166711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedc00cc380 0x7fedc01501b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 >> 172.21.15.6:0/986166711 conn(0x7fedc00b1480 msgr2=0x7fedc0158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 shutdown_connections 2024-03-22T13:02:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:02.315+0000 7fedc5000640 1 -- 172.21.15.6:0/986166711 wait complete. 2024-03-22T13:02:02.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:02.937 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:03.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:04.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.142+0000 7f9bf1e00640 1 Processor -- start 2024-03-22T13:02:04.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.143+0000 7f9bf1e00640 1 -- start start 2024-03-22T13:02:04.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.144+0000 7f9bf1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bec06d840 0x7f9bec15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:04.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.144+0000 7f9bf1e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec06c7a0 con 0x7f9bec0ccad0 2024-03-22T13:02:04.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.144+0000 7f9bf1e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec06ca80 con 0x7f9bec0ca520 2024-03-22T13:02:04.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.144+0000 7f9bf1e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec0cd000 con 0x7f9bec06d840 2024-03-22T13:02:04.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.145+0000 7f9beb400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bec06d840 0x7f9bec15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:04.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.145+0000 7f9bea000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9bec0ccad0 0x7f9bec162da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50672/0 (socket says 172.21.15.6:50672) 2024-03-22T13:02:04.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.145+0000 7f9bea000640 1 -- 172.21.15.6:0/18543828 learned_addr learned my addr 172.21.15.6:0/18543828 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:04.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.149+0000 7f9beb400640 1 -- 172.21.15.6:0/18543828 >> v1:172.21.15.92:6789/0 conn(0x7f9bec0ca520 legacy=0x7f9bec0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:04.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.149+0000 7f9beb400640 1 -- 172.21.15.6:0/18543828 >> v1:172.21.15.6:6789/0 conn(0x7f9bec0ccad0 legacy=0x7f9bec162da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:04.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.149+0000 7f9beb400640 1 -- 172.21.15.6:0/18543828 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec06cd60 con 0x7f9bec06d840 2024-03-22T13:02:04.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.154+0000 7f9beb400640 1 --2- 172.21.15.6:0/18543828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec06d840 0x7f9bec15b0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009d90 tx=0x7f9be402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a70a0d586dd840d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:04.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.159+0000 7f9be9600640 1 -- 172.21.15.6:0/18543828 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4039070 con 0x7f9bec06d840 2024-03-22T13:02:04.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.159+0000 7f9be9600640 1 -- 172.21.15.6:0/18543828 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be402cd00 con 0x7f9bec06d840 2024-03-22T13:02:04.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.159+0000 7f9be9600640 1 -- 172.21.15.6:0/18543828 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4034660 con 0x7f9bec06d840 2024-03-22T13:02:04.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.160+0000 7f9bf1e00640 1 -- 172.21.15.6:0/18543828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec06d840 msgr2=0x7f9bec15b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:04.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.160+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/18543828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec06d840 0x7f9bec15b0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009d90 tx=0x7f9be402bf50 comp rx=0 tx=0).stop 2024-03-22T13:02:04.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.160+0000 7f9bf1e00640 1 -- 172.21.15.6:0/18543828 shutdown_connections 2024-03-22T13:02:04.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.160+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/18543828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec06d840 0x7f9bec15b0e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:04.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.160+0000 7f9bf1e00640 1 -- 172.21.15.6:0/18543828 >> 172.21.15.6:0/18543828 conn(0x7f9bec0b16b0 msgr2=0x7f9bec0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:04.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.160+0000 7f9bf1e00640 1 -- 172.21.15.6:0/18543828 shutdown_connections 2024-03-22T13:02:04.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.160+0000 7f9bf1e00640 1 -- 172.21.15.6:0/18543828 wait complete. 2024-03-22T13:02:04.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.162+0000 7f9bf1e00640 1 Processor -- start 2024-03-22T13:02:04.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.162+0000 7f9bf1e00640 1 -- start start 2024-03-22T13:02:04.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.163+0000 7f9bf1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bec06d840 0x7f9bec1e3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:04.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.163+0000 7f9bf1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec0ca520 0x7f9bec15c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:04.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.163+0000 7f9bf1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bec0ccad0 0x7f9bec1598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:04.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.163+0000 7f9bf1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec154c20 con 0x7f9bec06d840 2024-03-22T13:02:04.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.163+0000 7f9bf1e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec06d320 con 0x7f9bec0ca520 2024-03-22T13:02:04.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.163+0000 7f9bf1e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec15b980 con 0x7f9bec0ccad0 2024-03-22T13:02:04.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.164+0000 7f9bea000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec0ca520 0x7f9bec15c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:04.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.164+0000 7f9beb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bec0ccad0 0x7f9bec1598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:04.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.164+0000 7f9beaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bec06d840 0x7f9bec1e3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:04.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.164+0000 7f9beb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bec0ccad0 0x7f9bec1598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44534/0 (socket says 172.21.15.6:44534) 2024-03-22T13:02:04.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.164+0000 7f9beb400640 1 -- 172.21.15.6:0/1282127943 learned_addr learned my addr 172.21.15.6:0/1282127943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:04.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.168+0000 7f9beb400640 1 -- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec0ca520 msgr2=0x7f9bec15c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:04.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.168+0000 7f9beb400640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec0ca520 0x7f9bec15c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:04.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.168+0000 7f9beb400640 1 -- 172.21.15.6:0/1282127943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bec06d840 msgr2=0x7f9bec1e3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:04.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.168+0000 7f9beb400640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bec06d840 0x7f9bec1e3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:04.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.168+0000 7f9beb400640 1 -- 172.21.15.6:0/1282127943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec1e3dc0 con 0x7f9bec0ccad0 2024-03-22T13:02:04.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.170+0000 7f9beb400640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bec0ccad0 0x7f9bec1598f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009eb0 tx=0x7f9be4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:04.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.174+0000 7f9be3400640 1 -- 172.21.15.6:0/1282127943 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4044020 con 0x7f9bec0ccad0 2024-03-22T13:02:04.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.174+0000 7f9be3400640 1 -- 172.21.15.6:0/1282127943 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be403f670 con 0x7f9bec0ccad0 2024-03-22T13:02:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.174+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec1574d0 con 0x7f9bec0ccad0 2024-03-22T13:02:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.175+0000 7f9be3400640 1 -- 172.21.15.6:0/1282127943 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4039040 con 0x7f9bec0ccad0 2024-03-22T13:02:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.175+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec157a90 con 0x7f9bec0ccad0 2024-03-22T13:02:04.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.176+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec0756c0 con 0x7f9bec0ccad0 2024-03-22T13:02:04.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.199+0000 7f9be3400640 1 -- 172.21.15.6:0/1282127943 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9be4047020 con 0x7f9bec0ccad0 2024-03-22T13:02:04.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.200+0000 7f9be3400640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bcc06bc20 0x7f9bcc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.200+0000 7f9beaa00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bcc06bc20 0x7f9bcc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:04.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.201+0000 7f9beaa00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bcc06bc20 0x7f9bcc06e0e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc001780 tx=0x7f9bdc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:04.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.438+0000 7f9be3400640 1 -- 172.21.15.6:0/1282127943 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9be4079f20 con 0x7f9bec0ccad0 2024-03-22T13:02:04.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.545+0000 7f9be3400640 1 -- 172.21.15.6:0/1282127943 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9be404e030 con 0x7f9bec0ccad0 2024-03-22T13:02:04.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.806+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bec0cb080 con 0x7f9bec0ccad0 2024-03-22T13:02:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.822+0000 7f9be3400640 1 -- 172.21.15.6:0/1282127943 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f9be407be80 con 0x7f9bec0ccad0 2024-03-22T13:02:04.822 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:04.822 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:04.823 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bcc06bc20 msgr2=0x7f9bcc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bcc06bc20 0x7f9bcc06e0e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc001780 tx=0x7f9bdc009040 comp rx=0 tx=0).stop 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bec0ccad0 msgr2=0x7f9bec1598f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bec0ccad0 0x7f9bec1598f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009eb0 tx=0x7f9be4004a40 comp rx=0 tx=0).stop 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 shutdown_connections 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bec0ccad0 0x7f9bec1598f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9bcc06bc20 0x7f9bcc06e0e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bec0ca520 0x7f9bec15c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 --2- 172.21.15.6:0/1282127943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bec06d840 0x7f9bec1e3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 >> 172.21.15.6:0/1282127943 conn(0x7f9bec0b16b0 msgr2=0x7f9bec15aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 shutdown_connections 2024-03-22T13:02:04.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:04.825+0000 7f9bf1e00640 1 -- 172.21.15.6:0/1282127943 wait complete. 2024-03-22T13:02:05.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:05.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:06.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:06.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.644+0000 7f0be4e00640 1 Processor -- start 2024-03-22T13:02:06.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.644+0000 7f0be4e00640 1 -- start start 2024-03-22T13:02:06.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.645+0000 7f0be4e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be00ce8a0 0x7f0be006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:06.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.645+0000 7f0be4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0be015ae30 0x7f0be00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.646+0000 7f0be4e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be015b260 con 0x7f0be00cc380 2024-03-22T13:02:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.646+0000 7f0be4e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be00cee00 con 0x7f0be015ae30 2024-03-22T13:02:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.646+0000 7f0be4e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be00cf0c0 con 0x7f0be00ce8a0 2024-03-22T13:02:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.646+0000 7f0bdea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be00ce8a0 0x7f0be006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.647+0000 7f0bdea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be00ce8a0 0x7f0be006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44544/0 (socket says 172.21.15.6:44544) 2024-03-22T13:02:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.647+0000 7f0bdea00640 1 -- 172.21.15.6:0/3981504707 learned_addr learned my addr 172.21.15.6:0/3981504707 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.649+0000 7f0bdea00640 1 -- 172.21.15.6:0/3981504707 >> v2:172.21.15.6:3300/0 conn(0x7f0be015ae30 msgr2=0x7f0be00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.649+0000 7f0bdea00640 1 --2- 172.21.15.6:0/3981504707 >> v2:172.21.15.6:3300/0 conn(0x7f0be015ae30 0x7f0be00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.649+0000 7f0bdea00640 1 -- 172.21.15.6:0/3981504707 >> v1:172.21.15.6:6789/0 conn(0x7f0be00cc380 legacy=0x7f0be00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:06.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.649+0000 7f0bdea00640 1 -- 172.21.15.6:0/3981504707 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be00cf3a0 con 0x7f0be00ce8a0 2024-03-22T13:02:06.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.651+0000 7f0bdea00640 1 --2- 172.21.15.6:0/3981504707 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be00ce8a0 0x7f0be006c230 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009870 tx=0x7f0bcc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e82e398ec4d44efc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:06.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.656+0000 7f0bde000640 1 -- 172.21.15.6:0/3981504707 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc035020 con 0x7f0be00ce8a0 2024-03-22T13:02:06.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.656+0000 7f0bde000640 1 -- 172.21.15.6:0/3981504707 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc031640 con 0x7f0be00ce8a0 2024-03-22T13:02:06.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.656+0000 7f0bde000640 1 -- 172.21.15.6:0/3981504707 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc02d040 con 0x7f0be00ce8a0 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 -- 172.21.15.6:0/3981504707 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be00ce8a0 msgr2=0x7f0be006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 --2- 172.21.15.6:0/3981504707 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be00ce8a0 0x7f0be006c230 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009870 tx=0x7f0bcc02b8f0 comp rx=0 tx=0).stop 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 -- 172.21.15.6:0/3981504707 shutdown_connections 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 --2- 172.21.15.6:0/3981504707 >> v2:172.21.15.6:3300/0 conn(0x7f0be015ae30 0x7f0be00cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 --2- 172.21.15.6:0/3981504707 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be00ce8a0 0x7f0be006c230 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 -- 172.21.15.6:0/3981504707 >> 172.21.15.6:0/3981504707 conn(0x7f0be00b1480 msgr2=0x7f0be00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 -- 172.21.15.6:0/3981504707 shutdown_connections 2024-03-22T13:02:06.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.657+0000 7f0be4e00640 1 -- 172.21.15.6:0/3981504707 wait complete. 2024-03-22T13:02:06.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.659+0000 7f0be4e00640 1 Processor -- start 2024-03-22T13:02:06.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.659+0000 7f0be4e00640 1 -- start start 2024-03-22T13:02:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.660+0000 7f0be4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0be00cc380 0x7f0be00b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.660+0000 7f0be4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be00ce8a0 0x7f0be015ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.660+0000 7f0be4e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be015ae30 0x7f0be015b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.660+0000 7f0be4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be015a910 con 0x7f0be00ce8a0 2024-03-22T13:02:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.660+0000 7f0be4e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be015a630 con 0x7f0be00cc380 2024-03-22T13:02:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.660+0000 7f0be4e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be01549d0 con 0x7f0be015ae30 2024-03-22T13:02:06.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.661+0000 7f0bdfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be015ae30 0x7f0be015b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:06.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.661+0000 7f0bdf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0be00cc380 0x7f0be00b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:06.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.661+0000 7f0bdfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be015ae30 0x7f0be015b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44558/0 (socket says 172.21.15.6:44558) 2024-03-22T13:02:06.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.661+0000 7f0bdfe00640 1 -- 172.21.15.6:0/1378937536 learned_addr learned my addr 172.21.15.6:0/1378937536 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:06.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.665+0000 7f0bdfe00640 1 -- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0be00cc380 msgr2=0x7f0be00b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:06.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.665+0000 7f0bdfe00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0be00cc380 0x7f0be00b0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:06.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.665+0000 7f0bdfe00640 1 -- 172.21.15.6:0/1378937536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be00ce8a0 msgr2=0x7f0be015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:06.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.665+0000 7f0bdfe00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be00ce8a0 0x7f0be015ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:06.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.665+0000 7f0bdfe00640 1 -- 172.21.15.6:0/1378937536 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be0153ad0 con 0x7f0be015ae30 2024-03-22T13:02:06.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.668+0000 7f0bdfe00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be015ae30 0x7f0be015b6b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b970 tx=0x7f0bd800be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.671+0000 7f0bd7400640 1 -- 172.21.15.6:0/1378937536 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8019070 con 0x7f0be015ae30 2024-03-22T13:02:06.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.671+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be0153d90 con 0x7f0be015ae30 2024-03-22T13:02:06.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.671+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be0152210 con 0x7f0be015ae30 2024-03-22T13:02:06.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.672+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be0075650 con 0x7f0be015ae30 2024-03-22T13:02:06.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.674+0000 7f0bd7400640 1 -- 172.21.15.6:0/1378937536 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd800cb40 con 0x7f0be015ae30 2024-03-22T13:02:06.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.674+0000 7f0bd7400640 1 -- 172.21.15.6:0/1378937536 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8023060 con 0x7f0be015ae30 2024-03-22T13:02:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.935+0000 7f0bd7400640 1 -- 172.21.15.6:0/1378937536 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0bd801c070 con 0x7f0be015ae30 2024-03-22T13:02:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.938+0000 7f0bd7400640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0bc006bbd0 0x7f0bc006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.938+0000 7f0bd7400640 1 -- 172.21.15.6:0/1378937536 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0bd808b350 con 0x7f0be015ae30 2024-03-22T13:02:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.938+0000 7f0bdf400640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0bc006bbd0 0x7f0bc006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.939+0000 7f0bdf400640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0bc006bbd0 0x7f0bc006e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006fd0 tx=0x7f0bd0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:06.950+0000 7f0bd7400640 1 -- 172.21.15.6:0/1378937536 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0bd8059440 con 0x7f0be015ae30 2024-03-22T13:02:07.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.192+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0be015e680 con 0x7f0be015ae30 2024-03-22T13:02:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.206+0000 7f0bd7400640 1 -- 172.21.15.6:0/1378937536 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f0bd805d4e0 con 0x7f0be015ae30 2024-03-22T13:02:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:07.207 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0bc006bbd0 msgr2=0x7f0bc006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0bc006bbd0 0x7f0bc006e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006fd0 tx=0x7f0bd0009060 comp rx=0 tx=0).stop 2024-03-22T13:02:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be015ae30 msgr2=0x7f0be015b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be015ae30 0x7f0be015b6b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b970 tx=0x7f0bd800be30 comp rx=0 tx=0).stop 2024-03-22T13:02:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 shutdown_connections 2024-03-22T13:02:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0be015ae30 0x7f0be015b6b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be00ce8a0 0x7f0be015ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0bc006bbd0 0x7f0bc006e090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 --2- 172.21.15.6:0/1378937536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0be00cc380 0x7f0be00b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 >> 172.21.15.6:0/1378937536 conn(0x7f0be00b1480 msgr2=0x7f0be00c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.209+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 shutdown_connections 2024-03-22T13:02:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:07.210+0000 7f0be4e00640 1 -- 172.21.15.6:0/1378937536 wait complete. 2024-03-22T13:02:07.831 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:07.831 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:08.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:09.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.035+0000 7fd360000640 1 Processor -- start 2024-03-22T13:02:09.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.036+0000 7fd360000640 1 -- start start 2024-03-22T13:02:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.037+0000 7fd360000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd3580ccd80 0x7fd3580cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.037+0000 7fd360000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cac20 0x7fd35806c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.037+0000 7fd360000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35815a6b0 con 0x7fd3580cf5c0 2024-03-22T13:02:09.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.038+0000 7fd360000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd358154d60 con 0x7fd3580ccd80 2024-03-22T13:02:09.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.038+0000 7fd360000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35815b4d0 con 0x7fd3580cac20 2024-03-22T13:02:09.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.038+0000 7fd35e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cac20 0x7fd35806c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:09.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.038+0000 7fd35cc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd3580cf5c0 0x7fd3580ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50680/0 (socket says 172.21.15.6:50680) 2024-03-22T13:02:09.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.038+0000 7fd35cc00640 1 -- 172.21.15.6:0/150460890 learned_addr learned my addr 172.21.15.6:0/150460890 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:09.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.038+0000 7fd35d600640 1 --2- 172.21.15.6:0/150460890 >> v2:172.21.15.92:3300/0 conn(0x7fd3580ccd80 0x7fd3580cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:09.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.041+0000 7fd35e000640 1 -- 172.21.15.6:0/150460890 >> v2:172.21.15.92:3300/0 conn(0x7fd3580ccd80 msgr2=0x7fd3580cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:09.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.041+0000 7fd35e000640 1 --2- 172.21.15.6:0/150460890 >> v2:172.21.15.92:3300/0 conn(0x7fd3580ccd80 0x7fd3580cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.041+0000 7fd35e000640 1 -- 172.21.15.6:0/150460890 >> v1:172.21.15.6:6789/0 conn(0x7fd3580cf5c0 legacy=0x7fd3580ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:09.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.041+0000 7fd35e000640 1 -- 172.21.15.6:0/150460890 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35815a950 con 0x7fd3580cac20 2024-03-22T13:02:09.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.044+0000 7fd35e000640 1 --2- 172.21.15.6:0/150460890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cac20 0x7fd35806c9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd350009d90 tx=0x7fd35002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7b1e7b763034f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:09.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.048+0000 7fd357e00640 1 -- 172.21.15.6:0/150460890 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350039070 con 0x7fd3580cac20 2024-03-22T13:02:09.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.048+0000 7fd357e00640 1 -- 172.21.15.6:0/150460890 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35002cc50 con 0x7fd3580cac20 2024-03-22T13:02:09.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.048+0000 7fd357e00640 1 -- 172.21.15.6:0/150460890 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350034440 con 0x7fd3580cac20 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.049+0000 7fd360000640 1 -- 172.21.15.6:0/150460890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cac20 msgr2=0x7fd35806c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.049+0000 7fd360000640 1 --2- 172.21.15.6:0/150460890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cac20 0x7fd35806c9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd350009d90 tx=0x7fd35002bea0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.050+0000 7fd360000640 1 -- 172.21.15.6:0/150460890 shutdown_connections 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.050+0000 7fd360000640 1 --2- 172.21.15.6:0/150460890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cac20 0x7fd35806c9c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.050+0000 7fd360000640 1 --2- 172.21.15.6:0/150460890 >> v2:172.21.15.92:3300/0 conn(0x7fd3580ccd80 0x7fd3580cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.050+0000 7fd360000640 1 -- 172.21.15.6:0/150460890 >> 172.21.15.6:0/150460890 conn(0x7fd3580b16c0 msgr2=0x7fd3580b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.050+0000 7fd360000640 1 -- 172.21.15.6:0/150460890 shutdown_connections 2024-03-22T13:02:09.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.050+0000 7fd360000640 1 -- 172.21.15.6:0/150460890 wait complete. 2024-03-22T13:02:09.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.052+0000 7fd360000640 1 Processor -- start 2024-03-22T13:02:09.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.053+0000 7fd360000640 1 -- start start 2024-03-22T13:02:09.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.054+0000 7fd360000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3580cac20 0x7fd35815be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:09.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.054+0000 7fd360000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3580ccd80 0x7fd3581f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:09.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.054+0000 7fd360000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cf5c0 0x7fd35815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:09.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.054+0000 7fd360000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35815b1f0 con 0x7fd3580ccd80 2024-03-22T13:02:09.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.054+0000 7fd360000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35815af10 con 0x7fd3580cac20 2024-03-22T13:02:09.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.054+0000 7fd360000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35806db10 con 0x7fd3580cf5c0 2024-03-22T13:02:09.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.055+0000 7fd35e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cf5c0 0x7fd35815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:09.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.055+0000 7fd35d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3580cac20 0x7fd35815be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:09.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.055+0000 7fd35cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3580ccd80 0x7fd3581f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:09.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.055+0000 7fd35e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cf5c0 0x7fd35815e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44578/0 (socket says 172.21.15.6:44578) 2024-03-22T13:02:09.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.055+0000 7fd35e000640 1 -- 172.21.15.6:0/952048801 learned_addr learned my addr 172.21.15.6:0/952048801 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:09.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.059+0000 7fd35e000640 1 -- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3580cac20 msgr2=0x7fd35815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:09.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.059+0000 7fd35e000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3580cac20 0x7fd35815be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.059+0000 7fd35e000640 1 -- 172.21.15.6:0/952048801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3580ccd80 msgr2=0x7fd3581f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:09.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.059+0000 7fd35e000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3580ccd80 0x7fd3581f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.059+0000 7fd35e000640 1 -- 172.21.15.6:0/952048801 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3581f5270 con 0x7fd3580cf5c0 2024-03-22T13:02:09.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.062+0000 7fd35e000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cf5c0 0x7fd35815e1e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd350009d90 tx=0x7fd350004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:09.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.067+0000 7fd355600640 1 -- 172.21.15.6:0/952048801 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350039050 con 0x7fd3580cf5c0 2024-03-22T13:02:09.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.067+0000 7fd355600640 1 -- 172.21.15.6:0/952048801 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd350045070 con 0x7fd3580cf5c0 2024-03-22T13:02:09.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.067+0000 7fd355600640 1 -- 172.21.15.6:0/952048801 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350009450 con 0x7fd3580cf5c0 2024-03-22T13:02:09.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.067+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35815b8a0 con 0x7fd3580cf5c0 2024-03-22T13:02:09.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.067+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3581f4530 con 0x7fd3580cf5c0 2024-03-22T13:02:09.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.068+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3581f4320 con 0x7fd3580cf5c0 2024-03-22T13:02:09.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.093+0000 7fd355600640 1 -- 172.21.15.6:0/952048801 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd35003e610 con 0x7fd3580cf5c0 2024-03-22T13:02:09.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.095+0000 7fd355600640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd33806bc50 0x7fd33806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:09.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.095+0000 7fd35d600640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd33806bc50 0x7fd33806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:09.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.096+0000 7fd35d600640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd33806bc50 0x7fd33806e110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009b40 tx=0x7fd34c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:09.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.096+0000 7fd355600640 1 -- 172.21.15.6:0/952048801 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd35007a410 con 0x7fd3580cf5c0 2024-03-22T13:02:09.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.384+0000 7fd355600640 1 -- 172.21.15.6:0/952048801 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd35007c370 con 0x7fd3580cf5c0 2024-03-22T13:02:09.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.643+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd35806ddd0 con 0x7fd3580cf5c0 2024-03-22T13:02:09.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.666+0000 7fd355600640 1 -- 172.21.15.6:0/952048801 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fd350080410 con 0x7fd3580cf5c0 2024-03-22T13:02:09.666 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:09.667 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:09.667 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:09.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.670+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd33806bc50 msgr2=0x7fd33806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.670+0000 7fd360000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd33806bc50 0x7fd33806e110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009b40 tx=0x7fd34c009190 comp rx=0 tx=0).stop 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.670+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cf5c0 msgr2=0x7fd35815e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.670+0000 7fd360000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cf5c0 0x7fd35815e1e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd350009d90 tx=0x7fd350004a40 comp rx=0 tx=0).stop 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 shutdown_connections 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3580cf5c0 0x7fd35815e1e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3580ccd80 0x7fd3581f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd33806bc50 0x7fd33806e110 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 --2- 172.21.15.6:0/952048801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3580cac20 0x7fd35815be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 >> 172.21.15.6:0/952048801 conn(0x7fd3580b16c0 msgr2=0x7fd3580af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 shutdown_connections 2024-03-22T13:02:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:09.671+0000 7fd360000640 1 -- 172.21.15.6:0/952048801 wait complete. 2024-03-22T13:02:10.293 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:10.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:11.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:11.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.496+0000 7f3dbd800640 1 Processor -- start 2024-03-22T13:02:11.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.496+0000 7f3dbd800640 1 -- start start 2024-03-22T13:02:11.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.498+0000 7f3dbd800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db8154980 0x7f3db80ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:11.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.498+0000 7f3dbd800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db80cc380 con 0x7f3db80cd300 2024-03-22T13:02:11.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.498+0000 7f3dbd800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db80cc660 con 0x7f3db815b0c0 2024-03-22T13:02:11.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.498+0000 7f3dbd800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db815a520 con 0x7f3db8154980 2024-03-22T13:02:11.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.499+0000 7f3db6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db8154980 0x7f3db80ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:11.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.499+0000 7f3db7400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3db815b0c0 0x7f3db80ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:51754/0 (socket says 172.21.15.6:51754) 2024-03-22T13:02:11.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.499+0000 7f3db7400640 1 -- 172.21.15.6:0/2316961474 learned_addr learned my addr 172.21.15.6:0/2316961474 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:11.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.502+0000 7f3db6a00640 1 -- 172.21.15.6:0/2316961474 >> v1:172.21.15.92:6789/0 conn(0x7f3db815b0c0 legacy=0x7f3db80ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:11.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.502+0000 7f3db6a00640 1 -- 172.21.15.6:0/2316961474 >> v1:172.21.15.6:6789/0 conn(0x7f3db80cd300 legacy=0x7f3db815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:11.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.502+0000 7f3db6a00640 1 -- 172.21.15.6:0/2316961474 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db80cc940 con 0x7f3db8154980 2024-03-22T13:02:11.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.504+0000 7f3db6a00640 1 --2- 172.21.15.6:0/2316961474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db8154980 0x7f3db80ce300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3da80097d0 tx=0x7f3da802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537934faa726dc21 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:11.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.509+0000 7f3db6000640 1 -- 172.21.15.6:0/2316961474 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da8035020 con 0x7f3db8154980 2024-03-22T13:02:11.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.509+0000 7f3db6000640 1 -- 172.21.15.6:0/2316961474 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da8031890 con 0x7f3db8154980 2024-03-22T13:02:11.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.509+0000 7f3db6000640 1 -- 172.21.15.6:0/2316961474 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da8031c60 con 0x7f3db8154980 2024-03-22T13:02:11.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.509+0000 7f3dbd800640 1 -- 172.21.15.6:0/2316961474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db8154980 msgr2=0x7f3db80ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:11.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.509+0000 7f3dbd800640 1 --2- 172.21.15.6:0/2316961474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db8154980 0x7f3db80ce300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3da80097d0 tx=0x7f3da802b930 comp rx=0 tx=0).stop 2024-03-22T13:02:11.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.510+0000 7f3dbd800640 1 -- 172.21.15.6:0/2316961474 shutdown_connections 2024-03-22T13:02:11.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.510+0000 7f3dbd800640 1 --2- 172.21.15.6:0/2316961474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db8154980 0x7f3db80ce300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:11.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.510+0000 7f3dbd800640 1 -- 172.21.15.6:0/2316961474 >> 172.21.15.6:0/2316961474 conn(0x7f3db80b16c0 msgr2=0x7f3db80b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:11.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.510+0000 7f3dbd800640 1 -- 172.21.15.6:0/2316961474 shutdown_connections 2024-03-22T13:02:11.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.510+0000 7f3dbd800640 1 -- 172.21.15.6:0/2316961474 wait complete. 2024-03-22T13:02:11.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.511+0000 7f3dbd800640 1 Processor -- start 2024-03-22T13:02:11.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.512+0000 7f3dbd800640 1 -- start start 2024-03-22T13:02:11.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.512+0000 7f3dbd800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db80cd300 0x7f3db81f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:11.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3dbd800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db8154980 0x7f3db81f2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:11.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3dbd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3db815b0c0 0x7f3db81f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:11.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3dbd800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db80cf5b0 con 0x7f3db8154980 2024-03-22T13:02:11.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3dbd800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db80ccf00 con 0x7f3db815b0c0 2024-03-22T13:02:11.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3dbd800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db80cf730 con 0x7f3db80cd300 2024-03-22T13:02:11.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3db6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db80cd300 0x7f3db81f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:11.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3db7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3db815b0c0 0x7f3db81f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:11.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.513+0000 7f3dae000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db8154980 0x7f3db81f2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:11.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.514+0000 7f3db6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db80cd300 0x7f3db81f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44604/0 (socket says 172.21.15.6:44604) 2024-03-22T13:02:11.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.514+0000 7f3db6a00640 1 -- 172.21.15.6:0/1211263370 learned_addr learned my addr 172.21.15.6:0/1211263370 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:11.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.517+0000 7f3db7400640 1 -- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db80cd300 msgr2=0x7f3db81f1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:11.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.517+0000 7f3db7400640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db80cd300 0x7f3db81f1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:11.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.517+0000 7f3db7400640 1 -- 172.21.15.6:0/1211263370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db8154980 msgr2=0x7f3db81f2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:11.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.517+0000 7f3db7400640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db8154980 0x7f3db81f2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:11.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.517+0000 7f3db7400640 1 -- 172.21.15.6:0/1211263370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db80b3bc0 con 0x7f3db815b0c0 2024-03-22T13:02:11.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.520+0000 7f3db7400640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3db815b0c0 0x7f3db81f0220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3db000cad0 tx=0x7f3db000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.525+0000 7f3daf400640 1 -- 172.21.15.6:0/1211263370 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0017070 con 0x7f3db815b0c0 2024-03-22T13:02:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.525+0000 7f3daf400640 1 -- 172.21.15.6:0/1211263370 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db000ab70 con 0x7f3db815b0c0 2024-03-22T13:02:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.525+0000 7f3daf400640 1 -- 172.21.15.6:0/1211263370 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0012400 con 0x7f3db815b0c0 2024-03-22T13:02:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.525+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db80b0330 con 0x7f3db815b0c0 2024-03-22T13:02:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.525+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db815e510 con 0x7f3db815b0c0 2024-03-22T13:02:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.526+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db80756c0 con 0x7f3db815b0c0 2024-03-22T13:02:11.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.562+0000 7f3daf400640 1 -- 172.21.15.6:0/1211263370 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3db001a070 con 0x7f3db815b0c0 2024-03-22T13:02:11.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.565+0000 7f3daf400640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d9406bc50 0x7f3d9406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:11.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.565+0000 7f3db6a00640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d9406bc50 0x7f3d9406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:11.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.566+0000 7f3db6a00640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d9406bc50 0x7f3d9406e110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3da802b970 tx=0x7f3da8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:11.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.566+0000 7f3daf400640 1 -- 172.21.15.6:0/1211263370 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3db0057020 con 0x7f3db815b0c0 2024-03-22T13:02:11.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:11.870+0000 7f3daf400640 1 -- 172.21.15.6:0/1211263370 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3db00b72e0 con 0x7f3db815b0c0 2024-03-22T13:02:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.136+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3db806c6e0 con 0x7f3db815b0c0 2024-03-22T13:02:12.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.231+0000 7f3daf400640 1 -- 172.21.15.6:0/1211263370 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f3db80b3bc0 con 0x7f3db815b0c0 2024-03-22T13:02:12.232 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:12.232 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:12.232 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d9406bc50 msgr2=0x7f3d9406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d9406bc50 0x7f3d9406e110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3da802b970 tx=0x7f3da8008030 comp rx=0 tx=0).stop 2024-03-22T13:02:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3db815b0c0 msgr2=0x7f3db81f0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3db815b0c0 0x7f3db81f0220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3db000cad0 tx=0x7f3db000cf90 comp rx=0 tx=0).stop 2024-03-22T13:02:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 shutdown_connections 2024-03-22T13:02:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3d9406bc50 0x7f3d9406e110 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3db815b0c0 0x7f3db81f0220 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:12.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db8154980 0x7f3db81f2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:12.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 --2- 172.21.15.6:0/1211263370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3db80cd300 0x7f3db81f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:12.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 >> 172.21.15.6:0/1211263370 conn(0x7f3db80b16c0 msgr2=0x7f3db80ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:12.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 shutdown_connections 2024-03-22T13:02:12.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:12.234+0000 7f3dbd800640 1 -- 172.21.15.6:0/1211263370 wait complete. 2024-03-22T13:02:12.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:12.857 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:13.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:14.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.062+0000 7f625e800640 1 Processor -- start 2024-03-22T13:02:14.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.062+0000 7f625e800640 1 -- start start 2024-03-22T13:02:14.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.063+0000 7f625e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580ce8a0 0x7f625806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:14.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.063+0000 7f625e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f625815ae30 0x7f62580cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:14.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.063+0000 7f625e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625815b260 con 0x7f62580cc380 2024-03-22T13:02:14.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.063+0000 7f625e800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62580cee00 con 0x7f625815ae30 2024-03-22T13:02:14.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.063+0000 7f625e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62580cf0c0 con 0x7f62580ce8a0 2024-03-22T13:02:14.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.064+0000 7f625ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580ce8a0 0x7f625806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:14.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.064+0000 7f625de00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f625815ae30 0x7f62580cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:14.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.064+0000 7f625d400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62580cc380 0x7f62580cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52036/0 (socket says 172.21.15.6:52036) 2024-03-22T13:02:14.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.064+0000 7f625d400640 1 -- 172.21.15.6:0/2302845345 learned_addr learned my addr 172.21.15.6:0/2302845345 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:14.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.067+0000 7f625de00640 1 -- 172.21.15.6:0/2302845345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580ce8a0 msgr2=0x7f625806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:14.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.067+0000 7f625de00640 1 --2- 172.21.15.6:0/2302845345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580ce8a0 0x7f625806c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.067+0000 7f625de00640 1 -- 172.21.15.6:0/2302845345 >> v1:172.21.15.6:6789/0 conn(0x7f62580cc380 legacy=0x7f62580cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:14.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.067+0000 7f625de00640 1 -- 172.21.15.6:0/2302845345 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580cf3a0 con 0x7f625815ae30 2024-03-22T13:02:14.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.070+0000 7f625de00640 1 --2- 172.21.15.6:0/2302845345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 0x7f62580cc760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f62500097a0 tx=0x7f625002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e958f3a3f8671ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:14.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.074+0000 7f6257e00640 1 -- 172.21.15.6:0/2302845345 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250035020 con 0x7f625815ae30 2024-03-22T13:02:14.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.074+0000 7f6257e00640 1 -- 172.21.15.6:0/2302845345 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6250031810 con 0x7f625815ae30 2024-03-22T13:02:14.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.074+0000 7f6257e00640 1 -- 172.21.15.6:0/2302845345 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250031be0 con 0x7f625815ae30 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 -- 172.21.15.6:0/2302845345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 msgr2=0x7f62580cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 --2- 172.21.15.6:0/2302845345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 0x7f62580cc760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f62500097a0 tx=0x7f625002b8b0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 -- 172.21.15.6:0/2302845345 shutdown_connections 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 --2- 172.21.15.6:0/2302845345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 0x7f62580cc760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 --2- 172.21.15.6:0/2302845345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580ce8a0 0x7f625806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 -- 172.21.15.6:0/2302845345 >> 172.21.15.6:0/2302845345 conn(0x7f62580b1480 msgr2=0x7f62580b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 -- 172.21.15.6:0/2302845345 shutdown_connections 2024-03-22T13:02:14.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.075+0000 7f625e800640 1 -- 172.21.15.6:0/2302845345 wait complete. 2024-03-22T13:02:14.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.077+0000 7f625e800640 1 Processor -- start 2024-03-22T13:02:14.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.077+0000 7f625e800640 1 -- start start 2024-03-22T13:02:14.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580cc380 0x7f62581e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:14.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62580ce8a0 0x7f62581e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:14.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 0x7f62581e3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:14.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625815a910 con 0x7f62580ce8a0 2024-03-22T13:02:14.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625815a630 con 0x7f625815ae30 2024-03-22T13:02:14.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62581549d0 con 0x7f62580cc380 2024-03-22T13:02:14.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 0x7f62581e3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:14.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.078+0000 7f625d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580cc380 0x7f62581e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:14.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.079+0000 7f625ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62580ce8a0 0x7f62581e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:14.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.079+0000 7f625d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580cc380 0x7f62581e5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60876/0 (socket says 172.21.15.6:60876) 2024-03-22T13:02:14.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.079+0000 7f625d400640 1 -- 172.21.15.6:0/96254569 learned_addr learned my addr 172.21.15.6:0/96254569 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:14.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.083+0000 7f625ca00640 1 -- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580cc380 msgr2=0x7f62581e5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:14.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.083+0000 7f625ca00640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580cc380 0x7f62581e5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.083+0000 7f625ca00640 1 -- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 msgr2=0x7f62581e3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:14.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.083+0000 7f625ca00640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 0x7f62581e3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.083+0000 7f625ca00640 1 -- 172.21.15.6:0/96254569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62581436f0 con 0x7f62580ce8a0 2024-03-22T13:02:14.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.085+0000 7f625ca00640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62580ce8a0 0x7f62581e5ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f624400e850 tx=0x7f624400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:14.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.094+0000 7f6255600640 1 -- 172.21.15.6:0/96254569 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6244017070 con 0x7f62580ce8a0 2024-03-22T13:02:14.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.094+0000 7f6255600640 1 -- 172.21.15.6:0/96254569 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624400c940 con 0x7f62580ce8a0 2024-03-22T13:02:14.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.094+0000 7f6255600640 1 -- 172.21.15.6:0/96254569 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624400cd10 con 0x7f62580ce8a0 2024-03-22T13:02:14.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.094+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625815bef0 con 0x7f62580ce8a0 2024-03-22T13:02:14.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.094+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625815c4b0 con 0x7f62580ce8a0 2024-03-22T13:02:14.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.095+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62580684e0 con 0x7f62580ce8a0 2024-03-22T13:02:14.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.405+0000 7f6255600640 1 -- 172.21.15.6:0/96254569 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f624401a070 con 0x7f62580ce8a0 2024-03-22T13:02:14.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.406+0000 7f6255600640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f623806bc50 0x7f623806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:14.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.406+0000 7f625d400640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f623806bc50 0x7f623806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.407+0000 7f625d400640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f623806bc50 0x7f623806e110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f624c001e90 tx=0x7f624c001a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:14.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.430+0000 7f6255600640 1 -- 172.21.15.6:0/96254569 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6244057630 con 0x7f62580ce8a0 2024-03-22T13:02:14.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.432+0000 7f6255600640 1 -- 172.21.15.6:0/96254569 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f62580684e0 con 0x7f62580ce8a0 2024-03-22T13:02:14.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.725+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6258075650 con 0x7f62580ce8a0 2024-03-22T13:02:14.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.756+0000 7f6255600640 1 -- 172.21.15.6:0/96254569 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f62581436f0 con 0x7f62580ce8a0 2024-03-22T13:02:14.756 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:14.756 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:14.756 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.759+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f623806bc50 msgr2=0x7f623806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.759+0000 7f625e800640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f623806bc50 0x7f623806e110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f624c001e90 tx=0x7f624c001a20 comp rx=0 tx=0).stop 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.759+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62580ce8a0 msgr2=0x7f62581e5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.759+0000 7f625e800640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62580ce8a0 0x7f62581e5ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f624400e850 tx=0x7f624400ed10 comp rx=0 tx=0).stop 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 shutdown_connections 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f623806bc50 0x7f623806e110 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f625815ae30 0x7f62581e3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62580ce8a0 0x7f62581e5ff0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 --2- 172.21.15.6:0/96254569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62580cc380 0x7f62581e5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 >> 172.21.15.6:0/96254569 conn(0x7f62580b1480 msgr2=0x7f62581496b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 shutdown_connections 2024-03-22T13:02:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:14.760+0000 7f625e800640 1 -- 172.21.15.6:0/96254569 wait complete. 2024-03-22T13:02:15.391 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:15.391 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:16.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:16.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.600+0000 7fdc1cc00640 1 Processor -- start 2024-03-22T13:02:16.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.600+0000 7fdc1cc00640 1 -- start start 2024-03-22T13:02:16.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.601+0000 7fdc1cc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc1806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:16.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.601+0000 7fdc1cc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc1815ae30 0x7fdc180cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:16.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.601+0000 7fdc1cc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1815b260 con 0x7fdc180ce8a0 2024-03-22T13:02:16.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.601+0000 7fdc1cc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc180cee00 con 0x7fdc1815ae30 2024-03-22T13:02:16.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.601+0000 7fdc1cc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180cf0c0 con 0x7fdc180cc380 2024-03-22T13:02:16.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.602+0000 7fdc17e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc1815ae30 0x7fdc180cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:16.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.602+0000 7fdc17400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc1806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:16.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.603+0000 7fdc17400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc1806c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60884/0 (socket says 172.21.15.6:60884) 2024-03-22T13:02:16.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.603+0000 7fdc17400640 1 -- 172.21.15.6:0/43986605 learned_addr learned my addr 172.21.15.6:0/43986605 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.605+0000 7fdc17400640 1 -- 172.21.15.6:0/43986605 >> v2:172.21.15.92:3300/0 conn(0x7fdc1815ae30 msgr2=0x7fdc180cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.605+0000 7fdc17400640 1 --2- 172.21.15.6:0/43986605 >> v2:172.21.15.92:3300/0 conn(0x7fdc1815ae30 0x7fdc180cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.605+0000 7fdc17400640 1 -- 172.21.15.6:0/43986605 >> v1:172.21.15.92:6789/0 conn(0x7fdc180ce8a0 legacy=0x7fdc180cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:16.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.605+0000 7fdc17400640 1 -- 172.21.15.6:0/43986605 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180cf3a0 con 0x7fdc180cc380 2024-03-22T13:02:16.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.608+0000 7fdc17400640 1 --2- 172.21.15.6:0/43986605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc1806c230 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009e50 tx=0x7fdc0802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1f28bfd2dcc878 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:16.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc16000640 1 -- 172.21.15.6:0/43986605 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0802ca70 con 0x7fdc180cc380 2024-03-22T13:02:16.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc16000640 1 -- 172.21.15.6:0/43986605 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0802cc10 con 0x7fdc180cc380 2024-03-22T13:02:16.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc16000640 1 -- 172.21.15.6:0/43986605 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc080343f0 con 0x7fdc180cc380 2024-03-22T13:02:16.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc1cc00640 1 -- 172.21.15.6:0/43986605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 msgr2=0x7fdc1806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:16.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/43986605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc1806c230 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009e50 tx=0x7fdc0802bed0 comp rx=0 tx=0).stop 2024-03-22T13:02:16.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc1cc00640 1 -- 172.21.15.6:0/43986605 shutdown_connections 2024-03-22T13:02:16.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/43986605 >> v2:172.21.15.92:3300/0 conn(0x7fdc1815ae30 0x7fdc180cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:16.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/43986605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc1806c230 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:16.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.612+0000 7fdc1cc00640 1 -- 172.21.15.6:0/43986605 >> 172.21.15.6:0/43986605 conn(0x7fdc180b1480 msgr2=0x7fdc180b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:16.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.613+0000 7fdc1cc00640 1 -- 172.21.15.6:0/43986605 shutdown_connections 2024-03-22T13:02:16.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.613+0000 7fdc1cc00640 1 -- 172.21.15.6:0/43986605 wait complete. 2024-03-22T13:02:16.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.615+0000 7fdc1cc00640 1 Processor -- start 2024-03-22T13:02:16.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.615+0000 7fdc1cc00640 1 -- start start 2024-03-22T13:02:16.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.616+0000 7fdc1cc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc18147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:16.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.616+0000 7fdc1cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ce8a0 0x7fdc18149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:16.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.616+0000 7fdc1cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1815ae30 0x7fdc181460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:16.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.616+0000 7fdc1cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1815a910 con 0x7fdc180ce8a0 2024-03-22T13:02:16.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.617+0000 7fdc1cc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1815a630 con 0x7fdc1815ae30 2024-03-22T13:02:16.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.617+0000 7fdc1cc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc181549d0 con 0x7fdc180cc380 2024-03-22T13:02:16.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.617+0000 7fdc17400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc18147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:16.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.617+0000 7fdc17e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1815ae30 0x7fdc181460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:16.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.617+0000 7fdc16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ce8a0 0x7fdc18149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:16.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.618+0000 7fdc17400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc18147ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60892/0 (socket says 172.21.15.6:60892) 2024-03-22T13:02:16.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.618+0000 7fdc17400640 1 -- 172.21.15.6:0/188302833 learned_addr learned my addr 172.21.15.6:0/188302833 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.621+0000 7fdc17400640 1 -- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1815ae30 msgr2=0x7fdc181460a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.621+0000 7fdc17400640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1815ae30 0x7fdc181460a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.621+0000 7fdc17400640 1 -- 172.21.15.6:0/188302833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ce8a0 msgr2=0x7fdc18149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.621+0000 7fdc17400640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ce8a0 0x7fdc18149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.622+0000 7fdc17400640 1 -- 172.21.15.6:0/188302833 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18148840 con 0x7fdc180cc380 2024-03-22T13:02:16.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.625+0000 7fdc17400640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc18147ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009e50 tx=0x7fdc08004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:16.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.629+0000 7fdc0f400640 1 -- 172.21.15.6:0/188302833 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08043020 con 0x7fdc180cc380 2024-03-22T13:02:16.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.629+0000 7fdc0f400640 1 -- 172.21.15.6:0/188302833 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0803e450 con 0x7fdc180cc380 2024-03-22T13:02:16.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.629+0000 7fdc0f400640 1 -- 172.21.15.6:0/188302833 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08009510 con 0x7fdc180cc380 2024-03-22T13:02:16.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.629+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc18148ae0 con 0x7fdc180cc380 2024-03-22T13:02:16.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.629+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc181455c0 con 0x7fdc180cc380 2024-03-22T13:02:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.630+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc180c9fc0 con 0x7fdc180cc380 2024-03-22T13:02:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.896+0000 7fdc0f400640 1 -- 172.21.15.6:0/188302833 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc08046020 con 0x7fdc180cc380 2024-03-22T13:02:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.897+0000 7fdc0f400640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf406bc50 0x7fdbf406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.897+0000 7fdc0f400640 1 -- 172.21.15.6:0/188302833 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdc080ad8e0 con 0x7fdc180cc380 2024-03-22T13:02:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.898+0000 7fdc16a00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf406bc50 0x7fdbf406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.899+0000 7fdc16a00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf406bc50 0x7fdbf406e110 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdc18151ee0 tx=0x7fdc00007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:16.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:16.907+0000 7fdc0f400640 1 -- 172.21.15.6:0/188302833 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc180c9fc0 con 0x7fdc180cc380 2024-03-22T13:02:17.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.161+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc18075650 con 0x7fdc180cc380 2024-03-22T13:02:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.174+0000 7fdc0f400640 1 -- 172.21.15.6:0/188302833 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fdc0804d030 con 0x7fdc180cc380 2024-03-22T13:02:17.174 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:17.174 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf406bc50 msgr2=0x7fdbf406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf406bc50 0x7fdbf406e110 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdc18151ee0 tx=0x7fdc00007e30 comp rx=0 tx=0).stop 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 msgr2=0x7fdc18147ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc18147ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009e50 tx=0x7fdc08004a40 comp rx=0 tx=0).stop 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 shutdown_connections 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf406bc50 0x7fdbf406e110 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc1815ae30 0x7fdc181460a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ce8a0 0x7fdc18149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 --2- 172.21.15.6:0/188302833 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cc380 0x7fdc18147ea0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 >> 172.21.15.6:0/188302833 conn(0x7fdc180b1480 msgr2=0x7fdc180cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 shutdown_connections 2024-03-22T13:02:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:17.177+0000 7fdc1cc00640 1 -- 172.21.15.6:0/188302833 wait complete. 2024-03-22T13:02:17.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:17.799 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:18.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:19.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.002+0000 7fb142600640 1 Processor -- start 2024-03-22T13:02:19.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.002+0000 7fb142600640 1 -- start start 2024-03-22T13:02:19.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.003+0000 7fb142600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb13c154240 0x7fb13c15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:19.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.003+0000 7fb142600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb13c0c7ee0 0x7fb13c0c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:19.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.003+0000 7fb142600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c154620 con 0x7fb13c0c68c0 2024-03-22T13:02:19.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.003+0000 7fb142600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c15e130 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.003+0000 7fb142600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c0c6e00 con 0x7fb13c154240 2024-03-22T13:02:19.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.004+0000 7fb13b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb13c154240 0x7fb13c15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:19.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.004+0000 7fb13aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb13c0c7ee0 0x7fb13c0c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:19.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.005+0000 7fb13b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb13c154240 0x7fb13c15ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:43800/0 (socket says 172.21.15.6:43800) 2024-03-22T13:02:19.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.005+0000 7fb13b400640 1 -- 172.21.15.6:0/2730708621 learned_addr learned my addr 172.21.15.6:0/2730708621 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:19.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.009+0000 7fb13b400640 1 -- 172.21.15.6:0/2730708621 >> v2:172.21.15.6:3300/0 conn(0x7fb13c0c7ee0 msgr2=0x7fb13c0c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:19.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.009+0000 7fb13b400640 1 --2- 172.21.15.6:0/2730708621 >> v2:172.21.15.6:3300/0 conn(0x7fb13c0c7ee0 0x7fb13c0c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.009+0000 7fb13b400640 1 -- 172.21.15.6:0/2730708621 >> v1:172.21.15.6:6789/0 conn(0x7fb13c0c68c0 legacy=0x7fb13c06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.009+0000 7fb13b400640 1 -- 172.21.15.6:0/2730708621 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0c70e0 con 0x7fb13c154240 2024-03-22T13:02:19.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.012+0000 7fb13b400640 1 --2- 172.21.15.6:0/2730708621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c154240 0x7fb13c15ac90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb13c071390 tx=0x7fb12c02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15a06791d55dfe0a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:19.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.017+0000 7fb13a000640 1 -- 172.21.15.6:0/2730708621 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c02f850 con 0x7fb13c154240 2024-03-22T13:02:19.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.017+0000 7fb13a000640 1 -- 172.21.15.6:0/2730708621 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c02f9f0 con 0x7fb13c154240 2024-03-22T13:02:19.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.017+0000 7fb13a000640 1 -- 172.21.15.6:0/2730708621 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c02fdc0 con 0x7fb13c154240 2024-03-22T13:02:19.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 -- 172.21.15.6:0/2730708621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c154240 msgr2=0x7fb13c15ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:19.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 --2- 172.21.15.6:0/2730708621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c154240 0x7fb13c15ac90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb13c071390 tx=0x7fb12c02ba30 comp rx=0 tx=0).stop 2024-03-22T13:02:19.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 -- 172.21.15.6:0/2730708621 shutdown_connections 2024-03-22T13:02:19.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 --2- 172.21.15.6:0/2730708621 >> v2:172.21.15.6:3300/0 conn(0x7fb13c0c7ee0 0x7fb13c0c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 --2- 172.21.15.6:0/2730708621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c154240 0x7fb13c15ac90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 -- 172.21.15.6:0/2730708621 >> 172.21.15.6:0/2730708621 conn(0x7fb13c0cb930 msgr2=0x7fb13c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:19.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 -- 172.21.15.6:0/2730708621 shutdown_connections 2024-03-22T13:02:19.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.018+0000 7fb142600640 1 -- 172.21.15.6:0/2730708621 wait complete. 2024-03-22T13:02:19.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.020+0000 7fb142600640 1 Processor -- start 2024-03-22T13:02:19.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.020+0000 7fb142600640 1 -- start start 2024-03-22T13:02:19.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.021+0000 7fb142600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb13c0c68c0 0x7fb13c15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:19.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.021+0000 7fb142600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c0c7ee0 0x7fb13c1e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:19.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.021+0000 7fb142600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb13c154240 0x7fb13c1e3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:19.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.021+0000 7fb142600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15ddc0 con 0x7fb13c0c68c0 2024-03-22T13:02:19.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.021+0000 7fb142600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15dae0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.021+0000 7fb142600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15e5c0 con 0x7fb13c154240 2024-03-22T13:02:19.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.022+0000 7fb13aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c0c7ee0 0x7fb13c1e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:19.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.022+0000 7fb13b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb13c0c68c0 0x7fb13c15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:19.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.022+0000 7fb13be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb13c154240 0x7fb13c1e3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:19.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.022+0000 7fb13aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c0c7ee0 0x7fb13c1e5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:43802/0 (socket says 172.21.15.6:43802) 2024-03-22T13:02:19.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.022+0000 7fb13aa00640 1 -- 172.21.15.6:0/1842171483 learned_addr learned my addr 172.21.15.6:0/1842171483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:19.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.026+0000 7fb13aa00640 1 -- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb13c154240 msgr2=0x7fb13c1e3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:19.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.026+0000 7fb13aa00640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb13c154240 0x7fb13c1e3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.026+0000 7fb13aa00640 1 -- 172.21.15.6:0/1842171483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb13c0c68c0 msgr2=0x7fb13c15cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:19.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.026+0000 7fb13aa00640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb13c0c68c0 0x7fb13c15cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.026+0000 7fb13aa00640 1 -- 172.21.15.6:0/1842171483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c1e6340 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.029+0000 7fb13aa00640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c0c7ee0 0x7fb13c1e5d20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb12800e9e0 tx=0x7fb12800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:19.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.034+0000 7fb133400640 1 -- 172.21.15.6:0/1842171483 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128017070 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.034+0000 7fb133400640 1 -- 172.21.15.6:0/1842171483 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12800c9d0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.034+0000 7fb133400640 1 -- 172.21.15.6:0/1842171483 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12800cda0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.034+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c0caae0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.034+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c0cb0a0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.035+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c0756c0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.328+0000 7fb133400640 1 -- 172.21.15.6:0/1842171483 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb12801a070 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.329+0000 7fb133400640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb11c06bc20 0x7fb11c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:19.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.330+0000 7fb13b400640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb11c06bc20 0x7fb11c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:19.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.331+0000 7fb13b400640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb11c06bc20 0x7fb11c06e0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb12c036000 tx=0x7fb12c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:19.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.331+0000 7fb133400640 1 -- 172.21.15.6:0/1842171483 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb1280576c0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.350+0000 7fb133400640 1 -- 172.21.15.6:0/1842171483 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb128059620 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.634+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb13c074d50 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.648+0000 7fb133400640 1 -- 172.21.15.6:0/1842171483 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fb12805d6c0 con 0x7fb13c0c7ee0 2024-03-22T13:02:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:19.648 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:19.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb11c06bc20 msgr2=0x7fb11c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb11c06bc20 0x7fb11c06e0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb12c036000 tx=0x7fb12c008030 comp rx=0 tx=0).stop 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c0c7ee0 msgr2=0x7fb13c1e5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c0c7ee0 0x7fb13c1e5d20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb12800e9e0 tx=0x7fb12800eea0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 shutdown_connections 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb13c154240 0x7fb13c1e3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb11c06bc20 0x7fb11c06e0e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb13c0c7ee0 0x7fb13c1e5d20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 --2- 172.21.15.6:0/1842171483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb13c0c68c0 0x7fb13c15cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.650+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 >> 172.21.15.6:0/1842171483 conn(0x7fb13c0cb930 msgr2=0x7fb13c14c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.651+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 shutdown_connections 2024-03-22T13:02:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:19.651+0000 7fb142600640 1 -- 172.21.15.6:0/1842171483 wait complete. 2024-03-22T13:02:20.270 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:20.270 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:21.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:21.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.469+0000 7f6765600640 1 Processor -- start 2024-03-22T13:02:21.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.469+0000 7f6765600640 1 -- start start 2024-03-22T13:02:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.471+0000 7f6765600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f676006fea0 0x7f676006b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.471+0000 7f6765600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67600c3d30 con 0x7f676006b080 2024-03-22T13:02:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.471+0000 7f6765600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67600c3a50 con 0x7f67600c2ea0 2024-03-22T13:02:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.471+0000 7f6765600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676015a450 con 0x7f676006fea0 2024-03-22T13:02:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.472+0000 7f675f400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f67600c2ea0 0x7f6760164c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37508/0 (socket says 172.21.15.6:37508) 2024-03-22T13:02:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.472+0000 7f675f400640 1 -- 172.21.15.6:0/3000906206 learned_addr learned my addr 172.21.15.6:0/3000906206 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.472+0000 7f675e000640 1 --2- 172.21.15.6:0/3000906206 >> v2:172.21.15.6:3300/0 conn(0x7f676006fea0 0x7f676006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:21.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.475+0000 7f675d600640 1 -- 172.21.15.6:0/3000906206 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55836440 0 0) 0x7f67600c3a50 con 0x7f67600c2ea0 2024-03-22T13:02:21.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.475+0000 7f675d600640 1 -- 172.21.15.6:0/3000906206 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6744003520 con 0x7f67600c2ea0 2024-03-22T13:02:21.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.476+0000 7f675e000640 1 -- 172.21.15.6:0/3000906206 >> v1:172.21.15.92:6789/0 conn(0x7f67600c2ea0 legacy=0x7f6760164c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:21.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.476+0000 7f675e000640 1 -- 172.21.15.6:0/3000906206 >> v1:172.21.15.6:6789/0 conn(0x7f676006b080 legacy=0x7f676015d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:21.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.476+0000 7f675e000640 1 -- 172.21.15.6:0/3000906206 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676015a730 con 0x7f676006fea0 2024-03-22T13:02:21.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.482+0000 7f675e000640 1 --2- 172.21.15.6:0/3000906206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676006b460 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f674c0097a0 tx=0x7f674c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7b38dfe7badbc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:21.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.487+0000 7f675d600640 1 -- 172.21.15.6:0/3000906206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c02f810 con 0x7f676006fea0 2024-03-22T13:02:21.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.487+0000 7f675d600640 1 -- 172.21.15.6:0/3000906206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c02f9b0 con 0x7f676006fea0 2024-03-22T13:02:21.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.487+0000 7f675d600640 1 -- 172.21.15.6:0/3000906206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c02fd80 con 0x7f676006fea0 2024-03-22T13:02:21.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.488+0000 7f6765600640 1 -- 172.21.15.6:0/3000906206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 msgr2=0x7f676006b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:21.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.488+0000 7f6765600640 1 --2- 172.21.15.6:0/3000906206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676006b460 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f674c0097a0 tx=0x7f674c02b9f0 comp rx=0 tx=0).stop 2024-03-22T13:02:21.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.488+0000 7f6765600640 1 -- 172.21.15.6:0/3000906206 shutdown_connections 2024-03-22T13:02:21.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.488+0000 7f6765600640 1 --2- 172.21.15.6:0/3000906206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676006b460 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:21.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.488+0000 7f6765600640 1 -- 172.21.15.6:0/3000906206 >> 172.21.15.6:0/3000906206 conn(0x7f67600ccb50 msgr2=0x7f67600ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:21.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.488+0000 7f6765600640 1 -- 172.21.15.6:0/3000906206 shutdown_connections 2024-03-22T13:02:21.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.488+0000 7f6765600640 1 -- 172.21.15.6:0/3000906206 wait complete. 2024-03-22T13:02:21.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.491+0000 7f6765600640 1 Processor -- start 2024-03-22T13:02:21.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.491+0000 7f6765600640 1 -- start start 2024-03-22T13:02:21.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.492+0000 7f6765600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f676006b080 0x7f67601e4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:21.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.493+0000 7f6765600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676014fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:21.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.493+0000 7f6765600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67600c2ea0 0x7f6760150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:21.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.493+0000 7f6765600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676015e000 con 0x7f676006fea0 2024-03-22T13:02:21.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.493+0000 7f6765600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676015acf0 con 0x7f676006b080 2024-03-22T13:02:21.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.493+0000 7f6765600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676015b210 con 0x7f67600c2ea0 2024-03-22T13:02:21.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.493+0000 7f675e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676014fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:21.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.493+0000 7f675ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f676006b080 0x7f67601e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:21.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.494+0000 7f675f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67600c2ea0 0x7f6760150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:21.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.494+0000 7f675e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676014fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55614/0 (socket says 172.21.15.6:55614) 2024-03-22T13:02:21.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.494+0000 7f675e000640 1 -- 172.21.15.6:0/799867876 learned_addr learned my addr 172.21.15.6:0/799867876 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.497+0000 7f675f400640 1 -- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f676006b080 msgr2=0x7f67601e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.497+0000 7f675f400640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f676006b080 0x7f67601e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.497+0000 7f675f400640 1 -- 172.21.15.6:0/799867876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 msgr2=0x7f676014fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.497+0000 7f675f400640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676014fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.497+0000 7f675f400640 1 -- 172.21.15.6:0/799867876 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676014de40 con 0x7f67600c2ea0 2024-03-22T13:02:21.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.500+0000 7f675f400640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67600c2ea0 0x7f6760150060 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f675800cab0 tx=0x7f675800cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:21.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.504+0000 7f6756a00640 1 -- 172.21.15.6:0/799867876 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758017070 con 0x7f67600c2ea0 2024-03-22T13:02:21.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.504+0000 7f6756a00640 1 -- 172.21.15.6:0/799867876 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675800ac30 con 0x7f67600c2ea0 2024-03-22T13:02:21.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.504+0000 7f6756a00640 1 -- 172.21.15.6:0/799867876 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758012600 con 0x7f67600c2ea0 2024-03-22T13:02:21.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.504+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676015d760 con 0x7f67600c2ea0 2024-03-22T13:02:21.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.504+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67601f39c0 con 0x7f67600c2ea0 2024-03-22T13:02:21.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.506+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67600756c0 con 0x7f67600c2ea0 2024-03-22T13:02:21.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.524+0000 7f6756a00640 1 -- 172.21.15.6:0/799867876 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f675801a070 con 0x7f67600c2ea0 2024-03-22T13:02:21.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.525+0000 7f6756a00640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f674006bc20 0x7f674006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:21.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.525+0000 7f675ea00640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f674006bc20 0x7f674006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:21.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.526+0000 7f675ea00640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f674006bc20 0x7f674006e0e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f67500016e0 tx=0x7f6750009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:21.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.528+0000 7f6756a00640 1 -- 172.21.15.6:0/799867876 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6758057150 con 0x7f67600c2ea0 2024-03-22T13:02:21.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:21.781+0000 7f6756a00640 1 -- 172.21.15.6:0/799867876 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f675800ae20 con 0x7f67600c2ea0 2024-03-22T13:02:22.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.039+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f676015a120 con 0x7f67600c2ea0 2024-03-22T13:02:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.060+0000 7f6756a00640 1 -- 172.21.15.6:0/799867876 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f67580590b0 con 0x7f67600c2ea0 2024-03-22T13:02:22.061 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:22.061 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:22.061 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:22.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.063+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f674006bc20 msgr2=0x7f674006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.063+0000 7f6765600640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f674006bc20 0x7f674006e0e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f67500016e0 tx=0x7f6750009040 comp rx=0 tx=0).stop 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.063+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67600c2ea0 msgr2=0x7f6760150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.063+0000 7f6765600640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67600c2ea0 0x7f6760150060 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f675800cab0 tx=0x7f675800cf70 comp rx=0 tx=0).stop 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 shutdown_connections 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67600c2ea0 0x7f6760150060 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676006fea0 0x7f676014fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f674006bc20 0x7f674006e0e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 --2- 172.21.15.6:0/799867876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f676006b080 0x7f67601e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 >> 172.21.15.6:0/799867876 conn(0x7f67600ccb50 msgr2=0x7f67600c9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 shutdown_connections 2024-03-22T13:02:22.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:22.064+0000 7f6765600640 1 -- 172.21.15.6:0/799867876 wait complete. 2024-03-22T13:02:22.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:22.686 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:23.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.885+0000 7f5011600640 1 Processor -- start 2024-03-22T13:02:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.885+0000 7f5011600640 1 -- start start 2024-03-22T13:02:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.886+0000 7f5011600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f500c0c68c0 0x7f500c15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.886+0000 7f5011600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0c4740 con 0x7f500c154240 2024-03-22T13:02:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.886+0000 7f5011600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c154620 con 0x7f500c0c7ee0 2024-03-22T13:02:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.886+0000 7f5011600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c15e130 con 0x7f500c0c68c0 2024-03-22T13:02:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.887+0000 7f500aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f500c154240 0x7f500c0c4030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38794/0 (socket says 172.21.15.6:38794) 2024-03-22T13:02:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.887+0000 7f500aa00640 1 -- 172.21.15.6:0/2346917061 learned_addr learned my addr 172.21.15.6:0/2346917061 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.887+0000 7f500b400640 1 --2- 172.21.15.6:0/2346917061 >> v2:172.21.15.92:3300/0 conn(0x7f500c0c68c0 0x7f500c15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:23.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.890+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812122373 0 0) 0x7f500c0c4740 con 0x7f500c154240 2024-03-22T13:02:23.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.890+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff0003520 con 0x7f500c154240 2024-03-22T13:02:23.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1071596333 0 0) 0x7f4ff0003520 con 0x7f500c154240 2024-03-22T13:02:23.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 >> v2:172.21.15.92:3300/0 conn(0x7f500c0c68c0 msgr2=0x7f500c15ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:23.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 --2- 172.21.15.6:0/2346917061 >> v2:172.21.15.92:3300/0 conn(0x7f500c0c68c0 0x7f500c15ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:23.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 >> v1:172.21.15.92:6789/0 conn(0x7f500c0c7ee0 legacy=0x7f500c06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:23.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0c6e00 con 0x7f500c154240 2024-03-22T13:02:23.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f4ffc001670 con 0x7f500c154240 2024-03-22T13:02:23.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 >> v1:172.21.15.6:6789/0 conn(0x7f500c154240 legacy=0x7f500c0c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:23.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ff0003a00 0x7f4ff0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:23.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ff0004800 0x7f4ff0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:23.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff00051c0 0x7f4ff000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:23.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15e130 con 0x7f4ff00051c0 2024-03-22T13:02:23.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff000c0a0 con 0x7f4ff0004800 2024-03-22T13:02:23.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.894+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff000c2a0 con 0x7f4ff0003a00 2024-03-22T13:02:23.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.895+0000 7f500aa00640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ff0003a00 0x7f4ff0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:23.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.898+0000 7f500b400640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff00051c0 0x7f4ff000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:23.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.898+0000 7f500aa00640 1 -- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ff0004800 msgr2=0x7f4ff0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:23.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.898+0000 7f500aa00640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ff0004800 0x7f4ff0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:23.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.898+0000 7f500aa00640 1 -- 172.21.15.6:0/2346917061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff00051c0 msgr2=0x7f4ff000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:23.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.898+0000 7f500aa00640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff00051c0 0x7f4ff000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:23.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.898+0000 7f500aa00640 1 -- 172.21.15.6:0/2346917061 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff0003520 con 0x7f4ff0003a00 2024-03-22T13:02:23.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.898+0000 7f500b400640 1 -- 172.21.15.6:0/2346917061 reap_dead start 2024-03-22T13:02:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.901+0000 7f500aa00640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ff0003a00 0x7f4ff0003e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00c110 tx=0x7f4ff0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2bc97436bee6b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:23.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.905+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc007c40 con 0x7f4ff0003a00 2024-03-22T13:02:23.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.905+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ffc030070 con 0x7f4ff0003a00 2024-03-22T13:02:23.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.905+0000 7f5009600640 1 -- 172.21.15.6:0/2346917061 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc005db0 con 0x7f4ff0003a00 2024-03-22T13:02:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.906+0000 7f5011600640 1 -- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ff0003a00 msgr2=0x7f4ff0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.906+0000 7f5011600640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ff0003a00 0x7f4ff0003e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00c110 tx=0x7f4ff0005830 comp rx=0 tx=0).stop 2024-03-22T13:02:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.906+0000 7f5011600640 1 -- 172.21.15.6:0/2346917061 shutdown_connections 2024-03-22T13:02:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.906+0000 7f5011600640 1 --2- 172.21.15.6:0/2346917061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ff0003a00 0x7f4ff0003e20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.906+0000 7f5011600640 1 -- 172.21.15.6:0/2346917061 >> 172.21.15.6:0/2346917061 conn(0x7f500c0cb930 msgr2=0x7f500c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.906+0000 7f5011600640 1 -- 172.21.15.6:0/2346917061 shutdown_connections 2024-03-22T13:02:23.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.906+0000 7f5011600640 1 -- 172.21.15.6:0/2346917061 wait complete. 2024-03-22T13:02:23.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.908+0000 7f5011600640 1 Processor -- start 2024-03-22T13:02:23.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.908+0000 7f5011600640 1 -- start start 2024-03-22T13:02:23.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.909+0000 7f5011600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff0003a00 0x7f500c1f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:23.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.909+0000 7f5011600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f500c151e50 0x7f500c0cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:23.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.909+0000 7f5011600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f500c1583a0 0x7f500c158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:23.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.909+0000 7f5011600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15dae0 con 0x7f4ff0003a00 2024-03-22T13:02:23.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.909+0000 7f5011600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15d800 con 0x7f500c1583a0 2024-03-22T13:02:23.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.909+0000 7f5011600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15ddc0 con 0x7f500c151e50 2024-03-22T13:02:23.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.910+0000 7f500b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f500c1583a0 0x7f500c158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:23.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.910+0000 7f500a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f500c151e50 0x7f500c0cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:23.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.910+0000 7f500aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff0003a00 0x7f500c1f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:23.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.910+0000 7f500a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f500c151e50 0x7f500c0cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40742/0 (socket says 172.21.15.6:40742) 2024-03-22T13:02:23.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.910+0000 7f500a000640 1 -- 172.21.15.6:0/1091552023 learned_addr learned my addr 172.21.15.6:0/1091552023 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:23.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.914+0000 7f500a000640 1 -- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f500c1583a0 msgr2=0x7f500c158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:23.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.914+0000 7f500a000640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f500c1583a0 0x7f500c158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:23.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.914+0000 7f500a000640 1 -- 172.21.15.6:0/1091552023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff0003a00 msgr2=0x7f500c1f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:23.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.914+0000 7f500a000640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff0003a00 0x7f500c1f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:23.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.914+0000 7f500a000640 1 -- 172.21.15.6:0/1091552023 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c158d60 con 0x7f500c151e50 2024-03-22T13:02:23.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.914+0000 7f500aa00640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff0003a00 0x7f500c1f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:02:23.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.917+0000 7f500a000640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f500c151e50 0x7f500c0cab70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c500 tx=0x7f4ff800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:23.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.922+0000 7f5002a00640 1 -- 172.21.15.6:0/1091552023 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80052d0 con 0x7f500c151e50 2024-03-22T13:02:23.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.922+0000 7f5002a00640 1 -- 172.21.15.6:0/1091552023 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff8005470 con 0x7f500c151e50 2024-03-22T13:02:23.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.922+0000 7f5002a00640 1 -- 172.21.15.6:0/1091552023 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8012850 con 0x7f500c151e50 2024-03-22T13:02:23.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.922+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c1f0f10 con 0x7f500c151e50 2024-03-22T13:02:23.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.922+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c1507d0 con 0x7f500c151e50 2024-03-22T13:02:23.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:23.923+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c0756c0 con 0x7f500c151e50 2024-03-22T13:02:24.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.212+0000 7f5002a00640 1 -- 172.21.15.6:0/1091552023 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4ff80129f0 con 0x7f500c151e50 2024-03-22T13:02:24.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.215+0000 7f5002a00640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4fec06bc00 0x7f4fec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:24.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.215+0000 7f500aa00640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4fec06bc00 0x7f4fec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.216+0000 7f5002a00640 1 -- 172.21.15.6:0/1091552023 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4ff80576e0 con 0x7f500c151e50 2024-03-22T13:02:24.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.216+0000 7f500aa00640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4fec06bc00 0x7f4fec06e0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc034000 tx=0x7f4ffc009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.234+0000 7f5002a00640 1 -- 172.21.15.6:0/1091552023 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f500c158d60 con 0x7f500c151e50 2024-03-22T13:02:24.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.486+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500c0c7ab0 con 0x7f500c151e50 2024-03-22T13:02:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.502+0000 7f5002a00640 1 -- 172.21.15.6:0/1091552023 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f4ff8059640 con 0x7f500c151e50 2024-03-22T13:02:24.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:24.502 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:24.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4fec06bc00 msgr2=0x7f4fec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:24.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4fec06bc00 0x7f4fec06e0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc034000 tx=0x7f4ffc009030 comp rx=0 tx=0).stop 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f500c151e50 msgr2=0x7f500c0cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f500c151e50 0x7f500c0cab70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c500 tx=0x7f4ff800c9c0 comp rx=0 tx=0).stop 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 shutdown_connections 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4fec06bc00 0x7f4fec06e0c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f500c1583a0 0x7f500c158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f500c151e50 0x7f500c0cab70 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800c500 tx=0x7f4ff800c9c0 comp rx=0 tx=0).stop 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 --2- 172.21.15.6:0/1091552023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ff0003a00 0x7f500c1f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.504+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 >> 172.21.15.6:0/1091552023 conn(0x7f500c0cb930 msgr2=0x7f500c0c4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.505+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 shutdown_connections 2024-03-22T13:02:24.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:24.505+0000 7f5011600640 1 -- 172.21.15.6:0/1091552023 wait complete. 2024-03-22T13:02:25.126 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:25.126 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:26.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.322+0000 7f3211400640 1 Processor -- start 2024-03-22T13:02:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.322+0000 7f3211400640 1 -- start start 2024-03-22T13:02:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.323+0000 7f3211400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f320c15b7a0 0x7f320c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.323+0000 7f3211400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c06c440 con 0x7f320c0cca20 2024-03-22T13:02:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.323+0000 7f3211400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c06c720 con 0x7f320c0cc640 2024-03-22T13:02:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.323+0000 7f3211400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c06d330 con 0x7f320c15b7a0 2024-03-22T13:02:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.324+0000 7f320be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f320c0cc640 0x7f320c06d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:49184/0 (socket says 172.21.15.6:49184) 2024-03-22T13:02:26.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.324+0000 7f320be00640 1 -- 172.21.15.6:0/2455268628 learned_addr learned my addr 172.21.15.6:0/2455268628 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.373+0000 7f3210a00640 1 --2- 172.21.15.6:0/2455268628 >> v2:172.21.15.6:3300/0 conn(0x7f320c15b7a0 0x7f320c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.381+0000 7f320aa00640 1 -- 172.21.15.6:0/2455268628 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999244830 0 0) 0x7f320c06c440 con 0x7f320c0cca20 2024-03-22T13:02:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.382+0000 7f320aa00640 1 -- 172.21.15.6:0/2455268628 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f0003520 con 0x7f320c0cca20 2024-03-22T13:02:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.385+0000 7f3210a00640 1 -- 172.21.15.6:0/2455268628 >> v1:172.21.15.92:6789/0 conn(0x7f320c0cc640 legacy=0x7f320c06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.385+0000 7f3210a00640 1 -- 172.21.15.6:0/2455268628 >> v1:172.21.15.6:6789/0 conn(0x7f320c0cca20 legacy=0x7f320c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.385+0000 7f3210a00640 1 -- 172.21.15.6:0/2455268628 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c06ca00 con 0x7f320c15b7a0 2024-03-22T13:02:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.397+0000 7f3210a00640 1 --2- 172.21.15.6:0/2455268628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c15b7a0 0x7f320c15e260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f320c06d4b0 tx=0x7f320402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32c2faf673062cd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.407+0000 7f320aa00640 1 -- 172.21.15.6:0/2455268628 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204039070 con 0x7f320c15b7a0 2024-03-22T13:02:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.407+0000 7f320aa00640 1 -- 172.21.15.6:0/2455268628 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320402cd80 con 0x7f320c15b7a0 2024-03-22T13:02:26.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.407+0000 7f320aa00640 1 -- 172.21.15.6:0/2455268628 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204034660 con 0x7f320c15b7a0 2024-03-22T13:02:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.408+0000 7f3211400640 1 -- 172.21.15.6:0/2455268628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c15b7a0 msgr2=0x7f320c15e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.408+0000 7f3211400640 1 --2- 172.21.15.6:0/2455268628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c15b7a0 0x7f320c15e260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f320c06d4b0 tx=0x7f320402bfd0 comp rx=0 tx=0).stop 2024-03-22T13:02:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.409+0000 7f3211400640 1 -- 172.21.15.6:0/2455268628 shutdown_connections 2024-03-22T13:02:26.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.409+0000 7f3211400640 1 --2- 172.21.15.6:0/2455268628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c15b7a0 0x7f320c15e260 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.409+0000 7f3211400640 1 -- 172.21.15.6:0/2455268628 >> 172.21.15.6:0/2455268628 conn(0x7f320c0cf9c0 msgr2=0x7f320c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.409+0000 7f3211400640 1 -- 172.21.15.6:0/2455268628 shutdown_connections 2024-03-22T13:02:26.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.409+0000 7f3211400640 1 -- 172.21.15.6:0/2455268628 wait complete. 2024-03-22T13:02:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.412+0000 7f3211400640 1 Processor -- start 2024-03-22T13:02:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.412+0000 7f3211400640 1 -- start start 2024-03-22T13:02:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.413+0000 7f3211400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f320c0cc640 0x7f320c1f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.413+0000 7f3211400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0cca20 0x7f320c157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.413+0000 7f3211400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f320c15b7a0 0x7f320c150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.413+0000 7f3211400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0caa50 con 0x7f320c0cca20 2024-03-22T13:02:26.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.413+0000 7f3211400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c06cfc0 con 0x7f320c0cc640 2024-03-22T13:02:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.413+0000 7f3211400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c15b000 con 0x7f320c15b7a0 2024-03-22T13:02:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.414+0000 7f320b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0cca20 0x7f320c157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.415+0000 7f320b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0cca20 0x7f320c157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47440/0 (socket says 172.21.15.6:47440) 2024-03-22T13:02:26.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.415+0000 7f320b400640 1 -- 172.21.15.6:0/909757049 learned_addr learned my addr 172.21.15.6:0/909757049 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.420+0000 7f320b400640 1 -- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f320c15b7a0 msgr2=0x7f320c150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.420+0000 7f320b400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f320c15b7a0 0x7f320c150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.420+0000 7f320b400640 1 -- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f320c0cc640 msgr2=0x7f320c1f13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:26.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.420+0000 7f320b400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f320c0cc640 0x7f320c1f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:26.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.421+0000 7f320b400640 1 -- 172.21.15.6:0/909757049 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c1f40a0 con 0x7f320c0cca20 2024-03-22T13:02:26.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.425+0000 7f320b400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0cca20 0x7f320c157a70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c430 tx=0x7f31f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:26.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.433+0000 7f3203e00640 1 -- 172.21.15.6:0/909757049 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8017070 con 0x7f320c0cca20 2024-03-22T13:02:26.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.433+0000 7f3203e00640 1 -- 172.21.15.6:0/909757049 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f80052d0 con 0x7f320c0cca20 2024-03-22T13:02:26.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.433+0000 7f3203e00640 1 -- 172.21.15.6:0/909757049 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8003830 con 0x7f320c0cca20 2024-03-22T13:02:26.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.433+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c1f4380 con 0x7f320c0cca20 2024-03-22T13:02:26.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.434+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c154ef0 con 0x7f320c0cca20 2024-03-22T13:02:26.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.435+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c075650 con 0x7f320c0cca20 2024-03-22T13:02:26.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.472+0000 7f3203e00640 1 -- 172.21.15.6:0/909757049 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f31f801a070 con 0x7f320c0cca20 2024-03-22T13:02:26.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.473+0000 7f3203e00640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31ec06bc20 0x7f31ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:26.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.474+0000 7f320be00640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31ec06bc20 0x7f31ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:26.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.475+0000 7f320be00640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31ec06bc20 0x7f31ec06e0e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f31fc000f60 tx=0x7f31fc007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:26.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.772+0000 7f3203e00640 1 -- 172.21.15.6:0/909757049 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f31f8056710 con 0x7f320c0cca20 2024-03-22T13:02:26.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:26.773+0000 7f3203e00640 1 -- 172.21.15.6:0/909757049 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f31f8021020 con 0x7f320c0cca20 2024-03-22T13:02:27.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.022+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f320c0af100 con 0x7f320c0cca20 2024-03-22T13:02:27.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.062+0000 7f3203e00640 1 -- 172.21.15.6:0/909757049 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f31f8058d70 con 0x7f320c0cca20 2024-03-22T13:02:27.063 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:27.063 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:27.063 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.065+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31ec06bc20 msgr2=0x7f31ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.065+0000 7f3211400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31ec06bc20 0x7f31ec06e0e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f31fc000f60 tx=0x7f31fc007a80 comp rx=0 tx=0).stop 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.065+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0cca20 msgr2=0x7f320c157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.065+0000 7f3211400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0cca20 0x7f320c157a70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31f800c430 tx=0x7f31f800c8f0 comp rx=0 tx=0).stop 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 shutdown_connections 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f320c15b7a0 0x7f320c150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f320c0cca20 0x7f320c157a70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31ec06bc20 0x7f31ec06e0e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 --2- 172.21.15.6:0/909757049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f320c0cc640 0x7f320c1f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 >> 172.21.15.6:0/909757049 conn(0x7f320c0cf9c0 msgr2=0x7f320c159150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:27.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 shutdown_connections 2024-03-22T13:02:27.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:27.066+0000 7f3211400640 1 -- 172.21.15.6:0/909757049 wait complete. 2024-03-22T13:02:27.688 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:27.689 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:28.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.894+0000 7f47dbe00640 1 Processor -- start 2024-03-22T13:02:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.895+0000 7f47dbe00640 1 -- start start 2024-03-22T13:02:28.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.896+0000 7f47dbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f47d41545a0 0x7f47d40c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:28.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.896+0000 7f47dbe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f47d40c96e0 0x7f47d415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:28.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.896+0000 7f47dbe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d415a3c0 con 0x7f47d40c9c30 2024-03-22T13:02:28.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.896+0000 7f47dbe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d40c4130 con 0x7f47d41545a0 2024-03-22T13:02:28.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.896+0000 7f47dbe00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d40c4410 con 0x7f47d40c96e0 2024-03-22T13:02:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.897+0000 7f47d9e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f47d40c96e0 0x7f47d415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.898+0000 7f47d9e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f47d40c96e0 0x7f47d415ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56448/0 (socket says 172.21.15.6:56448) 2024-03-22T13:02:28.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.898+0000 7f47d9e00640 1 -- 172.21.15.6:0/1629098709 learned_addr learned my addr 172.21.15.6:0/1629098709 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.900+0000 7f47d9e00640 1 -- 172.21.15.6:0/1629098709 >> v2:172.21.15.6:3300/0 conn(0x7f47d41545a0 msgr2=0x7f47d40c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.900+0000 7f47d9e00640 1 --2- 172.21.15.6:0/1629098709 >> v2:172.21.15.6:3300/0 conn(0x7f47d41545a0 0x7f47d40c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.900+0000 7f47d9e00640 1 -- 172.21.15.6:0/1629098709 >> v1:172.21.15.6:6789/0 conn(0x7f47d40c9c30 legacy=0x7f47d40ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:28.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.900+0000 7f47d9e00640 1 -- 172.21.15.6:0/1629098709 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d415a660 con 0x7f47d40c96e0 2024-03-22T13:02:28.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.902+0000 7f47d9e00640 1 --2- 172.21.15.6:0/1629098709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c96e0 0x7f47d415ae60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f47d4073780 tx=0x7f47cc02bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cb3a890fb32b512 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:28.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.907+0000 7f47d3e00640 1 -- 172.21.15.6:0/1629098709 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc02cb20 con 0x7f47d40c96e0 2024-03-22T13:02:28.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.907+0000 7f47d3e00640 1 -- 172.21.15.6:0/1629098709 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc02ccc0 con 0x7f47d40c96e0 2024-03-22T13:02:28.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.907+0000 7f47d3e00640 1 -- 172.21.15.6:0/1629098709 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc034610 con 0x7f47d40c96e0 2024-03-22T13:02:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 -- 172.21.15.6:0/1629098709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c96e0 msgr2=0x7f47d415ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 --2- 172.21.15.6:0/1629098709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c96e0 0x7f47d415ae60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f47d4073780 tx=0x7f47cc02bf80 comp rx=0 tx=0).stop 2024-03-22T13:02:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 -- 172.21.15.6:0/1629098709 shutdown_connections 2024-03-22T13:02:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 --2- 172.21.15.6:0/1629098709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c96e0 0x7f47d415ae60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 --2- 172.21.15.6:0/1629098709 >> v2:172.21.15.6:3300/0 conn(0x7f47d41545a0 0x7f47d40c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 -- 172.21.15.6:0/1629098709 >> 172.21.15.6:0/1629098709 conn(0x7f47d40714f0 msgr2=0x7f47d4071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 -- 172.21.15.6:0/1629098709 shutdown_connections 2024-03-22T13:02:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.908+0000 7f47dbe00640 1 -- 172.21.15.6:0/1629098709 wait complete. 2024-03-22T13:02:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.910+0000 7f47dbe00640 1 Processor -- start 2024-03-22T13:02:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.911+0000 7f47dbe00640 1 -- start start 2024-03-22T13:02:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.912+0000 7f47dbe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f47d40c96e0 0x7f47d41f0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.912+0000 7f47dbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c9c30 0x7f47d40c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.912+0000 7f47dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d41545a0 0x7f47d41555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.912+0000 7f47dbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d4154980 con 0x7f47d41545a0 2024-03-22T13:02:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.912+0000 7f47dbe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d40c46f0 con 0x7f47d40c9c30 2024-03-22T13:02:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.912+0000 7f47dbe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d40cadc0 con 0x7f47d40c96e0 2024-03-22T13:02:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.913+0000 7f47d9400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f47d40c96e0 0x7f47d41f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.913+0000 7f47d8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c9c30 0x7f47d40c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.914+0000 7f47d9400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f47d40c96e0 0x7f47d41f0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40756/0 (socket says 172.21.15.6:40756) 2024-03-22T13:02:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.914+0000 7f47d9400640 1 -- 172.21.15.6:0/3595215807 learned_addr learned my addr 172.21.15.6:0/3595215807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.915+0000 7f47d9e00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d41545a0 0x7f47d41555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.917+0000 7f47d8a00640 1 -- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f47d40c96e0 msgr2=0x7f47d41f0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.917+0000 7f47d8a00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f47d40c96e0 0x7f47d41f0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.917+0000 7f47d8a00640 1 -- 172.21.15.6:0/3595215807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d41545a0 msgr2=0x7f47d41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.917+0000 7f47d8a00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d41545a0 0x7f47d41555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.917+0000 7f47d8a00640 1 -- 172.21.15.6:0/3595215807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4156530 con 0x7f47d40c9c30 2024-03-22T13:02:28.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.922+0000 7f47d8a00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c9c30 0x7f47d40c70c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47c000b3e0 tx=0x7f47c000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:28.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.925+0000 7f47d1600640 1 -- 172.21.15.6:0/3595215807 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0015020 con 0x7f47d40c9c30 2024-03-22T13:02:28.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.925+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d41f0ed0 con 0x7f47d40c9c30 2024-03-22T13:02:28.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.925+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d41f1490 con 0x7f47d40c9c30 2024-03-22T13:02:28.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:28.926+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d40cd4e0 con 0x7f47d40c9c30 2024-03-22T13:02:29.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.129+0000 7f47d1600640 1 -- 172.21.15.6:0/3595215807 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c0011830 con 0x7f47d40c9c30 2024-03-22T13:02:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.211+0000 7f47d1600640 1 -- 172.21.15.6:0/3595215807 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0011c20 con 0x7f47d40c9c30 2024-03-22T13:02:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.214+0000 7f47d1600640 1 -- 172.21.15.6:0/3595215807 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f47c0018020 con 0x7f47d40c9c30 2024-03-22T13:02:29.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.216+0000 7f47d1600640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f47b406bc00 0x7f47b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:29.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.217+0000 7f47d1600640 1 -- 172.21.15.6:0/3595215807 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f47c0011dc0 con 0x7f47d40c9c30 2024-03-22T13:02:29.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.217+0000 7f47d9400640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f47b406bc00 0x7f47b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.218+0000 7f47d9400640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f47b406bc00 0x7f47b406e0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009a40 tx=0x7f47c80091c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.229+0000 7f47d1600640 1 -- 172.21.15.6:0/3595215807 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f47d40cd4e0 con 0x7f47d40c9c30 2024-03-22T13:02:29.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.526+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47d41f16d0 con 0x7f47d40c9c30 2024-03-22T13:02:29.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.548+0000 7f47d1600640 1 -- 172.21.15.6:0/3595215807 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f47c001f020 con 0x7f47d40c9c30 2024-03-22T13:02:29.548 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:29.549 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:29.549 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f47b406bc00 msgr2=0x7f47b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f47b406bc00 0x7f47b406e0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009a40 tx=0x7f47c80091c0 comp rx=0 tx=0).stop 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c9c30 msgr2=0x7f47d40c70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c9c30 0x7f47d40c70c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47c000b3e0 tx=0x7f47c000b8a0 comp rx=0 tx=0).stop 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 shutdown_connections 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47d41545a0 0x7f47d41555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f47b406bc00 0x7f47b406e0c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47d40c9c30 0x7f47d40c70c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:29.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 --2- 172.21.15.6:0/3595215807 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f47d40c96e0 0x7f47d41f0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:29.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 >> 172.21.15.6:0/3595215807 conn(0x7f47d40714f0 msgr2=0x7f47d40cbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:29.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 shutdown_connections 2024-03-22T13:02:29.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:29.552+0000 7f47dbe00640 1 -- 172.21.15.6:0/3595215807 wait complete. 2024-03-22T13:02:30.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:30.173 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:30.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:02:30.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:02:31.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.370+0000 7f6ceca00640 1 Processor -- start 2024-03-22T13:02:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.370+0000 7f6ceca00640 1 -- start start 2024-03-22T13:02:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.372+0000 7f6ceca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ce80ca520 0x7f6ce806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.372+0000 7f6ceca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6ce80ccad0 0x7f6ce815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.372+0000 7f6ceca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce806c7a0 con 0x7f6ce815b0e0 2024-03-22T13:02:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.372+0000 7f6ceca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce80cd000 con 0x7f6ce80ccad0 2024-03-22T13:02:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.372+0000 7f6ceca00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce806ca80 con 0x7f6ce80ca520 2024-03-22T13:02:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.373+0000 7f6ce5e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ce80ca520 0x7f6ce806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.373+0000 7f6ce6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6ce80ccad0 0x7f6ce815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.373+0000 7f6ce5e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6ce80ca520 0x7f6ce806d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56466/0 (socket says 172.21.15.6:56466) 2024-03-22T13:02:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.373+0000 7f6ce5e00640 1 -- 172.21.15.6:0/1315242874 learned_addr learned my addr 172.21.15.6:0/1315242874 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.376+0000 7f6ce6800640 1 -- 172.21.15.6:0/1315242874 >> v2:172.21.15.92:3300/0 conn(0x7f6ce80ca520 msgr2=0x7f6ce806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.377+0000 7f6ce6800640 1 --2- 172.21.15.6:0/1315242874 >> v2:172.21.15.92:3300/0 conn(0x7f6ce80ca520 0x7f6ce806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.377+0000 7f6ce6800640 1 -- 172.21.15.6:0/1315242874 >> v1:172.21.15.6:6789/0 conn(0x7f6ce815b0e0 legacy=0x7f6ce80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.377+0000 7f6ce6800640 1 -- 172.21.15.6:0/1315242874 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce806cd60 con 0x7f6ce80ccad0 2024-03-22T13:02:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.381+0000 7f6ce6800640 1 --2- 172.21.15.6:0/1315242874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ccad0 0x7f6ce815b980 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80d0ca0 tx=0x7f6ce002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd85fd6e196e4129 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:31.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.388+0000 7f6ce4a00640 1 -- 172.21.15.6:0/1315242874 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0039070 con 0x7f6ce80ccad0 2024-03-22T13:02:31.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.388+0000 7f6ce4a00640 1 -- 172.21.15.6:0/1315242874 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce002cc80 con 0x7f6ce80ccad0 2024-03-22T13:02:31.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.388+0000 7f6ce4a00640 1 -- 172.21.15.6:0/1315242874 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0034660 con 0x7f6ce80ccad0 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.388+0000 7f6ceca00640 1 -- 172.21.15.6:0/1315242874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ccad0 msgr2=0x7f6ce815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.388+0000 7f6ceca00640 1 --2- 172.21.15.6:0/1315242874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ccad0 0x7f6ce815b980 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80d0ca0 tx=0x7f6ce002bed0 comp rx=0 tx=0).stop 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.389+0000 7f6ceca00640 1 -- 172.21.15.6:0/1315242874 shutdown_connections 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.389+0000 7f6ceca00640 1 --2- 172.21.15.6:0/1315242874 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ccad0 0x7f6ce815b980 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.389+0000 7f6ceca00640 1 --2- 172.21.15.6:0/1315242874 >> v2:172.21.15.92:3300/0 conn(0x7f6ce80ca520 0x7f6ce806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.389+0000 7f6ceca00640 1 -- 172.21.15.6:0/1315242874 >> 172.21.15.6:0/1315242874 conn(0x7f6ce80b16b0 msgr2=0x7f6ce80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.389+0000 7f6ceca00640 1 -- 172.21.15.6:0/1315242874 shutdown_connections 2024-03-22T13:02:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.389+0000 7f6ceca00640 1 -- 172.21.15.6:0/1315242874 wait complete. 2024-03-22T13:02:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.391+0000 7f6ceca00640 1 Processor -- start 2024-03-22T13:02:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.391+0000 7f6ceca00640 1 -- start start 2024-03-22T13:02:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.392+0000 7f6ceca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ca520 0x7f6ce8154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.392+0000 7f6ceca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce80ccad0 0x7f6ce81e5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.392+0000 7f6ceca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce815b0e0 0x7f6ce81e61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.392+0000 7f6ceca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce8154c20 con 0x7f6ce80ca520 2024-03-22T13:02:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.392+0000 7f6ceca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce806d320 con 0x7f6ce815b0e0 2024-03-22T13:02:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.392+0000 7f6ceca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce80ca900 con 0x7f6ce80ccad0 2024-03-22T13:02:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.393+0000 7f6ce5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ca520 0x7f6ce8154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.393+0000 7f6ce6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce815b0e0 0x7f6ce81e61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.393+0000 7f6ce5400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce80ccad0 0x7f6ce81e5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.393+0000 7f6ce5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ca520 0x7f6ce8154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47478/0 (socket says 172.21.15.6:47478) 2024-03-22T13:02:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.393+0000 7f6ce5e00640 1 -- 172.21.15.6:0/2233081400 learned_addr learned my addr 172.21.15.6:0/2233081400 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.397+0000 7f6ce5e00640 1 -- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce80ccad0 msgr2=0x7f6ce81e5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.397+0000 7f6ce5e00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce80ccad0 0x7f6ce81e5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.397+0000 7f6ce5e00640 1 -- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce815b0e0 msgr2=0x7f6ce81e61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.397+0000 7f6ce5e00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce815b0e0 0x7f6ce81e61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.397+0000 7f6ce5e00640 1 -- 172.21.15.6:0/2233081400 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce81f11a0 con 0x7f6ce80ca520 2024-03-22T13:02:31.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.400+0000 7f6ce5e00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ca520 0x7f6ce8154390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009fd0 tx=0x7f6cd800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:31.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.404+0000 7f6cde000640 1 -- 172.21.15.6:0/2233081400 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8017070 con 0x7f6ce80ca520 2024-03-22T13:02:31.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.404+0000 7f6cde000640 1 -- 172.21.15.6:0/2233081400 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd800c940 con 0x7f6ce80ca520 2024-03-22T13:02:31.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.404+0000 7f6cde000640 1 -- 172.21.15.6:0/2233081400 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd800cd10 con 0x7f6ce80ca520 2024-03-22T13:02:31.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.404+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce81585d0 con 0x7f6ce80ca520 2024-03-22T13:02:31.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.404+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce8158e30 con 0x7f6ce80ca520 2024-03-22T13:02:31.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.406+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce8158af0 con 0x7f6ce80ca520 2024-03-22T13:02:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.736+0000 7f6cde000640 1 -- 172.21.15.6:0/2233081400 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6cd801a070 con 0x7f6ce80ca520 2024-03-22T13:02:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.739+0000 7f6cde000640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6cc806bc20 0x7f6cc806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.739+0000 7f6ce5400640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6cc806bc20 0x7f6cc806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:31.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.740+0000 7f6ce5400640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6cc806bc20 0x7f6cc806e0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ce806d910 tx=0x7f6cd4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:31.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.777+0000 7f6cde000640 1 -- 172.21.15.6:0/2233081400 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6cd8057610 con 0x7f6ce80ca520 2024-03-22T13:02:31.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:31.778+0000 7f6cde000640 1 -- 172.21.15.6:0/2233081400 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6ce8158af0 con 0x7f6ce80ca520 2024-03-22T13:02:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.050+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ce80756c0 con 0x7f6ce80ca520 2024-03-22T13:02:32.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.100+0000 7f6cde000640 1 -- 172.21.15.6:0/2233081400 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f6cd8059570 con 0x7f6ce80ca520 2024-03-22T13:02:32.101 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:32.101 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:32.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6cc806bc20 msgr2=0x7f6cc806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:32.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6cc806bc20 0x7f6cc806e0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ce806d910 tx=0x7f6cd4009040 comp rx=0 tx=0).stop 2024-03-22T13:02:32.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ca520 msgr2=0x7f6ce8154390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:32.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ca520 0x7f6ce8154390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009fd0 tx=0x7f6cd800ed40 comp rx=0 tx=0).stop 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 shutdown_connections 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6cc806bc20 0x7f6cc806e0e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ce815b0e0 0x7f6ce81e61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ce80ccad0 0x7f6ce81e5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 --2- 172.21.15.6:0/2233081400 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ce80ca520 0x7f6ce8154390 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 >> 172.21.15.6:0/2233081400 conn(0x7f6ce80b16b0 msgr2=0x7f6ce80b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 shutdown_connections 2024-03-22T13:02:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:32.103+0000 7f6ceca00640 1 -- 172.21.15.6:0/2233081400 wait complete. 2024-03-22T13:02:32.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:32.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:33.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:33.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.921+0000 7fe045600640 1 Processor -- start 2024-03-22T13:02:33.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.921+0000 7fe045600640 1 -- start start 2024-03-22T13:02:33.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.923+0000 7fe045600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0400ce8a0 0x7fe04006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:33.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.923+0000 7fe045600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe04015ae30 0x7fe0400cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:33.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.923+0000 7fe045600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04015b240 con 0x7fe0400cc380 2024-03-22T13:02:33.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.923+0000 7fe045600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0400cee00 con 0x7fe04015ae30 2024-03-22T13:02:33.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.923+0000 7fe045600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0400cf0c0 con 0x7fe0400ce8a0 2024-03-22T13:02:33.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.924+0000 7fe044c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe04015ae30 0x7fe0400cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:33.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.924+0000 7fe03f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0400ce8a0 0x7fe04006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:33.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.924+0000 7fe03f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0400ce8a0 0x7fe04006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37912/0 (socket says 172.21.15.6:37912) 2024-03-22T13:02:33.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.924+0000 7fe03f400640 1 -- 172.21.15.6:0/1486341384 learned_addr learned my addr 172.21.15.6:0/1486341384 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:33.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.927+0000 7fe03f400640 1 -- 172.21.15.6:0/1486341384 >> v2:172.21.15.6:3300/0 conn(0x7fe04015ae30 msgr2=0x7fe0400cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:33.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.927+0000 7fe03f400640 1 --2- 172.21.15.6:0/1486341384 >> v2:172.21.15.6:3300/0 conn(0x7fe04015ae30 0x7fe0400cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:33.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.927+0000 7fe03f400640 1 -- 172.21.15.6:0/1486341384 >> v1:172.21.15.6:6789/0 conn(0x7fe0400cc380 legacy=0x7fe0400cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:33.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.927+0000 7fe03f400640 1 -- 172.21.15.6:0/1486341384 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0400cf3a0 con 0x7fe0400ce8a0 2024-03-22T13:02:33.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.929+0000 7fe03f400640 1 --2- 172.21.15.6:0/1486341384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400ce8a0 0x7fe04006c230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe0400cb7f0 tx=0x7fe02c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce67713918da49af server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.933+0000 7fe03ea00640 1 -- 172.21.15.6:0/1486341384 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c034070 con 0x7fe0400ce8a0 2024-03-22T13:02:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.934+0000 7fe03ea00640 1 -- 172.21.15.6:0/1486341384 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02c030860 con 0x7fe0400ce8a0 2024-03-22T13:02:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.934+0000 7fe03ea00640 1 -- 172.21.15.6:0/1486341384 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c030c30 con 0x7fe0400ce8a0 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.934+0000 7fe045600640 1 -- 172.21.15.6:0/1486341384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400ce8a0 msgr2=0x7fe04006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.934+0000 7fe045600640 1 --2- 172.21.15.6:0/1486341384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400ce8a0 0x7fe04006c230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe0400cb7f0 tx=0x7fe02c02b9a0 comp rx=0 tx=0).stop 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.935+0000 7fe045600640 1 -- 172.21.15.6:0/1486341384 shutdown_connections 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.935+0000 7fe045600640 1 --2- 172.21.15.6:0/1486341384 >> v2:172.21.15.6:3300/0 conn(0x7fe04015ae30 0x7fe0400cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.935+0000 7fe045600640 1 --2- 172.21.15.6:0/1486341384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400ce8a0 0x7fe04006c230 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.935+0000 7fe045600640 1 -- 172.21.15.6:0/1486341384 >> 172.21.15.6:0/1486341384 conn(0x7fe0400b1480 msgr2=0x7fe0400b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.935+0000 7fe045600640 1 -- 172.21.15.6:0/1486341384 shutdown_connections 2024-03-22T13:02:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.935+0000 7fe045600640 1 -- 172.21.15.6:0/1486341384 wait complete. 2024-03-22T13:02:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.936+0000 7fe045600640 1 Processor -- start 2024-03-22T13:02:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.936+0000 7fe045600640 1 -- start start 2024-03-22T13:02:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.937+0000 7fe045600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400cc380 0x7fe04015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.937+0000 7fe045600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0400ce8a0 0x7fe040153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.937+0000 7fe045600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe04015ae30 0x7fe0400b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:33.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.937+0000 7fe045600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04015a910 con 0x7fe04015ae30 2024-03-22T13:02:33.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.937+0000 7fe045600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04015a630 con 0x7fe0400cc380 2024-03-22T13:02:33.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.937+0000 7fe045600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0401549d0 con 0x7fe0400ce8a0 2024-03-22T13:02:33.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.938+0000 7fe03fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400cc380 0x7fe04015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:33.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.938+0000 7fe03f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0400ce8a0 0x7fe040153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.939+0000 7fe044c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe04015ae30 0x7fe0400b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.939+0000 7fe03fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400cc380 0x7fe04015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37922/0 (socket says 172.21.15.6:37922) 2024-03-22T13:02:33.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.939+0000 7fe03fe00640 1 -- 172.21.15.6:0/1654973509 learned_addr learned my addr 172.21.15.6:0/1654973509 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.942+0000 7fe03fe00640 1 -- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0400ce8a0 msgr2=0x7fe040153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.942+0000 7fe03fe00640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0400ce8a0 0x7fe040153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:33.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.942+0000 7fe03fe00640 1 -- 172.21.15.6:0/1654973509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe04015ae30 msgr2=0x7fe0400b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.942+0000 7fe03fe00640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe04015ae30 0x7fe0400b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:33.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.942+0000 7fe03fe00640 1 -- 172.21.15.6:0/1654973509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0400b4c30 con 0x7fe0400cc380 2024-03-22T13:02:33.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.945+0000 7fe03fe00640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400cc380 0x7fe04015e360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe03000c420 tx=0x7fe03000c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:33.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.950+0000 7fe037e00640 1 -- 172.21.15.6:0/1654973509 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0300052d0 con 0x7fe0400cc380 2024-03-22T13:02:33.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.950+0000 7fe037e00640 1 -- 172.21.15.6:0/1654973509 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe030005490 con 0x7fe0400cc380 2024-03-22T13:02:33.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.950+0000 7fe037e00640 1 -- 172.21.15.6:0/1654973509 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030007260 con 0x7fe0400cc380 2024-03-22T13:02:33.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.950+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04015b4e0 con 0x7fe0400cc380 2024-03-22T13:02:33.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.950+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0400b3140 con 0x7fe0400cc380 2024-03-22T13:02:33.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.952+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe040075650 con 0x7fe0400cc380 2024-03-22T13:02:33.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.982+0000 7fe037e00640 1 -- 172.21.15.6:0/1654973509 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe030007400 con 0x7fe0400cc380 2024-03-22T13:02:33.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.984+0000 7fe037e00640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe02006bbd0 0x7fe02006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:33.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.984+0000 7fe03f400640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe02006bbd0 0x7fe02006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.985+0000 7fe03f400640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe02006bbd0 0x7fe02006e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe02c036000 tx=0x7fe02c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:33.987+0000 7fe037e00640 1 -- 172.21.15.6:0/1654973509 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe030056c30 con 0x7fe0400cc380 2024-03-22T13:02:34.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.248+0000 7fe037e00640 1 -- 172.21.15.6:0/1654973509 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe030058b90 con 0x7fe0400cc380 2024-03-22T13:02:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.544+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe04006de60 con 0x7fe0400cc380 2024-03-22T13:02:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.558+0000 7fe037e00640 1 -- 172.21.15.6:0/1654973509 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fe03005cc30 con 0x7fe0400cc380 2024-03-22T13:02:34.558 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:34.558 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe02006bbd0 msgr2=0x7fe02006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe02006bbd0 0x7fe02006e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe02c036000 tx=0x7fe02c008030 comp rx=0 tx=0).stop 2024-03-22T13:02:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400cc380 msgr2=0x7fe04015e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400cc380 0x7fe04015e360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe03000c420 tx=0x7fe03000c8e0 comp rx=0 tx=0).stop 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 shutdown_connections 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe04015ae30 0x7fe0400b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0400ce8a0 0x7fe040153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe02006bbd0 0x7fe02006e090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 --2- 172.21.15.6:0/1654973509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0400cc380 0x7fe04015e360 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 >> 172.21.15.6:0/1654973509 conn(0x7fe0400b1480 msgr2=0x7fe0401589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.562+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 shutdown_connections 2024-03-22T13:02:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:34.563+0000 7fe045600640 1 -- 172.21.15.6:0/1654973509 wait complete. 2024-03-22T13:02:35.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:35.191 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:36.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:36.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.398+0000 7f74c5400640 1 Processor -- start 2024-03-22T13:02:36.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.398+0000 7f74c5400640 1 -- start start 2024-03-22T13:02:36.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.400+0000 7f74c5400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c0152400 0x7f74c015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:36.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.400+0000 7f74c5400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f74c006f8c0 0x7f74c00c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.400+0000 7f74c5400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c01590b0 con 0x7f74c015bc80 2024-03-22T13:02:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.400+0000 7f74c5400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c00724d0 con 0x7f74c006f8c0 2024-03-22T13:02:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.400+0000 7f74c5400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c00721f0 con 0x7f74c0152400 2024-03-22T13:02:36.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.427+0000 7f74bf200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f74c006f8c0 0x7f74c00c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.428+0000 7f74bde00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f74c015bc80 0x7f74c00716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:44414/0 (socket says 172.21.15.6:44414) 2024-03-22T13:02:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.428+0000 7f74bde00640 1 -- 172.21.15.6:0/121375928 learned_addr learned my addr 172.21.15.6:0/121375928 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:36.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.431+0000 7f74bf200640 1 -- 172.21.15.6:0/121375928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c0152400 msgr2=0x7f74c015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:36.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.431+0000 7f74bf200640 1 --2- 172.21.15.6:0/121375928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c0152400 0x7f74c015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:36.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.431+0000 7f74bf200640 1 -- 172.21.15.6:0/121375928 >> v1:172.21.15.92:6789/0 conn(0x7f74c015bc80 legacy=0x7f74c00716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:36.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.431+0000 7f74bf200640 1 -- 172.21.15.6:0/121375928 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c01582f0 con 0x7f74c006f8c0 2024-03-22T13:02:36.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.438+0000 7f74bf200640 1 --2- 172.21.15.6:0/121375928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c5040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009d90 tx=0x7f74b802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ff76a9f7915251b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:36.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.444+0000 7f74bd400640 1 -- 172.21.15.6:0/121375928 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8039070 con 0x7f74c006f8c0 2024-03-22T13:02:36.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.444+0000 7f74bd400640 1 -- 172.21.15.6:0/121375928 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b802cce0 con 0x7f74c006f8c0 2024-03-22T13:02:36.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.444+0000 7f74bd400640 1 -- 172.21.15.6:0/121375928 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8034660 con 0x7f74c006f8c0 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 -- 172.21.15.6:0/121375928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 msgr2=0x7f74c00c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 --2- 172.21.15.6:0/121375928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c5040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009d90 tx=0x7f74b802bf30 comp rx=0 tx=0).stop 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 -- 172.21.15.6:0/121375928 shutdown_connections 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 --2- 172.21.15.6:0/121375928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c5040 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 --2- 172.21.15.6:0/121375928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c0152400 0x7f74c015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 -- 172.21.15.6:0/121375928 >> 172.21.15.6:0/121375928 conn(0x7f74c00cb930 msgr2=0x7f74c00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 -- 172.21.15.6:0/121375928 shutdown_connections 2024-03-22T13:02:36.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.445+0000 7f74c5400640 1 -- 172.21.15.6:0/121375928 wait complete. 2024-03-22T13:02:36.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.448+0000 7f74c5400640 1 Processor -- start 2024-03-22T13:02:36.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.448+0000 7f74c5400640 1 -- start start 2024-03-22T13:02:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.450+0000 7f74c5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.450+0000 7f74c5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0152400 0x7f74c0155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.450+0000 7f74c5400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c015bc80 0x7f74c014d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.450+0000 7f74c5400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0158b90 con 0x7f74c0152400 2024-03-22T13:02:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.450+0000 7f74c5400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c01588b0 con 0x7f74c006f8c0 2024-03-22T13:02:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.450+0000 7f74c5400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c01527e0 con 0x7f74c015bc80 2024-03-22T13:02:36.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.450+0000 7f74be800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:36.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.451+0000 7f74bde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0152400 0x7f74c0155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:36.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.451+0000 7f74be800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37940/0 (socket says 172.21.15.6:37940) 2024-03-22T13:02:36.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.451+0000 7f74be800640 1 -- 172.21.15.6:0/870019818 learned_addr learned my addr 172.21.15.6:0/870019818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:36.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.456+0000 7f74be800640 1 -- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c015bc80 msgr2=0x7f74c014d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:36.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.456+0000 7f74be800640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c015bc80 0x7f74c014d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:36.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.456+0000 7f74be800640 1 -- 172.21.15.6:0/870019818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0152400 msgr2=0x7f74c0155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:36.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.456+0000 7f74be800640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0152400 0x7f74c0155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:36.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.456+0000 7f74be800640 1 -- 172.21.15.6:0/870019818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c01f4810 con 0x7f74c006f8c0 2024-03-22T13:02:36.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.459+0000 7f74be800640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c1c20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f74b000b4a0 tx=0x7f74b000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:36.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.464+0000 7f74b6a00640 1 -- 172.21.15.6:0/870019818 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0014070 con 0x7f74c006f8c0 2024-03-22T13:02:36.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.464+0000 7f74b6a00640 1 -- 172.21.15.6:0/870019818 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0010610 con 0x7f74c006f8c0 2024-03-22T13:02:36.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.464+0000 7f74b6a00640 1 -- 172.21.15.6:0/870019818 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0010a00 con 0x7f74c006f8c0 2024-03-22T13:02:36.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.465+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c0150100 con 0x7f74c006f8c0 2024-03-22T13:02:36.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.465+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c01503a0 con 0x7f74c006f8c0 2024-03-22T13:02:36.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.466+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c0151f60 con 0x7f74c006f8c0 2024-03-22T13:02:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.490+0000 7f74b6a00640 1 -- 172.21.15.6:0/870019818 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74b0017070 con 0x7f74c006f8c0 2024-03-22T13:02:36.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.491+0000 7f74b6a00640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74a006bbd0 0x7f74a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:36.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.491+0000 7f74bde00640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74a006bbd0 0x7f74a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:36.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.492+0000 7f74bde00640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74a006bbd0 0x7f74a006e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f74c014e370 tx=0x7f74ac007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:36.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.492+0000 7f74b6a00640 1 -- 172.21.15.6:0/870019818 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f74b0056de0 con 0x7f74c006f8c0 2024-03-22T13:02:36.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:36.776+0000 7f74b6a00640 1 -- 172.21.15.6:0/870019818 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f74c0151f60 con 0x7f74c006f8c0 2024-03-22T13:02:37.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.032+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c00756c0 con 0x7f74c006f8c0 2024-03-22T13:02:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.048+0000 7f74b6a00640 1 -- 172.21.15.6:0/870019818 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f74b0058d40 con 0x7f74c006f8c0 2024-03-22T13:02:37.049 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:37.049 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74a006bbd0 msgr2=0x7f74a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74a006bbd0 0x7f74a006e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f74c014e370 tx=0x7f74ac007e80 comp rx=0 tx=0).stop 2024-03-22T13:02:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 msgr2=0x7f74c00c1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c1c20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f74b000b4a0 tx=0x7f74b000b960 comp rx=0 tx=0).stop 2024-03-22T13:02:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 shutdown_connections 2024-03-22T13:02:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74c015bc80 0x7f74c014d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74c0152400 0x7f74c0155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74a006bbd0 0x7f74a006e090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 --2- 172.21.15.6:0/870019818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74c006f8c0 0x7f74c00c1c20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 >> 172.21.15.6:0/870019818 conn(0x7f74c00cb930 msgr2=0x7f74c00c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 shutdown_connections 2024-03-22T13:02:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:37.052+0000 7f74c5400640 1 -- 172.21.15.6:0/870019818 wait complete. 2024-03-22T13:02:37.675 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:37.676 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:38.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:38.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.884+0000 7fd26ea00640 1 Processor -- start 2024-03-22T13:02:38.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.884+0000 7fd26ea00640 1 -- start start 2024-03-22T13:02:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.886+0000 7fd26ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd26806f8c0 0x7fd2680716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.886+0000 7fd26ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd268071c10 0x7fd2680c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.886+0000 7fd26ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2681590b0 con 0x7fd268152400 2024-03-22T13:02:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.886+0000 7fd26ea00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2680724d0 con 0x7fd268071c10 2024-03-22T13:02:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.886+0000 7fd26ea00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680721f0 con 0x7fd26806f8c0 2024-03-22T13:02:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.887+0000 7fd267400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd26806f8c0 0x7fd2680716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.887+0000 7fd26ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd268071c10 0x7fd2680c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.887+0000 7fd267400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd26806f8c0 0x7fd2680716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35740/0 (socket says 172.21.15.6:35740) 2024-03-22T13:02:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.887+0000 7fd267400640 1 -- 172.21.15.6:0/3267673779 learned_addr learned my addr 172.21.15.6:0/3267673779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.891+0000 7fd267400640 1 -- 172.21.15.6:0/3267673779 >> v2:172.21.15.92:3300/0 conn(0x7fd268071c10 msgr2=0x7fd2680c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.891+0000 7fd267400640 1 --2- 172.21.15.6:0/3267673779 >> v2:172.21.15.92:3300/0 conn(0x7fd268071c10 0x7fd2680c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.891+0000 7fd267400640 1 -- 172.21.15.6:0/3267673779 >> v1:172.21.15.6:6789/0 conn(0x7fd268152400 legacy=0x7fd26815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.891+0000 7fd267400640 1 -- 172.21.15.6:0/3267673779 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2681582f0 con 0x7fd26806f8c0 2024-03-22T13:02:38.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.891+0000 7fd26ca00640 1 --2- 172.21.15.6:0/3267673779 >> v2:172.21.15.92:3300/0 conn(0x7fd268071c10 0x7fd2680c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:02:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.894+0000 7fd267400640 1 --2- 172.21.15.6:0/3267673779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd26806f8c0 0x7fd2680716d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd254009870 tx=0x7fd25402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1207f6e549d2452 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.898+0000 7fd266a00640 1 -- 172.21.15.6:0/3267673779 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254034070 con 0x7fd26806f8c0 2024-03-22T13:02:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.898+0000 7fd266a00640 1 -- 172.21.15.6:0/3267673779 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254030840 con 0x7fd26806f8c0 2024-03-22T13:02:38.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.898+0000 7fd266a00640 1 -- 172.21.15.6:0/3267673779 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254030c10 con 0x7fd26806f8c0 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 -- 172.21.15.6:0/3267673779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd26806f8c0 msgr2=0x7fd2680716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3267673779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd26806f8c0 0x7fd2680716d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd254009870 tx=0x7fd25402b980 comp rx=0 tx=0).stop 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 -- 172.21.15.6:0/3267673779 shutdown_connections 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3267673779 >> v2:172.21.15.92:3300/0 conn(0x7fd268071c10 0x7fd2680c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3267673779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd26806f8c0 0x7fd2680716d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 -- 172.21.15.6:0/3267673779 >> 172.21.15.6:0/3267673779 conn(0x7fd2680cb930 msgr2=0x7fd2680cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 -- 172.21.15.6:0/3267673779 shutdown_connections 2024-03-22T13:02:38.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.899+0000 7fd26ea00640 1 -- 172.21.15.6:0/3267673779 wait complete. 2024-03-22T13:02:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.902+0000 7fd26ea00640 1 Processor -- start 2024-03-22T13:02:38.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.902+0000 7fd26ea00640 1 -- start start 2024-03-22T13:02:38.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.903+0000 7fd26ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd26806f8c0 0x7fd268155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:38.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.903+0000 7fd26ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd268071c10 0x7fd2681e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:38.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.904+0000 7fd26ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd268152400 0x7fd2680c1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:38.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.904+0000 7fd26ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd268158b90 con 0x7fd268152400 2024-03-22T13:02:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.904+0000 7fd26ea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2681588b0 con 0x7fd26806f8c0 2024-03-22T13:02:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.904+0000 7fd26ea00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2681527e0 con 0x7fd268071c10 2024-03-22T13:02:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.904+0000 7fd267e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd26806f8c0 0x7fd268155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.905+0000 7fd267400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd268071c10 0x7fd2681e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.905+0000 7fd267e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd26806f8c0 0x7fd268155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37958/0 (socket says 172.21.15.6:37958) 2024-03-22T13:02:38.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.905+0000 7fd267e00640 1 -- 172.21.15.6:0/3263899519 learned_addr learned my addr 172.21.15.6:0/3263899519 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:38.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.906+0000 7fd26ca00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd268152400 0x7fd2680c1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:38.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.909+0000 7fd267400640 1 -- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd26806f8c0 msgr2=0x7fd268155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:38.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.909+0000 7fd267400640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd26806f8c0 0x7fd268155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:38.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.909+0000 7fd267400640 1 -- 172.21.15.6:0/3263899519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd268152400 msgr2=0x7fd2680c1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:38.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.909+0000 7fd267400640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd268152400 0x7fd2680c1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:38.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.909+0000 7fd267400640 1 -- 172.21.15.6:0/3263899519 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2681e43c0 con 0x7fd268071c10 2024-03-22T13:02:38.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.912+0000 7fd267400640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd268071c10 0x7fd2681e34f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd254033000 tx=0x7fd254004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.917+0000 7fd25fe00640 1 -- 172.21.15.6:0/3263899519 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254048070 con 0x7fd268071c10 2024-03-22T13:02:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.917+0000 7fd25fe00640 1 -- 172.21.15.6:0/3263899519 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254030640 con 0x7fd268071c10 2024-03-22T13:02:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.917+0000 7fd25fe00640 1 -- 172.21.15.6:0/3263899519 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254034040 con 0x7fd268071c10 2024-03-22T13:02:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.917+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2681f0540 con 0x7fd268071c10 2024-03-22T13:02:38.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.917+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2681e40e0 con 0x7fd268071c10 2024-03-22T13:02:38.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.919+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2680756c0 con 0x7fd268071c10 2024-03-22T13:02:38.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.943+0000 7fd25fe00640 1 -- 172.21.15.6:0/3263899519 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd25404b070 con 0x7fd268071c10 2024-03-22T13:02:38.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.945+0000 7fd25fe00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd24806bc20 0x7fd24806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.945+0000 7fd267e00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd24806bc20 0x7fd24806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:38.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.946+0000 7fd25fe00640 1 -- 172.21.15.6:0/3263899519 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd2540307f0 con 0x7fd268071c10 2024-03-22T13:02:38.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:38.946+0000 7fd267e00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd24806bc20 0x7fd24806e0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd2580099a0 tx=0x7fd2580090e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:39.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.218+0000 7fd25fe00640 1 -- 172.21.15.6:0/3263899519 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd254052020 con 0x7fd268071c10 2024-03-22T13:02:39.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.474+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2681512d0 con 0x7fd268071c10 2024-03-22T13:02:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.488+0000 7fd25fe00640 1 -- 172.21.15.6:0/3263899519 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fd25407b3b0 con 0x7fd268071c10 2024-03-22T13:02:39.488 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:39.488 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:39.488 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd24806bc20 msgr2=0x7fd24806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd24806bc20 0x7fd24806e0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd2580099a0 tx=0x7fd2580090e0 comp rx=0 tx=0).stop 2024-03-22T13:02:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd268071c10 msgr2=0x7fd2681e34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd268071c10 0x7fd2681e34f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd254033000 tx=0x7fd254004730 comp rx=0 tx=0).stop 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 shutdown_connections 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd268152400 0x7fd2680c1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd268071c10 0x7fd2681e34f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd24806bc20 0x7fd24806e0e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 --2- 172.21.15.6:0/3263899519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd26806f8c0 0x7fd268155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 >> 172.21.15.6:0/3263899519 conn(0x7fd2680cb930 msgr2=0x7fd26814f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 shutdown_connections 2024-03-22T13:02:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:39.490+0000 7fd26ea00640 1 -- 172.21.15.6:0/3263899519 wait complete. 2024-03-22T13:02:40.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:40.120 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:41.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:41.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.323+0000 7fd16ac00640 1 Processor -- start 2024-03-22T13:02:41.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.323+0000 7fd16ac00640 1 -- start start 2024-03-22T13:02:41.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.325+0000 7fd16ac00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1640ca520 0x7fd16406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:41.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.325+0000 7fd16ac00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd16415b0e0 0x7fd16415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:41.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.325+0000 7fd16ac00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1640ca900 0x7fd1640cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:41.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.325+0000 7fd16ac00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640cd000 con 0x7fd1640ca900 2024-03-22T13:02:41.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.325+0000 7fd16ac00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16406c7a0 con 0x7fd16415b0e0 2024-03-22T13:02:41.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.325+0000 7fd16ac00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16406ca80 con 0x7fd1640ca520 2024-03-22T13:02:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.326+0000 7fd168c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1640ca900 0x7fd1640cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.326+0000 7fd163e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1640ca520 0x7fd16406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.326+0000 7fd163400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd16415b0e0 0x7fd16415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.327+0000 7fd163e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1640ca520 0x7fd16406d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35756/0 (socket says 172.21.15.6:35756) 2024-03-22T13:02:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.327+0000 7fd163e00640 1 -- 172.21.15.6:0/3666299920 learned_addr learned my addr 172.21.15.6:0/3666299920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.327+0000 7fd163400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd16415b0e0 0x7fd16415b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37962/0 (socket says 172.21.15.6:37962) 2024-03-22T13:02:41.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.329+0000 7fd168c00640 1 -- 172.21.15.6:0/3666299920 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1640ca520 msgr2=0x7fd16406d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:41.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.329+0000 7fd168c00640 1 --2- 172.21.15.6:0/3666299920 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1640ca520 0x7fd16406d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.329+0000 7fd168c00640 1 -- 172.21.15.6:0/3666299920 >> v2:172.21.15.92:3300/0 conn(0x7fd16415b0e0 msgr2=0x7fd16415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:41.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.329+0000 7fd168c00640 1 --2- 172.21.15.6:0/3666299920 >> v2:172.21.15.92:3300/0 conn(0x7fd16415b0e0 0x7fd16415b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.329+0000 7fd168c00640 1 -- 172.21.15.6:0/3666299920 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16406cd60 con 0x7fd1640ca900 2024-03-22T13:02:41.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.332+0000 7fd168c00640 1 --2- 172.21.15.6:0/3666299920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640cc3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd16408d040 tx=0x7fd15c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ca7c471222aad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:41.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.337+0000 7fd162a00640 1 -- 172.21.15.6:0/3666299920 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c039070 con 0x7fd1640ca900 2024-03-22T13:02:41.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.337+0000 7fd162a00640 1 -- 172.21.15.6:0/3666299920 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c02cbd0 con 0x7fd1640ca900 2024-03-22T13:02:41.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.337+0000 7fd162a00640 1 -- 172.21.15.6:0/3666299920 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c034440 con 0x7fd1640ca900 2024-03-22T13:02:41.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.338+0000 7fd16ac00640 1 -- 172.21.15.6:0/3666299920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 msgr2=0x7fd1640cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:41.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.338+0000 7fd16ac00640 1 --2- 172.21.15.6:0/3666299920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640cc3c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd16408d040 tx=0x7fd15c02be20 comp rx=0 tx=0).stop 2024-03-22T13:02:41.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.338+0000 7fd16ac00640 1 -- 172.21.15.6:0/3666299920 shutdown_connections 2024-03-22T13:02:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.338+0000 7fd16ac00640 1 --2- 172.21.15.6:0/3666299920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640cc3c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.338+0000 7fd16ac00640 1 --2- 172.21.15.6:0/3666299920 >> v2:172.21.15.92:3300/0 conn(0x7fd16415b0e0 0x7fd16415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.338+0000 7fd16ac00640 1 --2- 172.21.15.6:0/3666299920 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1640ca520 0x7fd16406d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.338+0000 7fd16ac00640 1 -- 172.21.15.6:0/3666299920 >> 172.21.15.6:0/3666299920 conn(0x7fd1640b16b0 msgr2=0x7fd1640b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.339+0000 7fd16ac00640 1 -- 172.21.15.6:0/3666299920 shutdown_connections 2024-03-22T13:02:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.339+0000 7fd16ac00640 1 -- 172.21.15.6:0/3666299920 wait complete. 2024-03-22T13:02:41.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.341+0000 7fd16ac00640 1 Processor -- start 2024-03-22T13:02:41.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.341+0000 7fd16ac00640 1 -- start start 2024-03-22T13:02:41.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.342+0000 7fd16ac00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1640ca520 0x7fd1641e7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:41.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.342+0000 7fd16ac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:41.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.342+0000 7fd16ac00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd16415b0e0 0x7fd1640c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:41.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.342+0000 7fd16ac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640cad60 con 0x7fd1640ca900 2024-03-22T13:02:41.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.342+0000 7fd16ac00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16406d320 con 0x7fd1640ca520 2024-03-22T13:02:41.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.342+0000 7fd16ac00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd164154c20 con 0x7fd16415b0e0 2024-03-22T13:02:41.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.343+0000 7fd163e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1640ca520 0x7fd1641e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:41.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.343+0000 7fd163400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:41.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.343+0000 7fd168c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd16415b0e0 0x7fd1640c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:41.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.344+0000 7fd163400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35548/0 (socket says 172.21.15.6:35548) 2024-03-22T13:02:41.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.344+0000 7fd163400640 1 -- 172.21.15.6:0/2631825801 learned_addr learned my addr 172.21.15.6:0/2631825801 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:41.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.347+0000 7fd163400640 1 -- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd16415b0e0 msgr2=0x7fd1640c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:41.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.347+0000 7fd163400640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd16415b0e0 0x7fd1640c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.347+0000 7fd163400640 1 -- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1640ca520 msgr2=0x7fd1641e7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:41.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.347+0000 7fd163400640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1640ca520 0x7fd1641e7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.347+0000 7fd163400640 1 -- 172.21.15.6:0/2631825801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164157dd0 con 0x7fd1640ca900 2024-03-22T13:02:41.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.350+0000 7fd163400640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640c70a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e8a0 tx=0x7fd15000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:41.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.355+0000 7fd15be00640 1 -- 172.21.15.6:0/2631825801 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150017070 con 0x7fd1640ca900 2024-03-22T13:02:41.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.355+0000 7fd15be00640 1 -- 172.21.15.6:0/2631825801 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15000c940 con 0x7fd1640ca900 2024-03-22T13:02:41.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.355+0000 7fd15be00640 1 -- 172.21.15.6:0/2631825801 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15000cd30 con 0x7fd1640ca900 2024-03-22T13:02:41.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.355+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd164157b30 con 0x7fd1640ca900 2024-03-22T13:02:41.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.356+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1640c8580 con 0x7fd1640ca900 2024-03-22T13:02:41.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.357+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1640c91f0 con 0x7fd1640ca900 2024-03-22T13:02:41.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.672+0000 7fd15be00640 1 -- 172.21.15.6:0/2631825801 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd15001a070 con 0x7fd1640ca900 2024-03-22T13:02:41.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.673+0000 7fd15be00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd14406bc20 0x7fd14406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:41.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.674+0000 7fd163e00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd14406bc20 0x7fd14406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:41.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.674+0000 7fd163e00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd14406bc20 0x7fd14406e0e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd154006cc0 tx=0x7fd154009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:41.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.698+0000 7fd15be00640 1 -- 172.21.15.6:0/2631825801 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd150057580 con 0x7fd1640ca900 2024-03-22T13:02:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.699+0000 7fd15be00640 1 -- 172.21.15.6:0/2631825801 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1500594e0 con 0x7fd1640ca900 2024-03-22T13:02:41.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.956+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd16415a3a0 con 0x7fd1640ca900 2024-03-22T13:02:41.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.983+0000 7fd15be00640 1 -- 172.21.15.6:0/2631825801 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fd15005d580 con 0x7fd1640ca900 2024-03-22T13:02:41.984 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:41.984 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:41.984 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd14406bc20 msgr2=0x7fd14406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd14406bc20 0x7fd14406e0e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd154006cc0 tx=0x7fd154009240 comp rx=0 tx=0).stop 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 msgr2=0x7fd1640c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640c70a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e8a0 tx=0x7fd15000ed60 comp rx=0 tx=0).stop 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 shutdown_connections 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd16415b0e0 0x7fd1640c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1640ca900 0x7fd1640c70a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd14406bc20 0x7fd14406e0e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 --2- 172.21.15.6:0/2631825801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1640ca520 0x7fd1641e7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 >> 172.21.15.6:0/2631825801 conn(0x7fd1640b16b0 msgr2=0x7fd1640afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.986+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 shutdown_connections 2024-03-22T13:02:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:41.987+0000 7fd16ac00640 1 -- 172.21.15.6:0/2631825801 wait complete. 2024-03-22T13:02:42.609 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:42.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:43.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:43.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.801+0000 7ff87a000640 1 Processor -- start 2024-03-22T13:02:43.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.801+0000 7ff87a000640 1 -- start start 2024-03-22T13:02:43.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.802+0000 7ff87a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff87406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:43.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.802+0000 7ff87a000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8740ce8a0 0x7ff87415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:43.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.802+0000 7ff87a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff8740cc760 0x7ff8740cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:43.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.802+0000 7ff87a000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8740cee00 con 0x7ff8740cc760 2024-03-22T13:02:43.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.802+0000 7ff87a000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8740cf0c0 con 0x7ff8740ce8a0 2024-03-22T13:02:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.802+0000 7ff87a000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740cf3a0 con 0x7ff8740cc380 2024-03-22T13:02:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.803+0000 7ff878c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff87406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.803+0000 7ff873e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8740ce8a0 0x7ff87415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.804+0000 7ff878c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff87406c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44992/0 (socket says 172.21.15.6:44992) 2024-03-22T13:02:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.804+0000 7ff878c00640 1 -- 172.21.15.6:0/3974770752 learned_addr learned my addr 172.21.15.6:0/3974770752 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.806+0000 7ff878c00640 1 -- 172.21.15.6:0/3974770752 >> v2:172.21.15.92:3300/0 conn(0x7ff8740ce8a0 msgr2=0x7ff87415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.806+0000 7ff878c00640 1 --2- 172.21.15.6:0/3974770752 >> v2:172.21.15.92:3300/0 conn(0x7ff8740ce8a0 0x7ff87415ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.806+0000 7ff878c00640 1 -- 172.21.15.6:0/3974770752 >> v2:172.21.15.6:3300/0 conn(0x7ff8740cc760 msgr2=0x7ff8740cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.806+0000 7ff878c00640 1 --2- 172.21.15.6:0/3974770752 >> v2:172.21.15.6:3300/0 conn(0x7ff8740cc760 0x7ff8740cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:43.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.806+0000 7ff878c00640 1 -- 172.21.15.6:0/3974770752 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8740cf680 con 0x7ff8740cc380 2024-03-22T13:02:43.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.809+0000 7ff878c00640 1 --2- 172.21.15.6:0/3974770752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff87406c230 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff8740d0a70 tx=0x7ff86802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abbabf4133dc4921 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:43.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.815+0000 7ff873400640 1 -- 172.21.15.6:0/3974770752 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868035020 con 0x7ff8740cc380 2024-03-22T13:02:43.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.815+0000 7ff873400640 1 -- 172.21.15.6:0/3974770752 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff868031870 con 0x7ff8740cc380 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.815+0000 7ff873400640 1 -- 172.21.15.6:0/3974770752 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868031c40 con 0x7ff8740cc380 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 -- 172.21.15.6:0/3974770752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 msgr2=0x7ff87406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 --2- 172.21.15.6:0/3974770752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff87406c230 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff8740d0a70 tx=0x7ff86802b910 comp rx=0 tx=0).stop 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 -- 172.21.15.6:0/3974770752 shutdown_connections 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 --2- 172.21.15.6:0/3974770752 >> v2:172.21.15.6:3300/0 conn(0x7ff8740cc760 0x7ff8740cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 --2- 172.21.15.6:0/3974770752 >> v2:172.21.15.92:3300/0 conn(0x7ff8740ce8a0 0x7ff87415ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 --2- 172.21.15.6:0/3974770752 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff87406c230 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 -- 172.21.15.6:0/3974770752 >> 172.21.15.6:0/3974770752 conn(0x7ff8740b1480 msgr2=0x7ff8740b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 -- 172.21.15.6:0/3974770752 shutdown_connections 2024-03-22T13:02:43.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.816+0000 7ff87a000640 1 -- 172.21.15.6:0/3974770752 wait complete. 2024-03-22T13:02:43.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.819+0000 7ff87a000640 1 Processor -- start 2024-03-22T13:02:43.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.819+0000 7ff87a000640 1 -- start start 2024-03-22T13:02:43.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.821+0000 7ff87a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff8741e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:43.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.821+0000 7ff87a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8740cc760 0x7ff874148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.821+0000 7ff87a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8740ce8a0 0x7ff8741e59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.821+0000 7ff87a000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8741549d0 con 0x7ff8740cc760 2024-03-22T13:02:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.821+0000 7ff87a000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87415a910 con 0x7ff8740ce8a0 2024-03-22T13:02:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.821+0000 7ff87a000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740ccb90 con 0x7ff8740cc380 2024-03-22T13:02:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.821+0000 7ff878c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff8741e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.822+0000 7ff878c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff8741e5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45006/0 (socket says 172.21.15.6:45006) 2024-03-22T13:02:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.822+0000 7ff878c00640 1 -- 172.21.15.6:0/1393911551 learned_addr learned my addr 172.21.15.6:0/1393911551 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:43.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.825+0000 7ff873e00640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8740cc760 0x7ff874148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.826+0000 7ff879600640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8740ce8a0 0x7ff8741e59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.827+0000 7ff878c00640 1 -- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8740ce8a0 msgr2=0x7ff8741e59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.827+0000 7ff878c00640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8740ce8a0 0x7ff8741e59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.827+0000 7ff878c00640 1 -- 172.21.15.6:0/1393911551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8740cc760 msgr2=0x7ff874148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.827+0000 7ff878c00640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8740cc760 0x7ff874148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.827+0000 7ff878c00640 1 -- 172.21.15.6:0/1393911551 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8741f0ba0 con 0x7ff8740cc380 2024-03-22T13:02:43.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.832+0000 7ff878c00640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff8741e5260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff868009960 tx=0x7ff8680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:43.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.837+0000 7ff870c00640 1 -- 172.21.15.6:0/1393911551 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868035040 con 0x7ff8740cc380 2024-03-22T13:02:43.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.837+0000 7ff870c00640 1 -- 172.21.15.6:0/1393911551 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff868030d00 con 0x7ff8740cc380 2024-03-22T13:02:43.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.837+0000 7ff870c00640 1 -- 172.21.15.6:0/1393911551 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8680456c0 con 0x7ff8740cc380 2024-03-22T13:02:43.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.837+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8741f0e80 con 0x7ff8740cc380 2024-03-22T13:02:43.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.837+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8741f1440 con 0x7ff8740cc380 2024-03-22T13:02:43.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:43.838+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8740c2e10 con 0x7ff8740cc380 2024-03-22T13:02:44.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.114+0000 7ff870c00640 1 -- 172.21.15.6:0/1393911551 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff868038020 con 0x7ff8740cc380 2024-03-22T13:02:44.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.116+0000 7ff870c00640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff85406bc50 0x7ff85406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:44.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.117+0000 7ff873e00640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff85406bc50 0x7ff85406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.118+0000 7ff873e00640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff85406bc50 0x7ff85406e110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff8740bfa50 tx=0x7ff860009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.119+0000 7ff870c00640 1 -- 172.21.15.6:0/1393911551 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff868079660 con 0x7ff8740cc380 2024-03-22T13:02:44.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.136+0000 7ff870c00640 1 -- 172.21.15.6:0/1393911551 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff8740c2e10 con 0x7ff8740cc380 2024-03-22T13:02:44.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.392+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff874075650 con 0x7ff8740cc380 2024-03-22T13:02:44.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.429+0000 7ff870c00640 1 -- 172.21.15.6:0/1393911551 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7ff868054080 con 0x7ff8740cc380 2024-03-22T13:02:44.430 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:44.430 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:44.430 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff85406bc50 msgr2=0x7ff85406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff85406bc50 0x7ff85406e110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff8740bfa50 tx=0x7ff860009190 comp rx=0 tx=0).stop 2024-03-22T13:02:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 msgr2=0x7ff8741e5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff8741e5260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff868009960 tx=0x7ff8680046c0 comp rx=0 tx=0).stop 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 shutdown_connections 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff85406bc50 0x7ff85406e110 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8740ce8a0 0x7ff8741e59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8740cc760 0x7ff874148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 --2- 172.21.15.6:0/1393911551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8740cc380 0x7ff8741e5260 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 >> 172.21.15.6:0/1393911551 conn(0x7ff8740b1480 msgr2=0x7ff874159000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 shutdown_connections 2024-03-22T13:02:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:44.432+0000 7ff87a000640 1 -- 172.21.15.6:0/1393911551 wait complete. 2024-03-22T13:02:45.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:45.053 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:46.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:46.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.247+0000 7fc8c3000640 1 Processor -- start 2024-03-22T13:02:46.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.247+0000 7fc8c3000640 1 -- start start 2024-03-22T13:02:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.249+0000 7fc8c3000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc8bc0c55f0 0x7fc8bc158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.249+0000 7fc8c3000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8bc06ede0 0x7fc8bc06f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.249+0000 7fc8c3000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc06e0f0 con 0x7fc8bc0c5210 2024-03-22T13:02:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.249+0000 7fc8c3000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8bc0c6f90 con 0x7fc8bc06ede0 2024-03-22T13:02:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.249+0000 7fc8c3000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8bc06d350 con 0x7fc8bc0c55f0 2024-03-22T13:02:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.249+0000 7fc8c1200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc8bc0c55f0 0x7fc8bc158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:46.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.249+0000 7fc8c2600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8bc06ede0 0x7fc8bc06f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:46.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.250+0000 7fc8c1200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc8bc0c55f0 0x7fc8bc158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52878/0 (socket says 172.21.15.6:52878) 2024-03-22T13:02:46.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.250+0000 7fc8c1200640 1 -- 172.21.15.6:0/2145044616 learned_addr learned my addr 172.21.15.6:0/2145044616 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.252+0000 7fc8c1200640 1 -- 172.21.15.6:0/2145044616 >> v2:172.21.15.6:3300/0 conn(0x7fc8bc06ede0 msgr2=0x7fc8bc06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.252+0000 7fc8c1200640 1 --2- 172.21.15.6:0/2145044616 >> v2:172.21.15.6:3300/0 conn(0x7fc8bc06ede0 0x7fc8bc06f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.252+0000 7fc8c1200640 1 -- 172.21.15.6:0/2145044616 >> v1:172.21.15.6:6789/0 conn(0x7fc8bc0c5210 legacy=0x7fc8bc0c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.252+0000 7fc8c1200640 1 -- 172.21.15.6:0/2145044616 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0c6cb0 con 0x7fc8bc0c55f0 2024-03-22T13:02:46.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.254+0000 7fc8c1200640 1 --2- 172.21.15.6:0/2145044616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc0c55f0 0x7fc8bc158a60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc06e510 tx=0x7fc8a802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2819207272dea507 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.257+0000 7fc8bbe00640 1 -- 172.21.15.6:0/2145044616 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8035020 con 0x7fc8bc0c55f0 2024-03-22T13:02:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.258+0000 7fc8bbe00640 1 -- 172.21.15.6:0/2145044616 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a8031870 con 0x7fc8bc0c55f0 2024-03-22T13:02:46.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.259+0000 7fc8c3000640 1 -- 172.21.15.6:0/2145044616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc0c55f0 msgr2=0x7fc8bc158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:46.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.259+0000 7fc8c3000640 1 --2- 172.21.15.6:0/2145044616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc0c55f0 0x7fc8bc158a60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc06e510 tx=0x7fc8a802b910 comp rx=0 tx=0).stop 2024-03-22T13:02:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.259+0000 7fc8c3000640 1 -- 172.21.15.6:0/2145044616 shutdown_connections 2024-03-22T13:02:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.259+0000 7fc8c3000640 1 --2- 172.21.15.6:0/2145044616 >> v2:172.21.15.6:3300/0 conn(0x7fc8bc06ede0 0x7fc8bc06f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.259+0000 7fc8c3000640 1 --2- 172.21.15.6:0/2145044616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc0c55f0 0x7fc8bc158a60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.259+0000 7fc8c3000640 1 -- 172.21.15.6:0/2145044616 >> 172.21.15.6:0/2145044616 conn(0x7fc8bc0cb890 msgr2=0x7fc8bc0cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.260+0000 7fc8c3000640 1 -- 172.21.15.6:0/2145044616 shutdown_connections 2024-03-22T13:02:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.260+0000 7fc8c3000640 1 -- 172.21.15.6:0/2145044616 wait complete. 2024-03-22T13:02:46.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.262+0000 7fc8c3000640 1 Processor -- start 2024-03-22T13:02:46.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.262+0000 7fc8c3000640 1 -- start start 2024-03-22T13:02:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.264+0000 7fc8c3000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc06ede0 0x7fc8bc0c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.264+0000 7fc8c3000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8bc0c5210 0x7fc8bc15a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.264+0000 7fc8c3000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8bc0c55f0 0x7fc8bc1f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.264+0000 7fc8c3000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc06dbd0 con 0x7fc8bc0c55f0 2024-03-22T13:02:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.264+0000 7fc8c3000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc06d8f0 con 0x7fc8bc06ede0 2024-03-22T13:02:46.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.264+0000 7fc8c3000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc0c69a0 con 0x7fc8bc0c5210 2024-03-22T13:02:46.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.265+0000 7fc8c1c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc06ede0 0x7fc8bc0c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:46.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.265+0000 7fc8c2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8bc0c55f0 0x7fc8bc1f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:46.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.265+0000 7fc8c1c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc06ede0 0x7fc8bc0c38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52880/0 (socket says 172.21.15.6:52880) 2024-03-22T13:02:46.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.265+0000 7fc8c1c00640 1 -- 172.21.15.6:0/1469288302 learned_addr learned my addr 172.21.15.6:0/1469288302 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:46.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.265+0000 7fc8c1200640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8bc0c5210 0x7fc8bc15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:46.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.269+0000 7fc8c1200640 1 -- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc06ede0 msgr2=0x7fc8bc0c38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:46.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.269+0000 7fc8c1200640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc06ede0 0x7fc8bc0c38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.269+0000 7fc8c1200640 1 -- 172.21.15.6:0/1469288302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8bc0c55f0 msgr2=0x7fc8bc1f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:46.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.269+0000 7fc8c1200640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8bc0c55f0 0x7fc8bc1f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.269+0000 7fc8c1200640 1 -- 172.21.15.6:0/1469288302 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc154b10 con 0x7fc8bc0c5210 2024-03-22T13:02:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.272+0000 7fc8c1200640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8bc0c5210 0x7fc8bc15a490 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc06d4d0 tx=0x7fc8a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:46.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.276+0000 7fc8b9600640 1 -- 172.21.15.6:0/1469288302 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8035040 con 0x7fc8bc0c5210 2024-03-22T13:02:46.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.276+0000 7fc8b9600640 1 -- 172.21.15.6:0/1469288302 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a8003820 con 0x7fc8bc0c5210 2024-03-22T13:02:46.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.276+0000 7fc8b9600640 1 -- 172.21.15.6:0/1469288302 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8003bf0 con 0x7fc8bc0c5210 2024-03-22T13:02:46.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.276+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc0c47a0 con 0x7fc8bc0c5210 2024-03-22T13:02:46.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.276+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc0c4d60 con 0x7fc8bc0c5210 2024-03-22T13:02:46.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.278+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8bc0c79f0 con 0x7fc8bc0c5210 2024-03-22T13:02:46.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.310+0000 7fc8b9600640 1 -- 172.21.15.6:0/1469288302 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc8a8038020 con 0x7fc8bc0c5210 2024-03-22T13:02:46.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.311+0000 7fc8b9600640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc89c06bc20 0x7fc89c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:46.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.312+0000 7fc8c1c00640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc89c06bc20 0x7fc89c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:46.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.312+0000 7fc8c1c00640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc89c06bc20 0x7fc89c06e0e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4007fd0 tx=0x7fc8b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:46.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.315+0000 7fc8b9600640 1 -- 172.21.15.6:0/1469288302 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc8a80793b0 con 0x7fc8bc0c5210 2024-03-22T13:02:46.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.572+0000 7fc8b9600640 1 -- 172.21.15.6:0/1469288302 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc8bc0c79f0 con 0x7fc8bc0c5210 2024-03-22T13:02:46.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.824+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8bc075650 con 0x7fc8bc0c5210 2024-03-22T13:02:46.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.838+0000 7fc8b9600640 1 -- 172.21.15.6:0/1469288302 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fc8a807b310 con 0x7fc8bc0c5210 2024-03-22T13:02:46.838 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:46.839 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:46.839 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:46.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc89c06bc20 msgr2=0x7fc89c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:46.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc89c06bc20 0x7fc89c06e0e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4007fd0 tx=0x7fc8b4009040 comp rx=0 tx=0).stop 2024-03-22T13:02:46.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8bc0c5210 msgr2=0x7fc8bc15a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:46.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8bc0c5210 0x7fc8bc15a490 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc06d4d0 tx=0x7fc8a80046c0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 shutdown_connections 2024-03-22T13:02:46.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8bc0c55f0 0x7fc8bc1f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8bc0c5210 0x7fc8bc15a490 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc89c06bc20 0x7fc89c06e0e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 --2- 172.21.15.6:0/1469288302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8bc06ede0 0x7fc8bc0c38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:46.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 >> 172.21.15.6:0/1469288302 conn(0x7fc8bc0cb890 msgr2=0x7fc8bc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:46.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 shutdown_connections 2024-03-22T13:02:46.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:46.841+0000 7fc8c3000640 1 -- 172.21.15.6:0/1469288302 wait complete. 2024-03-22T13:02:47.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:47.464 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:48.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:48.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.675+0000 7f8fe5c00640 1 Processor -- start 2024-03-22T13:02:48.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.676+0000 7f8fe5c00640 1 -- start start 2024-03-22T13:02:48.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.677+0000 7f8fe5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8fe00cca20 0x7f8fe015e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:48.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.677+0000 7f8fe5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe00ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:48.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.677+0000 7f8fe5c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe006c440 con 0x7f8fe00cc640 2024-03-22T13:02:48.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.677+0000 7f8fe5c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe006d330 con 0x7f8fe00cca20 2024-03-22T13:02:48.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.677+0000 7f8fe5c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe006c720 con 0x7f8fe015b7a0 2024-03-22T13:02:48.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.678+0000 7f8fe5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe00ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.678+0000 7f8fe5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe00ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45022/0 (socket says 172.21.15.6:45022) 2024-03-22T13:02:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.678+0000 7f8fe5200640 1 -- 172.21.15.6:0/3263820312 learned_addr learned my addr 172.21.15.6:0/3263820312 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:48.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.679+0000 7f8fd7e00640 1 --2- 172.21.15.6:0/3263820312 >> v2:172.21.15.6:3300/0 conn(0x7f8fe00cca20 0x7f8fe015e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:48.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.683+0000 7f8fd7e00640 1 -- 172.21.15.6:0/3263820312 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 msgr2=0x7f8fe00ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.683+0000 7f8fd7e00640 1 --2- 172.21.15.6:0/3263820312 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe00ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.683+0000 7f8fd7e00640 1 -- 172.21.15.6:0/3263820312 >> v1:172.21.15.92:6789/0 conn(0x7f8fe00cc640 legacy=0x7f8fe006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.683+0000 7f8fd7e00640 1 -- 172.21.15.6:0/3263820312 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe006ca00 con 0x7f8fe00cca20 2024-03-22T13:02:48.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.685+0000 7f8fd7e00640 1 --2- 172.21.15.6:0/3263820312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe015e260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80097a0 tx=0x7f8fc802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785e9394b0ddabc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:48.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.690+0000 7f8fdf400640 1 -- 172.21.15.6:0/3263820312 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8035020 con 0x7f8fe00cca20 2024-03-22T13:02:48.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.690+0000 7f8fdf400640 1 -- 172.21.15.6:0/3263820312 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc8031830 con 0x7f8fe00cca20 2024-03-22T13:02:48.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.690+0000 7f8fdf400640 1 -- 172.21.15.6:0/3263820312 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8031c00 con 0x7f8fe00cca20 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3263820312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 msgr2=0x7f8fe015e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3263820312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe015e260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80097a0 tx=0x7f8fc802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3263820312 shutdown_connections 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3263820312 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe00ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3263820312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe015e260 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3263820312 >> 172.21.15.6:0/3263820312 conn(0x7f8fe00cf9c0 msgr2=0x7f8fe00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3263820312 shutdown_connections 2024-03-22T13:02:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.691+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3263820312 wait complete. 2024-03-22T13:02:48.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.693+0000 7f8fe5c00640 1 Processor -- start 2024-03-22T13:02:48.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.693+0000 7f8fe5c00640 1 -- start start 2024-03-22T13:02:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.694+0000 7f8fe5c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fe00cc640 0x7f8fe00af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.694+0000 7f8fe5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe00c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.694+0000 7f8fe5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe014ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.694+0000 7f8fe5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe015b000 con 0x7f8fe00cca20 2024-03-22T13:02:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.694+0000 7f8fe5c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe006cfc0 con 0x7f8fe00cc640 2024-03-22T13:02:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.694+0000 7f8fe5c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe015b180 con 0x7f8fe015b7a0 2024-03-22T13:02:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.695+0000 7f8fd7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe00c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:48.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.695+0000 7f8fd7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe00c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54166/0 (socket says 172.21.15.6:54166) 2024-03-22T13:02:48.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.695+0000 7f8fd7e00640 1 -- 172.21.15.6:0/3664233844 learned_addr learned my addr 172.21.15.6:0/3664233844 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.700+0000 7f8fd7e00640 1 -- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 msgr2=0x7f8fe014ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.700+0000 7f8fd7e00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe014ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:48.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.700+0000 7f8fd7e00640 1 -- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fe00cc640 msgr2=0x7f8fe00af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:48.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.700+0000 7f8fd7e00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fe00cc640 0x7f8fe00af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:48.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.701+0000 7f8fd7e00640 1 -- 172.21.15.6:0/3664233844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe01e3d00 con 0x7f8fe00cca20 2024-03-22T13:02:48.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.704+0000 7f8fd7e00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe00c7e00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80098c0 tx=0x7f8fc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:48.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.710+0000 7f8fdcc00640 1 -- 172.21.15.6:0/3664233844 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8035040 con 0x7f8fe00cca20 2024-03-22T13:02:48.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.710+0000 7f8fdcc00640 1 -- 172.21.15.6:0/3664233844 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc803b920 con 0x7f8fe00cca20 2024-03-22T13:02:48.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.710+0000 7f8fdcc00640 1 -- 172.21.15.6:0/3664233844 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc803bd10 con 0x7f8fe00cca20 2024-03-22T13:02:48.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.710+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe00c6d60 con 0x7f8fe00cca20 2024-03-22T13:02:48.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.710+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe00c7320 con 0x7f8fe00cca20 2024-03-22T13:02:48.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.712+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe0075650 con 0x7f8fe00cca20 2024-03-22T13:02:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.748+0000 7f8fdcc00640 1 -- 172.21.15.6:0/3664233844 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8fc8038020 con 0x7f8fe00cca20 2024-03-22T13:02:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.750+0000 7f8fdcc00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8fbc06bbd0 0x7f8fbc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.750+0000 7f8fdfe00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8fbc06bbd0 0x7f8fbc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:48.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:48.751+0000 7f8fdfe00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8fbc06bbd0 0x7f8fbc06e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0001560 tx=0x7f8fd0007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:49.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.041+0000 7f8fdcc00640 1 -- 172.21.15.6:0/3664233844 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8fc8039e40 con 0x7f8fe00cca20 2024-03-22T13:02:49.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.041+0000 7f8fdcc00640 1 -- 172.21.15.6:0/3664233844 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8fc807b9d0 con 0x7f8fe00cca20 2024-03-22T13:02:49.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.288+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fe00af100 con 0x7f8fe00cca20 2024-03-22T13:02:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.310+0000 7f8fdcc00640 1 -- 172.21.15.6:0/3664233844 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f8fc8031800 con 0x7f8fe00cca20 2024-03-22T13:02:49.310 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:49.310 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.312+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8fbc06bbd0 msgr2=0x7f8fbc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.312+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8fbc06bbd0 0x7f8fbc06e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0001560 tx=0x7f8fd0007990 comp rx=0 tx=0).stop 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.312+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 msgr2=0x7f8fe00c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.312+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe00c7e00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80098c0 tx=0x7f8fc80046c0 comp rx=0 tx=0).stop 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 shutdown_connections 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8fe015b7a0 0x7f8fe014ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fe00cca20 0x7f8fe00c7e00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8fbc06bbd0 0x7f8fbc06e090 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 --2- 172.21.15.6:0/3664233844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fe00cc640 0x7f8fe00af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 >> 172.21.15.6:0/3664233844 conn(0x7f8fe00cf9c0 msgr2=0x7f8fe015d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 shutdown_connections 2024-03-22T13:02:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:49.313+0000 7f8fe5c00640 1 -- 172.21.15.6:0/3664233844 wait complete. 2024-03-22T13:02:49.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:49.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:50.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:51.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.135+0000 7f8e4e200640 1 Processor -- start 2024-03-22T13:02:51.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.136+0000 7f8e4e200640 1 -- start start 2024-03-22T13:02:51.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.137+0000 7f8e4e200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8e480ca520 0x7f8e4806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:51.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.137+0000 7f8e4e200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e480ccad0 0x7f8e4815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:51.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.137+0000 7f8e4e200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4806c7a0 con 0x7f8e4815b0e0 2024-03-22T13:02:51.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.137+0000 7f8e4e200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e480cd000 con 0x7f8e480ca520 2024-03-22T13:02:51.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.137+0000 7f8e4e200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4806ca80 con 0x7f8e480ccad0 2024-03-22T13:02:51.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.138+0000 7f8e47e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e480ccad0 0x7f8e4815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:51.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.138+0000 7f8e46a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8e4815b0e0 0x7f8e480cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44290/0 (socket says 172.21.15.6:44290) 2024-03-22T13:02:51.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.138+0000 7f8e46a00640 1 -- 172.21.15.6:0/463965810 learned_addr learned my addr 172.21.15.6:0/463965810 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:51.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.138+0000 7f8e47400640 1 --2- 172.21.15.6:0/463965810 >> v2:172.21.15.92:3300/0 conn(0x7f8e480ca520 0x7f8e4806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:51.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.141+0000 7f8e47e00640 1 -- 172.21.15.6:0/463965810 >> v2:172.21.15.92:3300/0 conn(0x7f8e480ca520 msgr2=0x7f8e4806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:51.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.141+0000 7f8e47e00640 1 --2- 172.21.15.6:0/463965810 >> v2:172.21.15.92:3300/0 conn(0x7f8e480ca520 0x7f8e4806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.141+0000 7f8e47e00640 1 -- 172.21.15.6:0/463965810 >> v1:172.21.15.6:6789/0 conn(0x7f8e4815b0e0 legacy=0x7f8e480cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:51.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.141+0000 7f8e47e00640 1 -- 172.21.15.6:0/463965810 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4806cd60 con 0x7f8e480ccad0 2024-03-22T13:02:51.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.144+0000 7f8e47e00640 1 --2- 172.21.15.6:0/463965810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e480ccad0 0x7f8e4815b980 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097a0 tx=0x7f8e4002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8204ac19f96d5e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:51.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.146+0000 7f8e46000640 1 -- 172.21.15.6:0/463965810 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40035020 con 0x7f8e480ccad0 2024-03-22T13:02:51.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e46000640 1 -- 172.21.15.6:0/463965810 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e40031810 con 0x7f8e480ccad0 2024-03-22T13:02:51.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e46000640 1 -- 172.21.15.6:0/463965810 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40031be0 con 0x7f8e480ccad0 2024-03-22T13:02:51.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e4e200640 1 -- 172.21.15.6:0/463965810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e480ccad0 msgr2=0x7f8e4815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:51.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e4e200640 1 --2- 172.21.15.6:0/463965810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e480ccad0 0x7f8e4815b980 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097a0 tx=0x7f8e4002b8b0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e4e200640 1 -- 172.21.15.6:0/463965810 shutdown_connections 2024-03-22T13:02:51.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e4e200640 1 --2- 172.21.15.6:0/463965810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e480ccad0 0x7f8e4815b980 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e4e200640 1 --2- 172.21.15.6:0/463965810 >> v2:172.21.15.92:3300/0 conn(0x7f8e480ca520 0x7f8e4806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.149+0000 7f8e4e200640 1 -- 172.21.15.6:0/463965810 >> 172.21.15.6:0/463965810 conn(0x7f8e480b16b0 msgr2=0x7f8e480b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:51.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.150+0000 7f8e4e200640 1 -- 172.21.15.6:0/463965810 shutdown_connections 2024-03-22T13:02:51.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.150+0000 7f8e4e200640 1 -- 172.21.15.6:0/463965810 wait complete. 2024-03-22T13:02:51.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.151+0000 7f8e4e200640 1 Processor -- start 2024-03-22T13:02:51.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.151+0000 7f8e4e200640 1 -- start start 2024-03-22T13:02:51.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.152+0000 7f8e4e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e480ca520 0x7f8e48153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:51.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.152+0000 7f8e4e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e480ccad0 0x7f8e48153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:51.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.152+0000 7f8e4e200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e4815b0e0 0x7f8e481587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:51.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.152+0000 7f8e4e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48154c20 con 0x7f8e480ca520 2024-03-22T13:02:51.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.152+0000 7f8e4e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4806d320 con 0x7f8e480ccad0 2024-03-22T13:02:51.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.152+0000 7f8e4e200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480ca900 con 0x7f8e4815b0e0 2024-03-22T13:02:51.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.153+0000 7f8e46a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e480ccad0 0x7f8e48153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:51.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.153+0000 7f8e47e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e4815b0e0 0x7f8e481587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:51.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.153+0000 7f8e47400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e480ca520 0x7f8e48153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:51.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.153+0000 7f8e46a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e480ccad0 0x7f8e48153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52900/0 (socket says 172.21.15.6:52900) 2024-03-22T13:02:51.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.153+0000 7f8e46a00640 1 -- 172.21.15.6:0/1185586372 learned_addr learned my addr 172.21.15.6:0/1185586372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:51.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.158+0000 7f8e46a00640 1 -- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e4815b0e0 msgr2=0x7f8e481587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:51.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.158+0000 7f8e46a00640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e4815b0e0 0x7f8e481587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.158+0000 7f8e46a00640 1 -- 172.21.15.6:0/1185586372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e480ca520 msgr2=0x7f8e48153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:51.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.158+0000 7f8e46a00640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e480ca520 0x7f8e48153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.158+0000 7f8e46a00640 1 -- 172.21.15.6:0/1185586372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4815a3a0 con 0x7f8e480ccad0 2024-03-22T13:02:51.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.165+0000 7f8e46a00640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e480ccad0 0x7f8e48153dc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c3f0 tx=0x7f8e3000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:51.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.170+0000 7f8e3f400640 1 -- 172.21.15.6:0/1185586372 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30017070 con 0x7f8e480ccad0 2024-03-22T13:02:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.170+0000 7f8e3f400640 1 -- 172.21.15.6:0/1185586372 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e30005ce0 con 0x7f8e480ccad0 2024-03-22T13:02:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.170+0000 7f8e3f400640 1 -- 172.21.15.6:0/1185586372 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e300072f0 con 0x7f8e480ccad0 2024-03-22T13:02:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.170+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e480b3630 con 0x7f8e480ccad0 2024-03-22T13:02:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.170+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e480b5000 con 0x7f8e480ccad0 2024-03-22T13:02:51.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.171+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e480b3460 con 0x7f8e480ccad0 2024-03-22T13:02:51.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.202+0000 7f8e3f400640 1 -- 172.21.15.6:0/1185586372 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8e3001a070 con 0x7f8e480ccad0 2024-03-22T13:02:51.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.204+0000 7f8e3f400640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e2406bc20 0x7f8e2406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:51.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.205+0000 7f8e47400640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e2406bc20 0x7f8e2406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:51.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.205+0000 7f8e47400640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e2406bc20 0x7f8e2406e0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8e380113e0 tx=0x7f8e3800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:51.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.206+0000 7f8e3f400640 1 -- 172.21.15.6:0/1185586372 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8e30056700 con 0x7f8e480ccad0 2024-03-22T13:02:51.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.470+0000 7f8e3f400640 1 -- 172.21.15.6:0/1185586372 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8e30021020 con 0x7f8e480ccad0 2024-03-22T13:02:51.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.745+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e480756c0 con 0x7f8e480ccad0 2024-03-22T13:02:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.780+0000 7f8e3f400640 1 -- 172.21.15.6:0/1185586372 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f8e30058660 con 0x7f8e480ccad0 2024-03-22T13:02:51.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:51.780 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.782+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e2406bc20 msgr2=0x7f8e2406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.782+0000 7f8e4e200640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e2406bc20 0x7f8e2406e0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8e380113e0 tx=0x7f8e3800b040 comp rx=0 tx=0).stop 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.782+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e480ccad0 msgr2=0x7f8e48153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.782+0000 7f8e4e200640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e480ccad0 0x7f8e48153dc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000c3f0 tx=0x7f8e3000c8b0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 shutdown_connections 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8e4815b0e0 0x7f8e481587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8e2406bc20 0x7f8e2406e0e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e480ccad0 0x7f8e48153dc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 --2- 172.21.15.6:0/1185586372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e480ca520 0x7f8e48153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 >> 172.21.15.6:0/1185586372 conn(0x7f8e480b16b0 msgr2=0x7f8e480cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 shutdown_connections 2024-03-22T13:02:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:51.783+0000 7f8e4e200640 1 -- 172.21.15.6:0/1185586372 wait complete. 2024-03-22T13:02:52.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:52.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:53.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:53.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.612+0000 7fa9fb200640 1 Processor -- start 2024-03-22T13:02:53.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.612+0000 7fa9fb200640 1 -- start start 2024-03-22T13:02:53.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.613+0000 7fa9fb200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa9f40ce8a0 0x7fa9f406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:53.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.613+0000 7fa9fb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f415b240 con 0x7fa9f40cc380 2024-03-22T13:02:53.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.613+0000 7fa9fb200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f40cee00 con 0x7fa9f415ae30 2024-03-22T13:02:53.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.613+0000 7fa9fb200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f40cf0c0 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.614+0000 7fa9f9400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa9f40ce8a0 0x7fa9f406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:53.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.614+0000 7fa9f9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa9f40cc380 0x7fa9f40cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45878/0 (socket says 172.21.15.6:45878) 2024-03-22T13:02:53.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.614+0000 7fa9f9e00640 1 -- 172.21.15.6:0/1133936412 learned_addr learned my addr 172.21.15.6:0/1133936412 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:53.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.618+0000 7fa9f8a00640 1 -- 172.21.15.6:0/1133936412 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509113440 0 0) 0x7fa9f415b240 con 0x7fa9f40cc380 2024-03-22T13:02:53.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.618+0000 7fa9f8a00640 1 -- 172.21.15.6:0/1133936412 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9dc003520 con 0x7fa9f40cc380 2024-03-22T13:02:53.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.621+0000 7fa9f9400640 1 -- 172.21.15.6:0/1133936412 >> v1:172.21.15.92:6789/0 conn(0x7fa9f415ae30 legacy=0x7fa9f4164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:53.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.621+0000 7fa9f9400640 1 -- 172.21.15.6:0/1133936412 >> v1:172.21.15.6:6789/0 conn(0x7fa9f40cc380 legacy=0x7fa9f40cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:53.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.621+0000 7fa9f9400640 1 -- 172.21.15.6:0/1133936412 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f40cf3a0 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.625+0000 7fa9f9400640 1 --2- 172.21.15.6:0/1133936412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40ce8a0 0x7fa9f406c230 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40097a0 tx=0x7fa9e402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eaf097830733086 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:53.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.630+0000 7fa9f8a00640 1 -- 172.21.15.6:0/1133936412 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4034070 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.630+0000 7fa9f8a00640 1 -- 172.21.15.6:0/1133936412 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e4030610 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.630+0000 7fa9f8a00640 1 -- 172.21.15.6:0/1133936412 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e40309e0 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.631+0000 7fa9fb200640 1 -- 172.21.15.6:0/1133936412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40ce8a0 msgr2=0x7fa9f406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.631+0000 7fa9fb200640 1 --2- 172.21.15.6:0/1133936412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40ce8a0 0x7fa9f406c230 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40097a0 tx=0x7fa9e402b960 comp rx=0 tx=0).stop 2024-03-22T13:02:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.631+0000 7fa9fb200640 1 -- 172.21.15.6:0/1133936412 shutdown_connections 2024-03-22T13:02:53.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.631+0000 7fa9fb200640 1 --2- 172.21.15.6:0/1133936412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40ce8a0 0x7fa9f406c230 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:53.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.631+0000 7fa9fb200640 1 -- 172.21.15.6:0/1133936412 >> 172.21.15.6:0/1133936412 conn(0x7fa9f40b1480 msgr2=0x7fa9f40b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:53.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.631+0000 7fa9fb200640 1 -- 172.21.15.6:0/1133936412 shutdown_connections 2024-03-22T13:02:53.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.632+0000 7fa9fb200640 1 -- 172.21.15.6:0/1133936412 wait complete. 2024-03-22T13:02:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.633+0000 7fa9fb200640 1 Processor -- start 2024-03-22T13:02:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.633+0000 7fa9fb200640 1 -- start start 2024-03-22T13:02:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.634+0000 7fa9fb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40cc380 0x7fa9f41e5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.634+0000 7fa9fb200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9f40ce8a0 0x7fa9f40c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.634+0000 7fa9fb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f415ae30 0x7fa9f40c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.634+0000 7fa9fb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f415a910 con 0x7fa9f415ae30 2024-03-22T13:02:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.634+0000 7fa9fb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f415a630 con 0x7fa9f40cc380 2024-03-22T13:02:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.634+0000 7fa9fb200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f41549d0 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.635+0000 7fa9f9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40cc380 0x7fa9f41e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.635+0000 7fa9f9400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9f40ce8a0 0x7fa9f40c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.635+0000 7fa9fa800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f415ae30 0x7fa9f40c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.635+0000 7fa9f9400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9f40ce8a0 0x7fa9f40c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57916/0 (socket says 172.21.15.6:57916) 2024-03-22T13:02:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.635+0000 7fa9f9400640 1 -- 172.21.15.6:0/3492412714 learned_addr learned my addr 172.21.15.6:0/3492412714 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:53.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.639+0000 7fa9f9400640 1 -- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40cc380 msgr2=0x7fa9f41e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:53.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.639+0000 7fa9f9400640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40cc380 0x7fa9f41e5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:53.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.639+0000 7fa9f9400640 1 -- 172.21.15.6:0/3492412714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f415ae30 msgr2=0x7fa9f40c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:53.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.639+0000 7fa9f9400640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f415ae30 0x7fa9f40c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:53.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.639+0000 7fa9f9400640 1 -- 172.21.15.6:0/3492412714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f40c9600 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.641+0000 7fa9f9400640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9f40ce8a0 0x7fa9f40c6f10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009890 tx=0x7fa9e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.646+0000 7fa9ee000640 1 -- 172.21.15.6:0/3492412714 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4034040 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.646+0000 7fa9ee000640 1 -- 172.21.15.6:0/3492412714 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e4039990 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.646+0000 7fa9ee000640 1 -- 172.21.15.6:0/3492412714 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4039d60 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.646+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f40c98e0 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.646+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f415c6f0 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.647+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f4149600 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.936+0000 7fa9ee000640 1 -- 172.21.15.6:0/3492412714 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa9e4037070 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.939+0000 7fa9ee000640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa9d806bc20 0x7fa9d806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.939+0000 7fa9f9e00640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa9d806bc20 0x7fa9d806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:53.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.940+0000 7fa9f9e00640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa9d806bc20 0x7fa9d806e0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8005880 tx=0x7fa9e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:53.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.942+0000 7fa9ee000640 1 -- 172.21.15.6:0/3492412714 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa9e4079820 con 0x7fa9f40ce8a0 2024-03-22T13:02:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:53.961+0000 7fa9ee000640 1 -- 172.21.15.6:0/3492412714 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa9e407b780 con 0x7fa9f40ce8a0 2024-03-22T13:02:54.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.219+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9f415c220 con 0x7fa9f40ce8a0 2024-03-22T13:02:54.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.234+0000 7fa9ee000640 1 -- 172.21.15.6:0/3492412714 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fa9e407f820 con 0x7fa9f40ce8a0 2024-03-22T13:02:54.234 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:54.234 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:54.235 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.236+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa9d806bc20 msgr2=0x7fa9d806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.236+0000 7fa9fb200640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa9d806bc20 0x7fa9d806e0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8005880 tx=0x7fa9e8009040 comp rx=0 tx=0).stop 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.236+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9f40ce8a0 msgr2=0x7fa9f40c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.236+0000 7fa9fb200640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9f40ce8a0 0x7fa9f40c6f10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009890 tx=0x7fa9e40046c0 comp rx=0 tx=0).stop 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 shutdown_connections 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9f415ae30 0x7fa9f40c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9f40ce8a0 0x7fa9f40c6f10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa9d806bc20 0x7fa9d806e0e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 --2- 172.21.15.6:0/3492412714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9f40cc380 0x7fa9f41e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 >> 172.21.15.6:0/3492412714 conn(0x7fa9f40b1480 msgr2=0x7fa9f40b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 shutdown_connections 2024-03-22T13:02:54.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:54.237+0000 7fa9fb200640 1 -- 172.21.15.6:0/3492412714 wait complete. 2024-03-22T13:02:54.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:54.861 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:55.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:56.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.050+0000 7efd33200640 1 Processor -- start 2024-03-22T13:02:56.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.050+0000 7efd33200640 1 -- start start 2024-03-22T13:02:56.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.052+0000 7efd33200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c0c7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.052+0000 7efd33200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c152930 con 0x7efd2c152550 2024-03-22T13:02:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.052+0000 7efd33200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c0c7cb0 con 0x7efd2c0cd670 2024-03-22T13:02:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.052+0000 7efd33200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c0c7a90 con 0x7efd2c072750 2024-03-22T13:02:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.053+0000 7efd2b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c0c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.053+0000 7efd31200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efd2c0cd670 0x7efd2c072c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60442/0 (socket says 172.21.15.6:60442) 2024-03-22T13:02:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.053+0000 7efd31200640 1 -- 172.21.15.6:0/1201729207 learned_addr learned my addr 172.21.15.6:0/1201729207 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:02:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.056+0000 7efd2aa00640 1 -- 172.21.15.6:0/1201729207 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856088318 0 0) 0x7efd2c152930 con 0x7efd2c152550 2024-03-22T13:02:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.056+0000 7efd2aa00640 1 -- 172.21.15.6:0/1201729207 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd10003520 con 0x7efd2c152550 2024-03-22T13:02:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.056+0000 7efd2b400640 1 -- 172.21.15.6:0/1201729207 >> v1:172.21.15.92:6789/0 conn(0x7efd2c0cd670 legacy=0x7efd2c072c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.056+0000 7efd2b400640 1 -- 172.21.15.6:0/1201729207 >> v1:172.21.15.6:6789/0 conn(0x7efd2c152550 legacy=0x7efd2c1581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.056+0000 7efd2b400640 1 -- 172.21.15.6:0/1201729207 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c15b7e0 con 0x7efd2c072750 2024-03-22T13:02:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.059+0000 7efd2b400640 1 --2- 172.21.15.6:0/1201729207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c0c7550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efd140097d0 tx=0x7efd1402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=658bfd897d3656a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:56.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.062+0000 7efd2aa00640 1 -- 172.21.15.6:0/1201729207 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14035020 con 0x7efd2c072750 2024-03-22T13:02:56.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.065+0000 7efd2aa00640 1 -- 172.21.15.6:0/1201729207 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd14031800 con 0x7efd2c072750 2024-03-22T13:02:56.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.065+0000 7efd2aa00640 1 -- 172.21.15.6:0/1201729207 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14031bb0 con 0x7efd2c072750 2024-03-22T13:02:56.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.065+0000 7efd33200640 1 -- 172.21.15.6:0/1201729207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 msgr2=0x7efd2c0c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:56.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.065+0000 7efd33200640 1 --2- 172.21.15.6:0/1201729207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c0c7550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efd140097d0 tx=0x7efd1402b8a0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.066+0000 7efd33200640 1 -- 172.21.15.6:0/1201729207 shutdown_connections 2024-03-22T13:02:56.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.066+0000 7efd33200640 1 --2- 172.21.15.6:0/1201729207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c0c7550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.066+0000 7efd33200640 1 -- 172.21.15.6:0/1201729207 >> 172.21.15.6:0/1201729207 conn(0x7efd2c0ccb50 msgr2=0x7efd2c0ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:56.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.066+0000 7efd33200640 1 -- 172.21.15.6:0/1201729207 shutdown_connections 2024-03-22T13:02:56.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.066+0000 7efd33200640 1 -- 172.21.15.6:0/1201729207 wait complete. 2024-03-22T13:02:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.068+0000 7efd33200640 1 Processor -- start 2024-03-22T13:02:56.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.068+0000 7efd33200640 1 -- start start 2024-03-22T13:02:56.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd33200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c14ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd33200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c0cd670 0x7efd2c1509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd33200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c152550 0x7efd2c1f10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd33200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c1588e0 con 0x7efd2c0cd670 2024-03-22T13:02:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd33200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c15bda0 con 0x7efd2c152550 2024-03-22T13:02:56.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd33200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c15c5a0 con 0x7efd2c072750 2024-03-22T13:02:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd2b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c0cd670 0x7efd2c1509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.069+0000 7efd2be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c14ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.070+0000 7efd31200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c152550 0x7efd2c1f10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.070+0000 7efd2b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c0cd670 0x7efd2c1509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37224/0 (socket says 172.21.15.6:37224) 2024-03-22T13:02:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.070+0000 7efd2b400640 1 -- 172.21.15.6:0/2962670191 learned_addr learned my addr 172.21.15.6:0/2962670191 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:56.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.074+0000 7efd2b400640 1 -- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 msgr2=0x7efd2c14ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:56.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.074+0000 7efd2b400640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c14ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.074+0000 7efd2b400640 1 -- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c152550 msgr2=0x7efd2c1f10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:56.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.074+0000 7efd2b400640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c152550 0x7efd2c1f10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.074+0000 7efd2b400640 1 -- 172.21.15.6:0/2962670191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c15a6e0 con 0x7efd2c0cd670 2024-03-22T13:02:56.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.076+0000 7efd2b400640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c0cd670 0x7efd2c1509d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd14034040 tx=0x7efd140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:56.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.080+0000 7efd23e00640 1 -- 172.21.15.6:0/2962670191 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14035050 con 0x7efd2c0cd670 2024-03-22T13:02:56.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.081+0000 7efd23e00640 1 -- 172.21.15.6:0/2962670191 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd14003820 con 0x7efd2c0cd670 2024-03-22T13:02:56.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.081+0000 7efd23e00640 1 -- 172.21.15.6:0/2962670191 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14003bf0 con 0x7efd2c0cd670 2024-03-22T13:02:56.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.081+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2c14f730 con 0x7efd2c0cd670 2024-03-22T13:02:56.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.081+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2c157020 con 0x7efd2c0cd670 2024-03-22T13:02:56.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.082+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2c0756c0 con 0x7efd2c0cd670 2024-03-22T13:02:56.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.104+0000 7efd23e00640 1 -- 172.21.15.6:0/2962670191 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd14038020 con 0x7efd2c0cd670 2024-03-22T13:02:56.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.107+0000 7efd23e00640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.107+0000 7efd2be00640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.108+0000 7efd2be00640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd1c001e00 tx=0x7efd1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.427+0000 7efd23e00640 1 -- 172.21.15.6:0/2962670191 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7efd140790b0 con 0x7efd2c0cd670 2024-03-22T13:02:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.428+0000 7efd23e00640 1 -- 172.21.15.6:0/2962670191 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd1407b010 con 0x7efd2c0cd670 2024-03-22T13:02:56.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.692+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd2c0c7490 con 0x7efd2c0cd670 2024-03-22T13:02:56.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.716+0000 7efd23e00640 1 -- 172.21.15.6:0/2962670191 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7efd1407f0b0 con 0x7efd2c0cd670 2024-03-22T13:02:56.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:56.716 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:56.716 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.718+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 msgr2=0x7efd0806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.718+0000 7efd33200640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd1c001e00 tx=0x7efd1c009040 comp rx=0 tx=0).stop 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.718+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c0cd670 msgr2=0x7efd2c1509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.718+0000 7efd33200640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c0cd670 0x7efd2c1509d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd14034040 tx=0x7efd140046c0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 shutdown_connections 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd2c152550 0x7efd2c1f10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd2c0cd670 0x7efd2c1509d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 --2- 172.21.15.6:0/2962670191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd2c072750 0x7efd2c14ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 >> 172.21.15.6:0/2962670191 conn(0x7efd2c0ccb50 msgr2=0x7efd2c0c94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 shutdown_connections 2024-03-22T13:02:56.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:56.719+0000 7efd33200640 1 -- 172.21.15.6:0/2962670191 wait complete. 2024-03-22T13:02:57.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:57.339 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:02:58.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:02:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.548+0000 7ff48f200640 1 Processor -- start 2024-03-22T13:02:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.548+0000 7ff48f200640 1 -- start start 2024-03-22T13:02:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.549+0000 7ff48f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4900cc380 0x7ff49006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.549+0000 7ff48f200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4900ce8a0 0x7ff49015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.549+0000 7ff48f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900cc760 0x7ff4900cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.549+0000 7ff48f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4900cee00 con 0x7ff4900cc380 2024-03-22T13:02:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.549+0000 7ff48f200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4900cf0c0 con 0x7ff4900ce8a0 2024-03-22T13:02:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.549+0000 7ff48f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900cf3a0 con 0x7ff4900cc760 2024-03-22T13:02:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.550+0000 7ff48d400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4900ce8a0 0x7ff49015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.550+0000 7ff48e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900cc760 0x7ff4900cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.550+0000 7ff48d400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4900ce8a0 0x7ff49015ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:38716/0 (socket says 172.21.15.6:38716) 2024-03-22T13:02:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.551+0000 7ff48d400640 1 -- 172.21.15.6:0/2389690234 learned_addr learned my addr 172.21.15.6:0/2389690234 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.553+0000 7ff48d400640 1 -- 172.21.15.6:0/2389690234 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900cc760 msgr2=0x7ff4900cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.553+0000 7ff48d400640 1 --2- 172.21.15.6:0/2389690234 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900cc760 0x7ff4900cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.553+0000 7ff48d400640 1 -- 172.21.15.6:0/2389690234 >> v2:172.21.15.6:3300/0 conn(0x7ff4900cc380 msgr2=0x7ff49006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:02:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.553+0000 7ff48d400640 1 --2- 172.21.15.6:0/2389690234 >> v2:172.21.15.6:3300/0 conn(0x7ff4900cc380 0x7ff49006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.553+0000 7ff48d400640 1 -- 172.21.15.6:0/2389690234 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4900cf680 con 0x7ff4900ce8a0 2024-03-22T13:02:58.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.556+0000 7ff48d400640 1 --2- 172.21.15.6:0/2389690234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900ce8a0 0x7ff49015ae30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff4900cb7f0 tx=0x7ff47c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b5d33a77a61ff6a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.560+0000 7ff48ca00640 1 -- 172.21.15.6:0/2389690234 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c035020 con 0x7ff4900ce8a0 2024-03-22T13:02:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.560+0000 7ff48ca00640 1 -- 172.21.15.6:0/2389690234 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c031870 con 0x7ff4900ce8a0 2024-03-22T13:02:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.560+0000 7ff48ca00640 1 -- 172.21.15.6:0/2389690234 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c031c40 con 0x7ff4900ce8a0 2024-03-22T13:02:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 -- 172.21.15.6:0/2389690234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900ce8a0 msgr2=0x7ff49015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 --2- 172.21.15.6:0/2389690234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900ce8a0 0x7ff49015ae30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff4900cb7f0 tx=0x7ff47c02b910 comp rx=0 tx=0).stop 2024-03-22T13:02:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 -- 172.21.15.6:0/2389690234 shutdown_connections 2024-03-22T13:02:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 --2- 172.21.15.6:0/2389690234 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900cc760 0x7ff4900cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 --2- 172.21.15.6:0/2389690234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900ce8a0 0x7ff49015ae30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 --2- 172.21.15.6:0/2389690234 >> v2:172.21.15.6:3300/0 conn(0x7ff4900cc380 0x7ff49006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 -- 172.21.15.6:0/2389690234 >> 172.21.15.6:0/2389690234 conn(0x7ff4900b1480 msgr2=0x7ff4900b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 -- 172.21.15.6:0/2389690234 shutdown_connections 2024-03-22T13:02:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.561+0000 7ff48f200640 1 -- 172.21.15.6:0/2389690234 wait complete. 2024-03-22T13:02:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.563+0000 7ff48f200640 1 Processor -- start 2024-03-22T13:02:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.563+0000 7ff48f200640 1 -- start start 2024-03-22T13:02:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.564+0000 7ff48f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4900cc380 0x7ff49006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.564+0000 7ff48f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900cc760 0x7ff4901e5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.564+0000 7ff48f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900ce8a0 0x7ff4901e5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.564+0000 7ff48f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4901549d0 con 0x7ff4900cc380 2024-03-22T13:02:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.564+0000 7ff48f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49015a910 con 0x7ff4900cc760 2024-03-22T13:02:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.564+0000 7ff48f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900ccb90 con 0x7ff4900ce8a0 2024-03-22T13:02:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.565+0000 7ff48d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900cc760 0x7ff4901e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.565+0000 7ff48e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900ce8a0 0x7ff4901e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.565+0000 7ff48d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900cc760 0x7ff4901e5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:38732/0 (socket says 172.21.15.6:38732) 2024-03-22T13:02:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.565+0000 7ff48d400640 1 -- 172.21.15.6:0/4168434089 learned_addr learned my addr 172.21.15.6:0/4168434089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:02:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.569+0000 7ff48e800640 1 -- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900cc760 msgr2=0x7ff4901e5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.569+0000 7ff48e800640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900cc760 0x7ff4901e5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.569+0000 7ff48e800640 1 -- 172.21.15.6:0/4168434089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4900cc380 msgr2=0x7ff49006cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.569+0000 7ff48e800640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4900cc380 0x7ff49006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.569+0000 7ff48e800640 1 -- 172.21.15.6:0/4168434089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4901e4cf0 con 0x7ff4900ce8a0 2024-03-22T13:02:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.572+0000 7ff48e800640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900ce8a0 0x7ff4901e5650 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff48800eba0 tx=0x7ff48800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.577+0000 7ff486000640 1 -- 172.21.15.6:0/4168434089 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488018020 con 0x7ff4900ce8a0 2024-03-22T13:02:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.577+0000 7ff486000640 1 -- 172.21.15.6:0/4168434089 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48800cd20 con 0x7ff4900ce8a0 2024-03-22T13:02:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.577+0000 7ff486000640 1 -- 172.21.15.6:0/4168434089 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4880146c0 con 0x7ff4900ce8a0 2024-03-22T13:02:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.577+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4901e3880 con 0x7ff4900ce8a0 2024-03-22T13:02:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.577+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff490145570 con 0x7ff4900ce8a0 2024-03-22T13:02:58.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.579+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff490075650 con 0x7ff4900ce8a0 2024-03-22T13:02:58.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.664+0000 7ff486000640 1 -- 172.21.15.6:0/4168434089 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff488005d10 con 0x7ff4900ce8a0 2024-03-22T13:02:58.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.666+0000 7ff486000640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff47006bc50 0x7ff47006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:02:58.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.667+0000 7ff48de00640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff47006bc50 0x7ff47006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:02:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.668+0000 7ff48de00640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff47006bc50 0x7ff47006e110 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff480009670 tx=0x7ff480006bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:02:58.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.912+0000 7ff486000640 1 -- 172.21.15.6:0/4168434089 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff4880580c0 con 0x7ff4900ce8a0 2024-03-22T13:02:58.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:58.931+0000 7ff486000640 1 -- 172.21.15.6:0/4168434089 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff488022030 con 0x7ff4900ce8a0 2024-03-22T13:02:59.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.186+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff49006de60 con 0x7ff4900ce8a0 2024-03-22T13:02:59.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.209+0000 7ff486000640 1 -- 172.21.15.6:0/4168434089 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7ff488010040 con 0x7ff4900ce8a0 2024-03-22T13:02:59.209 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:02:59.209 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:02:59.209 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:02:59.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff47006bc50 msgr2=0x7ff47006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:59.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff47006bc50 0x7ff47006e110 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff480009670 tx=0x7ff480006bf0 comp rx=0 tx=0).stop 2024-03-22T13:02:59.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900ce8a0 msgr2=0x7ff4901e5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:02:59.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900ce8a0 0x7ff4901e5650 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff48800eba0 tx=0x7ff48800c490 comp rx=0 tx=0).stop 2024-03-22T13:02:59.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 shutdown_connections 2024-03-22T13:02:59.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4900ce8a0 0x7ff4901e5650 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:59.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff47006bc50 0x7ff47006e110 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:59.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4900cc760 0x7ff4901e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:59.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 --2- 172.21.15.6:0/4168434089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4900cc380 0x7ff49006cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:02:59.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 >> 172.21.15.6:0/4168434089 conn(0x7ff4900b1480 msgr2=0x7ff4900c21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:02:59.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 shutdown_connections 2024-03-22T13:02:59.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:02:59.212+0000 7ff48f200640 1 -- 172.21.15.6:0/4168434089 wait complete. 2024-03-22T13:02:59.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:02:59.835 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:00.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:01.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:03:01.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:03:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.041+0000 7f8142000640 1 Processor -- start 2024-03-22T13:03:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.041+0000 7f8142000640 1 -- start start 2024-03-22T13:03:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.042+0000 7f8142000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f813c0cc380 0x7f813c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.042+0000 7f8142000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c15ae30 0x7f813c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.043+0000 7f8142000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c15b260 con 0x7f813c0ce8a0 2024-03-22T13:03:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.043+0000 7f8142000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c0cee00 con 0x7f813c0cc380 2024-03-22T13:03:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.043+0000 7f8142000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0cf0c0 con 0x7f813c15ae30 2024-03-22T13:03:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.043+0000 7f8140c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f813c0cc380 0x7f813c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.043+0000 7f8141600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c15ae30 0x7f813c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.044+0000 7f813be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f813c0ce8a0 0x7f813c0cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60456/0 (socket says 172.21.15.6:60456) 2024-03-22T13:03:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.044+0000 7f813be00640 1 -- 172.21.15.6:0/2825933940 learned_addr learned my addr 172.21.15.6:0/2825933940 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.047+0000 7f8140c00640 1 -- 172.21.15.6:0/2825933940 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c15ae30 msgr2=0x7f813c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.047+0000 7f8140c00640 1 --2- 172.21.15.6:0/2825933940 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c15ae30 0x7f813c0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.047+0000 7f8140c00640 1 -- 172.21.15.6:0/2825933940 >> v1:172.21.15.92:6789/0 conn(0x7f813c0ce8a0 legacy=0x7f813c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.047+0000 7f8140c00640 1 -- 172.21.15.6:0/2825933940 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c0cf3a0 con 0x7f813c0cc380 2024-03-22T13:03:01.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.050+0000 7f8140c00640 1 --2- 172.21.15.6:0/2825933940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c0cc380 0x7f813c06c230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81300097a0 tx=0x7f813002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2998ce37b7b2b082 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:01.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.056+0000 7f813b400640 1 -- 172.21.15.6:0/2825933940 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130035020 con 0x7f813c0cc380 2024-03-22T13:03:01.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.056+0000 7f813b400640 1 -- 172.21.15.6:0/2825933940 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8130031810 con 0x7f813c0cc380 2024-03-22T13:03:01.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.056+0000 7f813b400640 1 -- 172.21.15.6:0/2825933940 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130031be0 con 0x7f813c0cc380 2024-03-22T13:03:01.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 -- 172.21.15.6:0/2825933940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c0cc380 msgr2=0x7f813c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:01.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 --2- 172.21.15.6:0/2825933940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c0cc380 0x7f813c06c230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81300097a0 tx=0x7f813002b8b0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 -- 172.21.15.6:0/2825933940 shutdown_connections 2024-03-22T13:03:01.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 --2- 172.21.15.6:0/2825933940 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c15ae30 0x7f813c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 --2- 172.21.15.6:0/2825933940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c0cc380 0x7f813c06c230 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 -- 172.21.15.6:0/2825933940 >> 172.21.15.6:0/2825933940 conn(0x7f813c0b1480 msgr2=0x7f813c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:01.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 -- 172.21.15.6:0/2825933940 shutdown_connections 2024-03-22T13:03:01.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.057+0000 7f8142000640 1 -- 172.21.15.6:0/2825933940 wait complete. 2024-03-22T13:03:01.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.060+0000 7f8142000640 1 Processor -- start 2024-03-22T13:03:01.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.060+0000 7f8142000640 1 -- start start 2024-03-22T13:03:01.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.061+0000 7f8142000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c0cc380 0x7f813c0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:01.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.061+0000 7f8142000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813c0ce8a0 0x7f813c15be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:01.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.061+0000 7f8142000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c15ae30 0x7f813c0b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:01.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.061+0000 7f8142000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c15a910 con 0x7f813c15ae30 2024-03-22T13:03:01.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.061+0000 7f8142000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c15a630 con 0x7f813c0ce8a0 2024-03-22T13:03:01.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.061+0000 7f8142000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1549d0 con 0x7f813c0cc380 2024-03-22T13:03:01.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.062+0000 7f8140c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c0cc380 0x7f813c0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:01.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.062+0000 7f8141600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c15ae30 0x7f813c0b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:01.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.062+0000 7f813be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813c0ce8a0 0x7f813c15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:01.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.063+0000 7f8140c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c0cc380 0x7f813c0c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57956/0 (socket says 172.21.15.6:57956) 2024-03-22T13:03:01.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.063+0000 7f8140c00640 1 -- 172.21.15.6:0/1555570699 learned_addr learned my addr 172.21.15.6:0/1555570699 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:01.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.066+0000 7f813be00640 1 -- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c0cc380 msgr2=0x7f813c0c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:01.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.066+0000 7f813be00640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c0cc380 0x7f813c0c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.066+0000 7f813be00640 1 -- 172.21.15.6:0/1555570699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c15ae30 msgr2=0x7f813c0b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:01.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.066+0000 7f813be00640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c15ae30 0x7f813c0b0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.067+0000 7f813be00640 1 -- 172.21.15.6:0/1555570699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c15d250 con 0x7f813c0ce8a0 2024-03-22T13:03:01.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.071+0000 7f813be00640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813c0ce8a0 0x7f813c15be50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8128000f60 tx=0x7f812800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:01.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.076+0000 7f8138c00640 1 -- 172.21.15.6:0/1555570699 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128017070 con 0x7f813c0ce8a0 2024-03-22T13:03:01.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.076+0000 7f8138c00640 1 -- 172.21.15.6:0/1555570699 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8128005ce0 con 0x7f813c0ce8a0 2024-03-22T13:03:01.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.076+0000 7f8138c00640 1 -- 172.21.15.6:0/1555570699 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128007310 con 0x7f813c0ce8a0 2024-03-22T13:03:01.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.076+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c1e35b0 con 0x7f813c0ce8a0 2024-03-22T13:03:01.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.076+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c15d7a0 con 0x7f813c0ce8a0 2024-03-22T13:03:01.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.078+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c075650 con 0x7f813c0ce8a0 2024-03-22T13:03:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.099+0000 7f8138c00640 1 -- 172.21.15.6:0/1555570699 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f812801a070 con 0x7f813c0ce8a0 2024-03-22T13:03:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.099+0000 7f8138c00640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f811c06bc20 0x7f811c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.100+0000 7f8140c00640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f811c06bc20 0x7f811c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:01.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.101+0000 7f8140c00640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f811c06bc20 0x7f811c06e0e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f813002b8f0 tx=0x7f813003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.341+0000 7f8138c00640 1 -- 172.21.15.6:0/1555570699 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8128056730 con 0x7f813c0ce8a0 2024-03-22T13:03:01.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.353+0000 7f8138c00640 1 -- 172.21.15.6:0/1555570699 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8128021020 con 0x7f813c0ce8a0 2024-03-22T13:03:01.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.631+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f813c15cf90 con 0x7f813c0ce8a0 2024-03-22T13:03:01.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.664+0000 7f8138c00640 1 -- 172.21.15.6:0/1555570699 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f813c15b260 con 0x7f813c0ce8a0 2024-03-22T13:03:01.664 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:01.664 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:01.665 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f811c06bc20 msgr2=0x7f811c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f811c06bc20 0x7f811c06e0e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f813002b8f0 tx=0x7f813003c040 comp rx=0 tx=0).stop 2024-03-22T13:03:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813c0ce8a0 msgr2=0x7f813c15be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813c0ce8a0 0x7f813c15be50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8128000f60 tx=0x7f812800c800 comp rx=0 tx=0).stop 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 shutdown_connections 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f813c15ae30 0x7f813c0b0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f811c06bc20 0x7f811c06e0e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f813c0ce8a0 0x7f813c15be50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 --2- 172.21.15.6:0/1555570699 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f813c0cc380 0x7f813c0c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 >> 172.21.15.6:0/1555570699 conn(0x7f813c0b1480 msgr2=0x7f813c158e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 shutdown_connections 2024-03-22T13:03:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:01.668+0000 7f8142000640 1 -- 172.21.15.6:0/1555570699 wait complete. 2024-03-22T13:03:02.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:02.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:03.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.502+0000 7f39d7200640 1 Processor -- start 2024-03-22T13:03:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.502+0000 7f39d7200640 1 -- start start 2024-03-22T13:03:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.503+0000 7f39d7200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00ce8a0 0x7f39d006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.503+0000 7f39d7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f39d015ae30 0x7f39d00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.503+0000 7f39d7200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d015b260 con 0x7f39d00cc380 2024-03-22T13:03:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.504+0000 7f39d7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d00cee00 con 0x7f39d015ae30 2024-03-22T13:03:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.504+0000 7f39d7200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00cf0c0 con 0x7f39d00ce8a0 2024-03-22T13:03:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.504+0000 7f39d5400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00ce8a0 0x7f39d006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.504+0000 7f39d5e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f39d00cc380 0x7f39d00cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53092/0 (socket says 172.21.15.6:53092) 2024-03-22T13:03:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.504+0000 7f39d5e00640 1 -- 172.21.15.6:0/1834633837 learned_addr learned my addr 172.21.15.6:0/1834633837 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.505+0000 7f39d6800640 1 --2- 172.21.15.6:0/1834633837 >> v2:172.21.15.6:3300/0 conn(0x7f39d015ae30 0x7f39d00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.508+0000 7f39d4a00640 1 -- 172.21.15.6:0/1834633837 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46948861 0 0) 0x7f39c4001110 con 0x7f39d00cc380 2024-03-22T13:03:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.508+0000 7f39d4a00640 1 -- 172.21.15.6:0/1834633837 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b8003520 con 0x7f39d00cc380 2024-03-22T13:03:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.508+0000 7f39d5400640 1 -- 172.21.15.6:0/1834633837 >> v2:172.21.15.6:3300/0 conn(0x7f39d015ae30 msgr2=0x7f39d00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.508+0000 7f39d5400640 1 --2- 172.21.15.6:0/1834633837 >> v2:172.21.15.6:3300/0 conn(0x7f39d015ae30 0x7f39d00cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.508+0000 7f39d5400640 1 -- 172.21.15.6:0/1834633837 >> v1:172.21.15.92:6789/0 conn(0x7f39d00cc380 legacy=0x7f39d00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.508+0000 7f39d5400640 1 -- 172.21.15.6:0/1834633837 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00cf3a0 con 0x7f39d00ce8a0 2024-03-22T13:03:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.511+0000 7f39d5400640 1 --2- 172.21.15.6:0/1834633837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00ce8a0 0x7f39d006c230 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39c00097a0 tx=0x7f39c002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56cb1fc087f80ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d4a00640 1 -- 172.21.15.6:0/1834633837 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0035020 con 0x7f39d00ce8a0 2024-03-22T13:03:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d4a00640 1 -- 172.21.15.6:0/1834633837 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c0031830 con 0x7f39d00ce8a0 2024-03-22T13:03:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d4a00640 1 -- 172.21.15.6:0/1834633837 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0031c00 con 0x7f39d00ce8a0 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d7200640 1 -- 172.21.15.6:0/1834633837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00ce8a0 msgr2=0x7f39d006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d7200640 1 --2- 172.21.15.6:0/1834633837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00ce8a0 0x7f39d006c230 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39c00097a0 tx=0x7f39c002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d7200640 1 -- 172.21.15.6:0/1834633837 shutdown_connections 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d7200640 1 --2- 172.21.15.6:0/1834633837 >> v2:172.21.15.6:3300/0 conn(0x7f39d015ae30 0x7f39d00cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d7200640 1 --2- 172.21.15.6:0/1834633837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00ce8a0 0x7f39d006c230 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.516+0000 7f39d7200640 1 -- 172.21.15.6:0/1834633837 >> 172.21.15.6:0/1834633837 conn(0x7f39d00b1480 msgr2=0x7f39d00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.517+0000 7f39d7200640 1 -- 172.21.15.6:0/1834633837 shutdown_connections 2024-03-22T13:03:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.517+0000 7f39d7200640 1 -- 172.21.15.6:0/1834633837 wait complete. 2024-03-22T13:03:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.518+0000 7f39d7200640 1 Processor -- start 2024-03-22T13:03:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.518+0000 7f39d7200640 1 -- start start 2024-03-22T13:03:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.519+0000 7f39d7200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00cc380 0x7f39d015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.519+0000 7f39d7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39d00ce8a0 0x7f39d01e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.519+0000 7f39d7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d015ae30 0x7f39d01e5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.519+0000 7f39d7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d015a910 con 0x7f39d015ae30 2024-03-22T13:03:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.519+0000 7f39d7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d015a630 con 0x7f39d00ce8a0 2024-03-22T13:03:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.519+0000 7f39d7200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d01549d0 con 0x7f39d00cc380 2024-03-22T13:03:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.520+0000 7f39d5e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00cc380 0x7f39d015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.520+0000 7f39d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39d00ce8a0 0x7f39d01e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.520+0000 7f39d6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d015ae30 0x7f39d01e5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.520+0000 7f39d5e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00cc380 0x7f39d015dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53280/0 (socket says 172.21.15.6:53280) 2024-03-22T13:03:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.520+0000 7f39d5e00640 1 -- 172.21.15.6:0/2683235960 learned_addr learned my addr 172.21.15.6:0/2683235960 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.525+0000 7f39d5e00640 1 -- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39d00ce8a0 msgr2=0x7f39d01e5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.525+0000 7f39d5e00640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39d00ce8a0 0x7f39d01e5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.525+0000 7f39d5e00640 1 -- 172.21.15.6:0/2683235960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d015ae30 msgr2=0x7f39d01e5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.525+0000 7f39d5e00640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d015ae30 0x7f39d01e5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:03.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.525+0000 7f39d5e00640 1 -- 172.21.15.6:0/2683235960 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d015bb20 con 0x7f39d00cc380 2024-03-22T13:03:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.528+0000 7f39d5e00640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00cc380 0x7f39d015dfe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f39c400c3d0 tx=0x7f39c400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.533+0000 7f39ca000640 1 -- 172.21.15.6:0/2683235960 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4017070 con 0x7f39d00cc380 2024-03-22T13:03:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.533+0000 7f39ca000640 1 -- 172.21.15.6:0/2683235960 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c4005ce0 con 0x7f39d00cc380 2024-03-22T13:03:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.533+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d015be00 con 0x7f39d00cc380 2024-03-22T13:03:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.533+0000 7f39ca000640 1 -- 172.21.15.6:0/2683235960 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4007310 con 0x7f39d00cc380 2024-03-22T13:03:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.533+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d015c3c0 con 0x7f39d00cc380 2024-03-22T13:03:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.534+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d0075650 con 0x7f39d00cc380 2024-03-22T13:03:03.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.566+0000 7f39ca000640 1 -- 172.21.15.6:0/2683235960 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f39c401a070 con 0x7f39d00cc380 2024-03-22T13:03:03.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.567+0000 7f39ca000640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f39b406bc20 0x7f39b406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:03.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.568+0000 7f39d5400640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f39b406bc20 0x7f39b406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:03.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.569+0000 7f39d5400640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f39b406bc20 0x7f39b406e0e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f39c0036000 tx=0x7f39c0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.810+0000 7f39ca000640 1 -- 172.21.15.6:0/2683235960 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f39c40567e0 con 0x7f39d00cc380 2024-03-22T13:03:03.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:03.831+0000 7f39ca000640 1 -- 172.21.15.6:0/2683235960 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f39c4021020 con 0x7f39d00cc380 2024-03-22T13:03:04.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.105+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39d00af180 con 0x7f39d00cc380 2024-03-22T13:03:04.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.118+0000 7f39ca000640 1 -- 172.21.15.6:0/2683235960 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f39d015b260 con 0x7f39d00cc380 2024-03-22T13:03:04.119 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:04.119 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:04.119 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f39b406bc20 msgr2=0x7f39b406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f39b406bc20 0x7f39b406e0e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f39c0036000 tx=0x7f39c0008030 comp rx=0 tx=0).stop 2024-03-22T13:03:04.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00cc380 msgr2=0x7f39d015dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00cc380 0x7f39d015dfe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f39c400c3d0 tx=0x7f39c400c890 comp rx=0 tx=0).stop 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 shutdown_connections 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d015ae30 0x7f39d01e5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f39b406bc20 0x7f39b406e0e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39d00ce8a0 0x7f39d01e5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 --2- 172.21.15.6:0/2683235960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39d00cc380 0x7f39d015dfe0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 >> 172.21.15.6:0/2683235960 conn(0x7f39d00b1480 msgr2=0x7f39d00cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 shutdown_connections 2024-03-22T13:03:04.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:04.121+0000 7f39d7200640 1 -- 172.21.15.6:0/2683235960 wait complete. 2024-03-22T13:03:04.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:04.743 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:05.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.954+0000 7f750e200640 1 Processor -- start 2024-03-22T13:03:05.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.954+0000 7f750e200640 1 -- start start 2024-03-22T13:03:05.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.955+0000 7f750e200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75080ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.955+0000 7f750e200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75080cdfd0 0x7f75080ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.955+0000 7f750e200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75080ce9a0 con 0x7f75080cc5f0 2024-03-22T13:03:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.955+0000 7f750e200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75080cdb40 con 0x7f75080cdfd0 2024-03-22T13:03:05.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.955+0000 7f750e200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750815a790 con 0x7f75080ca590 2024-03-22T13:03:05.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.956+0000 7f7507400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75080ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:05.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.956+0000 7f7506a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f75080cc5f0 0x7f75080cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53100/0 (socket says 172.21.15.6:53100) 2024-03-22T13:03:05.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.956+0000 7f7506a00640 1 -- 172.21.15.6:0/1358721974 learned_addr learned my addr 172.21.15.6:0/1358721974 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:05.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.956+0000 7f7507e00640 1 --2- 172.21.15.6:0/1358721974 >> v2:172.21.15.6:3300/0 conn(0x7f75080cdfd0 0x7f75080ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:05.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.959+0000 7f7507400640 1 -- 172.21.15.6:0/1358721974 >> v2:172.21.15.6:3300/0 conn(0x7f75080cdfd0 msgr2=0x7f75080ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:05.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.959+0000 7f7507400640 1 --2- 172.21.15.6:0/1358721974 >> v2:172.21.15.6:3300/0 conn(0x7f75080cdfd0 0x7f75080ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:05.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.959+0000 7f7507400640 1 -- 172.21.15.6:0/1358721974 >> v1:172.21.15.92:6789/0 conn(0x7f75080cc5f0 legacy=0x7f75080cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:05.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.959+0000 7f7507400640 1 -- 172.21.15.6:0/1358721974 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750815aa70 con 0x7f75080ca590 2024-03-22T13:03:05.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.962+0000 7f7507400640 1 --2- 172.21.15.6:0/1358721974 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75080ca970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f74f80097a0 tx=0x7f74f802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f24ef63ff2d9b92f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:05.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.964+0000 7f7506000640 1 -- 172.21.15.6:0/1358721974 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8034070 con 0x7f75080ca590 2024-03-22T13:03:05.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.967+0000 7f7506000640 1 -- 172.21.15.6:0/1358721974 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f8030610 con 0x7f75080ca590 2024-03-22T13:03:05.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.967+0000 7f7506000640 1 -- 172.21.15.6:0/1358721974 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f80309e0 con 0x7f75080ca590 2024-03-22T13:03:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.967+0000 7f750e200640 1 -- 172.21.15.6:0/1358721974 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 msgr2=0x7f75080ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.967+0000 7f750e200640 1 --2- 172.21.15.6:0/1358721974 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75080ca970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f74f80097a0 tx=0x7f74f802b960 comp rx=0 tx=0).stop 2024-03-22T13:03:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.968+0000 7f750e200640 1 -- 172.21.15.6:0/1358721974 shutdown_connections 2024-03-22T13:03:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.968+0000 7f750e200640 1 --2- 172.21.15.6:0/1358721974 >> v2:172.21.15.6:3300/0 conn(0x7f75080cdfd0 0x7f75080ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.968+0000 7f750e200640 1 --2- 172.21.15.6:0/1358721974 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75080ca970 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.968+0000 7f750e200640 1 -- 172.21.15.6:0/1358721974 >> 172.21.15.6:0/1358721974 conn(0x7f75080b1720 msgr2=0x7f75080b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.968+0000 7f750e200640 1 -- 172.21.15.6:0/1358721974 shutdown_connections 2024-03-22T13:03:05.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.968+0000 7f750e200640 1 -- 172.21.15.6:0/1358721974 wait complete. 2024-03-22T13:03:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.970+0000 7f750e200640 1 Processor -- start 2024-03-22T13:03:05.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.970+0000 7f750e200640 1 -- start start 2024-03-22T13:03:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.971+0000 7f750e200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75081f09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.971+0000 7f750e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75080cc5f0 0x7f750815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:05.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.971+0000 7f750e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080cdfd0 0x7f7508157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:05.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.971+0000 7f750e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750815b310 con 0x7f75080cc5f0 2024-03-22T13:03:05.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.971+0000 7f750e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750815b030 con 0x7f75080cdfd0 2024-03-22T13:03:05.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.971+0000 7f750e200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750815a480 con 0x7f75080ca590 2024-03-22T13:03:05.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.972+0000 7f7507400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75081f09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:05.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.972+0000 7f7507e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080cdfd0 0x7f7508157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:05.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.972+0000 7f7506a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75080cc5f0 0x7f750815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:05.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.972+0000 7f7507400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75081f09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53298/0 (socket says 172.21.15.6:53298) 2024-03-22T13:03:05.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.972+0000 7f7507400640 1 -- 172.21.15.6:0/2515813273 learned_addr learned my addr 172.21.15.6:0/2515813273 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:05.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.976+0000 7f7507400640 1 -- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080cdfd0 msgr2=0x7f7508157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:05.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.976+0000 7f7507400640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080cdfd0 0x7f7508157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:05.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.976+0000 7f7507400640 1 -- 172.21.15.6:0/2515813273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75080cc5f0 msgr2=0x7f750815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:05.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.976+0000 7f7507400640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75080cc5f0 0x7f750815dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:05.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.976+0000 7f7507400640 1 -- 172.21.15.6:0/2515813273 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75081e37f0 con 0x7f75080ca590 2024-03-22T13:03:05.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.979+0000 7f7507400640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75081f09a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f74f80098c0 tx=0x7f74f80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:05.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.982+0000 7f74ff400640 1 -- 172.21.15.6:0/2515813273 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8034040 con 0x7f75080ca590 2024-03-22T13:03:05.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.982+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75081e3a90 con 0x7f75080ca590 2024-03-22T13:03:05.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.982+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75081e4050 con 0x7f75080ca590 2024-03-22T13:03:05.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.984+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75081e4290 con 0x7f75080ca590 2024-03-22T13:03:05.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.985+0000 7f74ff400640 1 -- 172.21.15.6:0/2515813273 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f8030410 con 0x7f75080ca590 2024-03-22T13:03:05.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:05.985+0000 7f74ff400640 1 -- 172.21.15.6:0/2515813273 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8039a10 con 0x7f75080ca590 2024-03-22T13:03:06.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.313+0000 7f74ff400640 1 -- 172.21.15.6:0/2515813273 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74f8037070 con 0x7f75080ca590 2024-03-22T13:03:06.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.315+0000 7f74ff400640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74e806bbd0 0x7f74e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:06.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.316+0000 7f7506a00640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74e806bbd0 0x7f74e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:06.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.317+0000 7f7506a00640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74e806bbd0 0x7f74e806e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f75080b5b40 tx=0x7f74f4007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:06.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.320+0000 7f74ff400640 1 -- 172.21.15.6:0/2515813273 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f74f80798f0 con 0x7f75080ca590 2024-03-22T13:03:06.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.340+0000 7f74ff400640 1 -- 172.21.15.6:0/2515813273 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f75081e4290 con 0x7f75080ca590 2024-03-22T13:03:06.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.601+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75080756c0 con 0x7f75080ca590 2024-03-22T13:03:06.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.622+0000 7f74ff400640 1 -- 172.21.15.6:0/2515813273 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f74f807b850 con 0x7f75080ca590 2024-03-22T13:03:06.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:06.622 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74e806bbd0 msgr2=0x7f74e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74e806bbd0 0x7f74e806e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f75080b5b40 tx=0x7f74f4007e80 comp rx=0 tx=0).stop 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 msgr2=0x7f75081f09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75081f09a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f74f80098c0 tx=0x7f74f80046c0 comp rx=0 tx=0).stop 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 shutdown_connections 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f74e806bbd0 0x7f74e806e090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080cdfd0 0x7f7508157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75080cc5f0 0x7f750815dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 --2- 172.21.15.6:0/2515813273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ca590 0x7f75081f09a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 >> 172.21.15.6:0/2515813273 conn(0x7f75080b1720 msgr2=0x7f75080b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 shutdown_connections 2024-03-22T13:03:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:06.625+0000 7f750e200640 1 -- 172.21.15.6:0/2515813273 wait complete. 2024-03-22T13:03:07.246 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:07.246 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:08.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:08.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.483+0000 7f4de8c00640 1 Processor -- start 2024-03-22T13:03:08.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.483+0000 7f4de8c00640 1 -- start start 2024-03-22T13:03:08.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.485+0000 7f4de8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4de4152400 0x7f4de415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:08.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.485+0000 7f4de8c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4de406f8c0 0x7f4de40c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:08.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.485+0000 7f4de8c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de41590b0 con 0x7f4de415bc80 2024-03-22T13:03:08.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.485+0000 7f4de8c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de40724d0 con 0x7f4de4152400 2024-03-22T13:03:08.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.485+0000 7f4de8c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de40721f0 con 0x7f4de406f8c0 2024-03-22T13:03:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.485+0000 7f4de1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4de4152400 0x7f4de415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.486+0000 7f4de2800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4de406f8c0 0x7f4de40c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.486+0000 7f4de1400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4de415bc80 0x7f4de40716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53104/0 (socket says 172.21.15.6:53104) 2024-03-22T13:03:08.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.486+0000 7f4de1400640 1 -- 172.21.15.6:0/383136088 learned_addr learned my addr 172.21.15.6:0/383136088 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:08.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.488+0000 7f4de1e00640 1 -- 172.21.15.6:0/383136088 >> v2:172.21.15.92:3300/0 conn(0x7f4de406f8c0 msgr2=0x7f4de40c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:08.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.488+0000 7f4de1e00640 1 --2- 172.21.15.6:0/383136088 >> v2:172.21.15.92:3300/0 conn(0x7f4de406f8c0 0x7f4de40c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:08.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.488+0000 7f4de1e00640 1 -- 172.21.15.6:0/383136088 >> v1:172.21.15.92:6789/0 conn(0x7f4de415bc80 legacy=0x7f4de40716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:08.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.488+0000 7f4de1e00640 1 -- 172.21.15.6:0/383136088 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de41582f0 con 0x7f4de4152400 2024-03-22T13:03:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.491+0000 7f4de1e00640 1 --2- 172.21.15.6:0/383136088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415b740 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4de415c860 tx=0x7f4dd402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fa59265dfe3dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:08.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.495+0000 7f4de0a00640 1 -- 172.21.15.6:0/383136088 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4034070 con 0x7f4de4152400 2024-03-22T13:03:08.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.495+0000 7f4de0a00640 1 -- 172.21.15.6:0/383136088 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd4030610 con 0x7f4de4152400 2024-03-22T13:03:08.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.495+0000 7f4de0a00640 1 -- 172.21.15.6:0/383136088 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd40309e0 con 0x7f4de4152400 2024-03-22T13:03:08.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.495+0000 7f4de8c00640 1 -- 172.21.15.6:0/383136088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 msgr2=0x7f4de415b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.496+0000 7f4de8c00640 1 --2- 172.21.15.6:0/383136088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415b740 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4de415c860 tx=0x7f4dd402b960 comp rx=0 tx=0).stop 2024-03-22T13:03:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.496+0000 7f4de8c00640 1 -- 172.21.15.6:0/383136088 shutdown_connections 2024-03-22T13:03:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.496+0000 7f4de8c00640 1 --2- 172.21.15.6:0/383136088 >> v2:172.21.15.92:3300/0 conn(0x7f4de406f8c0 0x7f4de40c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.496+0000 7f4de8c00640 1 --2- 172.21.15.6:0/383136088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415b740 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.496+0000 7f4de8c00640 1 -- 172.21.15.6:0/383136088 >> 172.21.15.6:0/383136088 conn(0x7f4de40cb930 msgr2=0x7f4de40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.496+0000 7f4de8c00640 1 -- 172.21.15.6:0/383136088 shutdown_connections 2024-03-22T13:03:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.496+0000 7f4de8c00640 1 -- 172.21.15.6:0/383136088 wait complete. 2024-03-22T13:03:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.498+0000 7f4de8c00640 1 Processor -- start 2024-03-22T13:03:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.498+0000 7f4de8c00640 1 -- start start 2024-03-22T13:03:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.499+0000 7f4de8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4de406f8c0 0x7f4de414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.499+0000 7f4de8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.499+0000 7f4de8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de415bc80 0x7f4de40c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.499+0000 7f4de8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4158b90 con 0x7f4de4152400 2024-03-22T13:03:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.499+0000 7f4de8c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de41588b0 con 0x7f4de415bc80 2024-03-22T13:03:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.499+0000 7f4de8c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de41527e0 con 0x7f4de406f8c0 2024-03-22T13:03:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.500+0000 7f4de2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de415bc80 0x7f4de40c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.500+0000 7f4de1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.500+0000 7f4de2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de415bc80 0x7f4de40c34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47902/0 (socket says 172.21.15.6:47902) 2024-03-22T13:03:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.500+0000 7f4de2800640 1 -- 172.21.15.6:0/1517356033 learned_addr learned my addr 172.21.15.6:0/1517356033 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.504+0000 7f4de1400640 1 -- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4de406f8c0 msgr2=0x7f4de414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.504+0000 7f4de1400640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4de406f8c0 0x7f4de414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.504+0000 7f4de1400640 1 -- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de415bc80 msgr2=0x7f4de40c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.504+0000 7f4de1400640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de415bc80 0x7f4de40c34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.504+0000 7f4de1400640 1 -- 172.21.15.6:0/1517356033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40c3ea0 con 0x7f4de4152400 2024-03-22T13:03:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.507+0000 7f4de1400640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415a980 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0000f60 tx=0x7f4dd000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.509+0000 7f4dda000640 1 -- 172.21.15.6:0/1517356033 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0017070 con 0x7f4de4152400 2024-03-22T13:03:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.510+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de40c4180 con 0x7f4de4152400 2024-03-22T13:03:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.510+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de415a180 con 0x7f4de4152400 2024-03-22T13:03:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.511+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de40756c0 con 0x7f4de4152400 2024-03-22T13:03:08.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.512+0000 7f4dda000640 1 -- 172.21.15.6:0/1517356033 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd0005ce0 con 0x7f4de4152400 2024-03-22T13:03:08.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.512+0000 7f4dda000640 1 -- 172.21.15.6:0/1517356033 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0021060 con 0x7f4de4152400 2024-03-22T13:03:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.828+0000 7f4dda000640 1 -- 172.21.15.6:0/1517356033 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4dd001a070 con 0x7f4de4152400 2024-03-22T13:03:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.831+0000 7f4dda000640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4dc406bbd0 0x7f4dc406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:08.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.831+0000 7f4de1e00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4dc406bbd0 0x7f4dc406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:08.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.832+0000 7f4de1e00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4dc406bbd0 0x7f4dc406e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4036000 tx=0x7f4dd4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.843+0000 7f4dda000640 1 -- 172.21.15.6:0/1517356033 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4dd0000f90 con 0x7f4de4152400 2024-03-22T13:03:08.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:08.844+0000 7f4dda000640 1 -- 172.21.15.6:0/1517356033 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4dd0058900 con 0x7f4de4152400 2024-03-22T13:03:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.092+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4de414c480 con 0x7f4de4152400 2024-03-22T13:03:09.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.122+0000 7f4dda000640 1 -- 172.21.15.6:0/1517356033 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f4dd005c9a0 con 0x7f4de4152400 2024-03-22T13:03:09.123 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:09.123 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:09.124 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:09.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4dc406bbd0 msgr2=0x7f4dc406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:09.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4dc406bbd0 0x7f4dc406e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4036000 tx=0x7f4dd4030000 comp rx=0 tx=0).stop 2024-03-22T13:03:09.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 msgr2=0x7f4de415a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:09.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415a980 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0000f60 tx=0x7f4dd000c760 comp rx=0 tx=0).stop 2024-03-22T13:03:09.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 shutdown_connections 2024-03-22T13:03:09.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4dc406bbd0 0x7f4dc406e090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:09.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de415bc80 0x7f4de40c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:09.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4de4152400 0x7f4de415a980 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:09.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 --2- 172.21.15.6:0/1517356033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4de406f8c0 0x7f4de414d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:09.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.125+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 >> 172.21.15.6:0/1517356033 conn(0x7f4de40cb930 msgr2=0x7f4de40716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:09.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.126+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 shutdown_connections 2024-03-22T13:03:09.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:09.126+0000 7f4de8c00640 1 -- 172.21.15.6:0/1517356033 wait complete. 2024-03-22T13:03:09.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:09.756 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:10.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:10.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.963+0000 7f3d11c00640 1 Processor -- start 2024-03-22T13:03:10.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.963+0000 7f3d11c00640 1 -- start start 2024-03-22T13:03:10.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.965+0000 7f3d11c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d0c15af60 0x7f3d0c0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.965+0000 7f3d11c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d0c0cc860 0x7f3d0c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.965+0000 7f3d11c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.965+0000 7f3d11c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c0cc3d0 con 0x7f3d0c15af60 2024-03-22T13:03:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.965+0000 7f3d11c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c0ce380 con 0x7f3d0c0cc860 2024-03-22T13:03:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.965+0000 7f3d11c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c0ce660 con 0x7f3d0c0ccca0 2024-03-22T13:03:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.966+0000 7f3d11200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.966+0000 7f3d0b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d0c0cc860 0x7f3d0c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:10.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.966+0000 7f3d0b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d0c0cc860 0x7f3d0c06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47908/0 (socket says 172.21.15.6:47908) 2024-03-22T13:03:10.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.966+0000 7f3d0b400640 1 -- 172.21.15.6:0/1786535260 learned_addr learned my addr 172.21.15.6:0/1786535260 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:10.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.968+0000 7f3d0be00640 1 --2- 172.21.15.6:0/1786535260 >> v2:172.21.15.6:3300/0 conn(0x7f3d0c15af60 0x7f3d0c0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:10.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.969+0000 7f3d0b400640 1 -- 172.21.15.6:0/1786535260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 msgr2=0x7f3d0c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:10.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.969+0000 7f3d0b400640 1 --2- 172.21.15.6:0/1786535260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.969+0000 7f3d0b400640 1 -- 172.21.15.6:0/1786535260 >> v2:172.21.15.6:3300/0 conn(0x7f3d0c15af60 msgr2=0x7f3d0c0cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:10.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.969+0000 7f3d0b400640 1 --2- 172.21.15.6:0/1786535260 >> v2:172.21.15.6:3300/0 conn(0x7f3d0c15af60 0x7f3d0c0cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.969+0000 7f3d0b400640 1 -- 172.21.15.6:0/1786535260 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0ce940 con 0x7f3d0c0cc860 2024-03-22T13:03:10.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.972+0000 7f3d0b400640 1 --2- 172.21.15.6:0/1786535260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c06d820 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0d0ad0 tx=0x7f3cf802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3055935bc35d48e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:10.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.977+0000 7f3d0aa00640 1 -- 172.21.15.6:0/1786535260 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8034070 con 0x7f3d0c0cc860 2024-03-22T13:03:10.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.977+0000 7f3d0aa00640 1 -- 172.21.15.6:0/1786535260 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf8030860 con 0x7f3d0c0cc860 2024-03-22T13:03:10.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.977+0000 7f3d0aa00640 1 -- 172.21.15.6:0/1786535260 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8030c30 con 0x7f3d0c0cc860 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 -- 172.21.15.6:0/1786535260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c0cc860 msgr2=0x7f3d0c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 --2- 172.21.15.6:0/1786535260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c06d820 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0d0ad0 tx=0x7f3cf802b9a0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 -- 172.21.15.6:0/1786535260 shutdown_connections 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 --2- 172.21.15.6:0/1786535260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 --2- 172.21.15.6:0/1786535260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c06d820 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 --2- 172.21.15.6:0/1786535260 >> v2:172.21.15.6:3300/0 conn(0x7f3d0c15af60 0x7f3d0c0cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 -- 172.21.15.6:0/1786535260 >> 172.21.15.6:0/1786535260 conn(0x7f3d0c0cf9d0 msgr2=0x7f3d0c0cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 -- 172.21.15.6:0/1786535260 shutdown_connections 2024-03-22T13:03:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.978+0000 7f3d11c00640 1 -- 172.21.15.6:0/1786535260 wait complete. 2024-03-22T13:03:10.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.980+0000 7f3d11c00640 1 Processor -- start 2024-03-22T13:03:10.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.980+0000 7f3d11c00640 1 -- start start 2024-03-22T13:03:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.981+0000 7f3d11c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.981+0000 7f3d11c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.981+0000 7f3d11c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c15af60 0x7f3d0c0afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.981+0000 7f3d11c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c06dd60 con 0x7f3d0c0cc860 2024-03-22T13:03:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.981+0000 7f3d11c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c15a590 con 0x7f3d0c15af60 2024-03-22T13:03:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.981+0000 7f3d11c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c06dee0 con 0x7f3d0c0ccca0 2024-03-22T13:03:10.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.982+0000 7f3d11200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c15af60 0x7f3d0c0afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:10.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.982+0000 7f3d0be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:10.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.982+0000 7f3d0b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:10.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.982+0000 7f3d11200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c15af60 0x7f3d0c0afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47914/0 (socket says 172.21.15.6:47914) 2024-03-22T13:03:10.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.982+0000 7f3d11200640 1 -- 172.21.15.6:0/2601580929 learned_addr learned my addr 172.21.15.6:0/2601580929 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:10.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.986+0000 7f3d0be00640 1 -- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 msgr2=0x7f3d0c0c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:10.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.986+0000 7f3d0be00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.986+0000 7f3d0be00640 1 -- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c15af60 msgr2=0x7f3d0c0afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:10.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.986+0000 7f3d0be00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c15af60 0x7f3d0c0afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:10.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.986+0000 7f3d0be00640 1 -- 172.21.15.6:0/2601580929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0b3e60 con 0x7f3d0c0cc860 2024-03-22T13:03:10.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.988+0000 7f3d0be00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c150080 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00d520 tx=0x7f3cfc00d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:10.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.997+0000 7f3d03e00640 1 -- 172.21.15.6:0/2601580929 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc0125f0 con 0x7f3d0c0cc860 2024-03-22T13:03:10.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.997+0000 7f3d03e00640 1 -- 172.21.15.6:0/2601580929 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc019070 con 0x7f3d0c0cc860 2024-03-22T13:03:10.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.997+0000 7f3d03e00640 1 -- 172.21.15.6:0/2601580929 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc0129b0 con 0x7f3d0c0cc860 2024-03-22T13:03:10.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.997+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0c0b0c30 con 0x7f3d0c0cc860 2024-03-22T13:03:10.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.997+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0c0af580 con 0x7f3d0c0cc860 2024-03-22T13:03:10.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:10.999+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0c075650 con 0x7f3d0c0cc860 2024-03-22T13:03:11.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.021+0000 7f3d03e00640 1 -- 172.21.15.6:0/2601580929 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3cfc012b50 con 0x7f3d0c0cc860 2024-03-22T13:03:11.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.021+0000 7f3d03e00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3cec06bc50 0x7f3cec06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:11.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.022+0000 7f3d0b400640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3cec06bc50 0x7f3cec06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:11.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.022+0000 7f3d0b400640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3cec06bc50 0x7f3cec06e110 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8036000 tx=0x7f3cf8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:11.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.025+0000 7f3d03e00640 1 -- 172.21.15.6:0/2601580929 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3cfc012e40 con 0x7f3d0c0cc860 2024-03-22T13:03:11.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.321+0000 7f3d03e00640 1 -- 172.21.15.6:0/2601580929 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3cfc059be0 con 0x7f3d0c0cc860 2024-03-22T13:03:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.578+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d0c0b5950 con 0x7f3d0c0cc860 2024-03-22T13:03:11.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.605+0000 7f3d03e00640 1 -- 172.21.15.6:0/2601580929 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f3cfc05dc80 con 0x7f3d0c0cc860 2024-03-22T13:03:11.605 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:11.605 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:11.605 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.607+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3cec06bc50 msgr2=0x7f3cec06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.607+0000 7f3d11c00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3cec06bc50 0x7f3cec06e110 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8036000 tx=0x7f3cf8008030 comp rx=0 tx=0).stop 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.607+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d0c0cc860 msgr2=0x7f3d0c150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.607+0000 7f3d11c00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c150080 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00d520 tx=0x7f3cfc00d9e0 comp rx=0 tx=0).stop 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 shutdown_connections 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3cec06bc50 0x7f3cec06e110 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d0c15af60 0x7f3d0c0afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d0c0ccca0 0x7f3d0c0c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:11.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 --2- 172.21.15.6:0/2601580929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d0c0cc860 0x7f3d0c150080 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:11.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 >> 172.21.15.6:0/2601580929 conn(0x7f3d0c0cf9d0 msgr2=0x7f3d0c0c8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:11.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 shutdown_connections 2024-03-22T13:03:11.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:11.608+0000 7f3d11c00640 1 -- 172.21.15.6:0/2601580929 wait complete. 2024-03-22T13:03:12.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:12.233 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:13.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.940+0000 7f4f32200640 1 Processor -- start 2024-03-22T13:03:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.940+0000 7f4f32200640 1 -- start start 2024-03-22T13:03:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.941+0000 7f4f32200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0cc380 0x7f4f2c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.942+0000 7f4f32200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f2c15ae30 0x7f4f2c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.942+0000 7f4f32200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c15b260 con 0x7f4f2c0ce8a0 2024-03-22T13:03:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.942+0000 7f4f32200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c0cee00 con 0x7f4f2c15ae30 2024-03-22T13:03:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.942+0000 7f4f32200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0cf0c0 con 0x7f4f2c0cc380 2024-03-22T13:03:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.942+0000 7f4f30e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0cc380 0x7f4f2c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.943+0000 7f4f2be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4f2c0ce8a0 0x7f4f2c0cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:56810/0 (socket says 172.21.15.6:56810) 2024-03-22T13:03:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.943+0000 7f4f2be00640 1 -- 172.21.15.6:0/2422907096 learned_addr learned my addr 172.21.15.6:0/2422907096 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.943+0000 7f4f31800640 1 --2- 172.21.15.6:0/2422907096 >> v2:172.21.15.6:3300/0 conn(0x7f4f2c15ae30 0x7f4f2c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.945+0000 7f4f2b400640 1 -- 172.21.15.6:0/2422907096 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571217505 0 0) 0x7f4f18001110 con 0x7f4f2c0ce8a0 2024-03-22T13:03:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.945+0000 7f4f2b400640 1 -- 172.21.15.6:0/2422907096 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14003520 con 0x7f4f2c0ce8a0 2024-03-22T13:03:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.946+0000 7f4f30e00640 1 -- 172.21.15.6:0/2422907096 >> v2:172.21.15.6:3300/0 conn(0x7f4f2c15ae30 msgr2=0x7f4f2c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.946+0000 7f4f30e00640 1 --2- 172.21.15.6:0/2422907096 >> v2:172.21.15.6:3300/0 conn(0x7f4f2c15ae30 0x7f4f2c0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.946+0000 7f4f30e00640 1 -- 172.21.15.6:0/2422907096 >> v1:172.21.15.92:6789/0 conn(0x7f4f2c0ce8a0 legacy=0x7f4f2c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.946+0000 7f4f30e00640 1 -- 172.21.15.6:0/2422907096 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0cf3a0 con 0x7f4f2c0cc380 2024-03-22T13:03:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.949+0000 7f4f30e00640 1 --2- 172.21.15.6:0/2422907096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0cc380 0x7f4f2c06c230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4f200097a0 tx=0x7f4f2002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c48c34120f2f9d4d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.953+0000 7f4f2b400640 1 -- 172.21.15.6:0/2422907096 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20035020 con 0x7f4f2c0cc380 2024-03-22T13:03:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.953+0000 7f4f2b400640 1 -- 172.21.15.6:0/2422907096 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f20031830 con 0x7f4f2c0cc380 2024-03-22T13:03:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.953+0000 7f4f2b400640 1 -- 172.21.15.6:0/2422907096 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20031c00 con 0x7f4f2c0cc380 2024-03-22T13:03:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 -- 172.21.15.6:0/2422907096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0cc380 msgr2=0x7f4f2c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 --2- 172.21.15.6:0/2422907096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0cc380 0x7f4f2c06c230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4f200097a0 tx=0x7f4f2002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:03:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 -- 172.21.15.6:0/2422907096 shutdown_connections 2024-03-22T13:03:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 --2- 172.21.15.6:0/2422907096 >> v2:172.21.15.6:3300/0 conn(0x7f4f2c15ae30 0x7f4f2c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 --2- 172.21.15.6:0/2422907096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0cc380 0x7f4f2c06c230 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 -- 172.21.15.6:0/2422907096 >> 172.21.15.6:0/2422907096 conn(0x7f4f2c0b1480 msgr2=0x7f4f2c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 -- 172.21.15.6:0/2422907096 shutdown_connections 2024-03-22T13:03:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.954+0000 7f4f32200640 1 -- 172.21.15.6:0/2422907096 wait complete. 2024-03-22T13:03:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.956+0000 7f4f32200640 1 Processor -- start 2024-03-22T13:03:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.956+0000 7f4f32200640 1 -- start start 2024-03-22T13:03:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f32200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f2c0cc380 0x7f4f2c0b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f32200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0ce8a0 0x7f4f2c15d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f32200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f2c15ae30 0x7f4f2c15dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f32200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c15a910 con 0x7f4f2c0cc380 2024-03-22T13:03:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f32200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c15a630 con 0x7f4f2c15ae30 2024-03-22T13:03:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f32200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1549d0 con 0x7f4f2c0ce8a0 2024-03-22T13:03:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f2be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0ce8a0 0x7f4f2c15d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.958+0000 7f4f31800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f2c15ae30 0x7f4f2c15dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.959+0000 7f4f30e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f2c0cc380 0x7f4f2c0b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.959+0000 7f4f31800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f2c15ae30 0x7f4f2c15dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:38802/0 (socket says 172.21.15.6:38802) 2024-03-22T13:03:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.959+0000 7f4f31800640 1 -- 172.21.15.6:0/881545610 learned_addr learned my addr 172.21.15.6:0/881545610 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.964+0000 7f4f31800640 1 -- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0ce8a0 msgr2=0x7f4f2c15d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.964+0000 7f4f31800640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0ce8a0 0x7f4f2c15d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.964+0000 7f4f31800640 1 -- 172.21.15.6:0/881545610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f2c0cc380 msgr2=0x7f4f2c0b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.964+0000 7f4f31800640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f2c0cc380 0x7f4f2c0b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.964+0000 7f4f31800640 1 -- 172.21.15.6:0/881545610 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0bebd0 con 0x7f4f2c15ae30 2024-03-22T13:03:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.967+0000 7f4f31800640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f2c15ae30 0x7f4f2c15dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009dd0 tx=0x7f4f2400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.970+0000 7f4f28c00640 1 -- 172.21.15.6:0/881545610 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24018020 con 0x7f4f2c15ae30 2024-03-22T13:03:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.970+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c15b440 con 0x7f4f2c15ae30 2024-03-22T13:03:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.970+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c0be8f0 con 0x7f4f2c15ae30 2024-03-22T13:03:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.971+0000 7f4f28c00640 1 -- 172.21.15.6:0/881545610 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f24014480 con 0x7f4f2c15ae30 2024-03-22T13:03:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.971+0000 7f4f28c00640 1 -- 172.21.15.6:0/881545610 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24014830 con 0x7f4f2c15ae30 2024-03-22T13:03:13.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:13.972+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f08005290 con 0x7f4f2c15ae30 2024-03-22T13:03:14.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.243+0000 7f4f28c00640 1 -- 172.21.15.6:0/881545610 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4f24005d10 con 0x7f4f2c15ae30 2024-03-22T13:03:14.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.245+0000 7f4f28c00640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4f0c06bbd0 0x7f4f0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:14.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.245+0000 7f4f30e00640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4f0c06bbd0 0x7f4f0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:14.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.246+0000 7f4f30e00640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4f0c06bbd0 0x7f4f0c06e090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4f20036000 tx=0x7f4f2003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:14.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.246+0000 7f4f28c00640 1 -- 172.21.15.6:0/881545610 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4f240580c0 con 0x7f4f2c15ae30 2024-03-22T13:03:14.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.258+0000 7f4f28c00640 1 -- 172.21.15.6:0/881545610 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4f24022070 con 0x7f4f2c15ae30 2024-03-22T13:03:14.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.506+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f08005580 con 0x7f4f2c15ae30 2024-03-22T13:03:14.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.521+0000 7f4f28c00640 1 -- 172.21.15.6:0/881545610 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f4f24010040 con 0x7f4f2c15ae30 2024-03-22T13:03:14.521 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:14.521 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:14.521 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.523+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4f0c06bbd0 msgr2=0x7f4f0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.523+0000 7f4f32200640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4f0c06bbd0 0x7f4f0c06e090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4f20036000 tx=0x7f4f2003c040 comp rx=0 tx=0).stop 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.523+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f2c15ae30 msgr2=0x7f4f2c15dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.523+0000 7f4f32200640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f2c15ae30 0x7f4f2c15dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009dd0 tx=0x7f4f2400c490 comp rx=0 tx=0).stop 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 shutdown_connections 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4f0c06bbd0 0x7f4f0c06e090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f2c15ae30 0x7f4f2c15dcd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f2c0ce8a0 0x7f4f2c15d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 --2- 172.21.15.6:0/881545610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f2c0cc380 0x7f4f2c0b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 >> 172.21.15.6:0/881545610 conn(0x7f4f2c0b1480 msgr2=0x7f4f2c0c2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 shutdown_connections 2024-03-22T13:03:14.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:14.524+0000 7f4f32200640 1 -- 172.21.15.6:0/881545610 wait complete. 2024-03-22T13:03:15.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:15.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:16.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:16.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.341+0000 7f44e7000640 1 Processor -- start 2024-03-22T13:03:16.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.341+0000 7f44e7000640 1 -- start start 2024-03-22T13:03:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.343+0000 7f44e7000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44e00cc380 0x7f44e006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.343+0000 7f44e7000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f44e015ae30 0x7f44e00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.343+0000 7f44e7000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e015b240 con 0x7f44e00ce8a0 2024-03-22T13:03:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.343+0000 7f44e7000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e00cee00 con 0x7f44e00cc380 2024-03-22T13:03:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.343+0000 7f44e7000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e00cf0c0 con 0x7f44e015ae30 2024-03-22T13:03:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.344+0000 7f44e5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44e00cc380 0x7f44e006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.344+0000 7f44e5200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f44e00ce8a0 0x7f44e00cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:56820/0 (socket says 172.21.15.6:56820) 2024-03-22T13:03:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.344+0000 7f44e5200640 1 -- 172.21.15.6:0/1924182333 learned_addr learned my addr 172.21.15.6:0/1924182333 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.344+0000 7f44e6600640 1 --2- 172.21.15.6:0/1924182333 >> v2:172.21.15.92:3300/0 conn(0x7f44e015ae30 0x7f44e00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.347+0000 7f44e5c00640 1 -- 172.21.15.6:0/1924182333 >> v2:172.21.15.92:3300/0 conn(0x7f44e015ae30 msgr2=0x7f44e00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.347+0000 7f44e5c00640 1 --2- 172.21.15.6:0/1924182333 >> v2:172.21.15.92:3300/0 conn(0x7f44e015ae30 0x7f44e00cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.347+0000 7f44e5c00640 1 -- 172.21.15.6:0/1924182333 >> v1:172.21.15.92:6789/0 conn(0x7f44e00ce8a0 legacy=0x7f44e00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.347+0000 7f44e5c00640 1 -- 172.21.15.6:0/1924182333 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e00cf3a0 con 0x7f44e00cc380 2024-03-22T13:03:16.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.349+0000 7f44e5c00640 1 --2- 172.21.15.6:0/1924182333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00cc380 0x7f44e006c230 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44e00c9f50 tx=0x7f44d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670a51a990c7096f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.355+0000 7f44dfe00640 1 -- 172.21.15.6:0/1924182333 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4039070 con 0x7f44e00cc380 2024-03-22T13:03:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.355+0000 7f44dfe00640 1 -- 172.21.15.6:0/1924182333 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d402cc80 con 0x7f44e00cc380 2024-03-22T13:03:16.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.355+0000 7f44dfe00640 1 -- 172.21.15.6:0/1924182333 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4034660 con 0x7f44e00cc380 2024-03-22T13:03:16.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.356+0000 7f44e7000640 1 -- 172.21.15.6:0/1924182333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00cc380 msgr2=0x7f44e006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:16.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.356+0000 7f44e7000640 1 --2- 172.21.15.6:0/1924182333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00cc380 0x7f44e006c230 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44e00c9f50 tx=0x7f44d402bed0 comp rx=0 tx=0).stop 2024-03-22T13:03:16.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.356+0000 7f44e7000640 1 -- 172.21.15.6:0/1924182333 shutdown_connections 2024-03-22T13:03:16.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.356+0000 7f44e7000640 1 --2- 172.21.15.6:0/1924182333 >> v2:172.21.15.92:3300/0 conn(0x7f44e015ae30 0x7f44e00cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:16.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.356+0000 7f44e7000640 1 --2- 172.21.15.6:0/1924182333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00cc380 0x7f44e006c230 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:16.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.356+0000 7f44e7000640 1 -- 172.21.15.6:0/1924182333 >> 172.21.15.6:0/1924182333 conn(0x7f44e00b1480 msgr2=0x7f44e00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:16.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.356+0000 7f44e7000640 1 -- 172.21.15.6:0/1924182333 shutdown_connections 2024-03-22T13:03:16.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.357+0000 7f44e7000640 1 -- 172.21.15.6:0/1924182333 wait complete. 2024-03-22T13:03:16.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.359+0000 7f44e7000640 1 Processor -- start 2024-03-22T13:03:16.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.359+0000 7f44e7000640 1 -- start start 2024-03-22T13:03:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.361+0000 7f44e7000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44e00cc380 0x7f44e0159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.361+0000 7f44e7000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00ce8a0 0x7f44e0153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.361+0000 7f44e7000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e015ae30 0x7f44e01e3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.361+0000 7f44e7000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e015a910 con 0x7f44e00ce8a0 2024-03-22T13:03:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.361+0000 7f44e7000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e015a630 con 0x7f44e015ae30 2024-03-22T13:03:16.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.361+0000 7f44e7000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e01549d0 con 0x7f44e00cc380 2024-03-22T13:03:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.362+0000 7f44e6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e015ae30 0x7f44e01e3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.362+0000 7f44e5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00ce8a0 0x7f44e0153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.362+0000 7f44e5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44e00cc380 0x7f44e0159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.362+0000 7f44e6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e015ae30 0x7f44e01e3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:38822/0 (socket says 172.21.15.6:38822) 2024-03-22T13:03:16.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.362+0000 7f44e6600640 1 -- 172.21.15.6:0/2383666497 learned_addr learned my addr 172.21.15.6:0/2383666497 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:16.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.366+0000 7f44e5200640 1 -- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44e00cc380 msgr2=0x7f44e0159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:16.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.366+0000 7f44e5200640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44e00cc380 0x7f44e0159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:16.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.366+0000 7f44e5200640 1 -- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e015ae30 msgr2=0x7f44e01e3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:16.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.366+0000 7f44e5200640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e015ae30 0x7f44e01e3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:16.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.366+0000 7f44e5200640 1 -- 172.21.15.6:0/2383666497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0154190 con 0x7f44e00ce8a0 2024-03-22T13:03:16.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.369+0000 7f44e5200640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00ce8a0 0x7f44e0153b70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f44cc000f60 tx=0x7f44cc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:16.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.373+0000 7f44dd600640 1 -- 172.21.15.6:0/2383666497 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc017070 con 0x7f44e00ce8a0 2024-03-22T13:03:16.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.374+0000 7f44dd600640 1 -- 172.21.15.6:0/2383666497 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44cc005ce0 con 0x7f44e00ce8a0 2024-03-22T13:03:16.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.374+0000 7f44dd600640 1 -- 172.21.15.6:0/2383666497 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc007310 con 0x7f44e00ce8a0 2024-03-22T13:03:16.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.374+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e0158f70 con 0x7f44e00ce8a0 2024-03-22T13:03:16.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.374+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e01530f0 con 0x7f44e00ce8a0 2024-03-22T13:03:16.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.375+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e0075650 con 0x7f44e00ce8a0 2024-03-22T13:03:16.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.397+0000 7f44dd600640 1 -- 172.21.15.6:0/2383666497 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f44cc01a070 con 0x7f44e00ce8a0 2024-03-22T13:03:16.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.398+0000 7f44dd600640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f44c006bc20 0x7f44c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:16.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.399+0000 7f44e5c00640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f44c006bc20 0x7f44c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.400+0000 7f44e5c00640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f44c006bc20 0x7f44c006e0e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f44d4036000 tx=0x7f44d4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:16.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.714+0000 7f44dd600640 1 -- 172.21.15.6:0/2383666497 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f44cc0566f0 con 0x7f44e00ce8a0 2024-03-22T13:03:16.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:16.714+0000 7f44dd600640 1 -- 172.21.15.6:0/2383666497 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f44cc021020 con 0x7f44e00ce8a0 2024-03-22T13:03:17.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.042+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44e01532c0 con 0x7f44e00ce8a0 2024-03-22T13:03:17.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.078+0000 7f44dd600640 1 -- 172.21.15.6:0/2383666497 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f44cc058650 con 0x7f44e00ce8a0 2024-03-22T13:03:17.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:17.079 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:17.079 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f44c006bc20 msgr2=0x7f44c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f44c006bc20 0x7f44c006e0e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f44d4036000 tx=0x7f44d4030000 comp rx=0 tx=0).stop 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00ce8a0 msgr2=0x7f44e0153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00ce8a0 0x7f44e0153b70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f44cc000f60 tx=0x7f44cc00c800 comp rx=0 tx=0).stop 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 shutdown_connections 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f44c006bc20 0x7f44c006e0e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e015ae30 0x7f44e01e3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44e00ce8a0 0x7f44e0153b70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 --2- 172.21.15.6:0/2383666497 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44e00cc380 0x7f44e0159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 >> 172.21.15.6:0/2383666497 conn(0x7f44e00b1480 msgr2=0x7f44e00c2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:17.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 shutdown_connections 2024-03-22T13:03:17.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:17.081+0000 7f44e7000640 1 -- 172.21.15.6:0/2383666497 wait complete. 2024-03-22T13:03:17.711 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:17.712 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:18.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:18.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.911+0000 7f94f1e00640 1 Processor -- start 2024-03-22T13:03:18.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.911+0000 7f94f1e00640 1 -- start start 2024-03-22T13:03:18.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.913+0000 7f94f1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f94ec0cc3b0 0x7f94ec0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:18.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.913+0000 7f94f1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f94ec0cf770 0x7f94ec0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:18.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.913+0000 7f94f1e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0ce230 con 0x7f94ec0cccd0 2024-03-22T13:03:18.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.913+0000 7f94f1e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ec0ce510 con 0x7f94ec0cf770 2024-03-22T13:03:18.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.913+0000 7f94f1e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ec0ce7f0 con 0x7f94ec0cc3b0 2024-03-22T13:03:18.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.914+0000 7f94eaa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f94ec0cc3b0 0x7f94ec0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:18.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.914+0000 7f94eb400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f94ec0cccd0 0x7f94ec15ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:56824/0 (socket says 172.21.15.6:56824) 2024-03-22T13:03:18.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.914+0000 7f94eb400640 1 -- 172.21.15.6:0/4189862024 learned_addr learned my addr 172.21.15.6:0/4189862024 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:18.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.918+0000 7f94eaa00640 1 -- 172.21.15.6:0/4189862024 >> v2:172.21.15.6:3300/0 conn(0x7f94ec0cf770 msgr2=0x7f94ec0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.918+0000 7f94eaa00640 1 --2- 172.21.15.6:0/4189862024 >> v2:172.21.15.6:3300/0 conn(0x7f94ec0cf770 0x7f94ec0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.918+0000 7f94eaa00640 1 -- 172.21.15.6:0/4189862024 >> v1:172.21.15.92:6789/0 conn(0x7f94ec0cccd0 legacy=0x7f94ec15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.918+0000 7f94eaa00640 1 -- 172.21.15.6:0/4189862024 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0cead0 con 0x7f94ec0cc3b0 2024-03-22T13:03:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.921+0000 7f94eaa00640 1 --2- 172.21.15.6:0/4189862024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 0x7f94ec0cf230 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f94ec073780 tx=0x7f94dc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3802e7f7adb74ab server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.929+0000 7f94e9600640 1 -- 172.21.15.6:0/4189862024 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0315a0 con 0x7f94ec0cc3b0 2024-03-22T13:03:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.929+0000 7f94e9600640 1 -- 172.21.15.6:0/4189862024 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc037070 con 0x7f94ec0cc3b0 2024-03-22T13:03:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.929+0000 7f94e9600640 1 -- 172.21.15.6:0/4189862024 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc031960 con 0x7f94ec0cc3b0 2024-03-22T13:03:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.930+0000 7f94f1e00640 1 -- 172.21.15.6:0/4189862024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 msgr2=0x7f94ec0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.930+0000 7f94f1e00640 1 --2- 172.21.15.6:0/4189862024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 0x7f94ec0cf230 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f94ec073780 tx=0x7f94dc02b840 comp rx=0 tx=0).stop 2024-03-22T13:03:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.931+0000 7f94f1e00640 1 -- 172.21.15.6:0/4189862024 shutdown_connections 2024-03-22T13:03:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.931+0000 7f94f1e00640 1 --2- 172.21.15.6:0/4189862024 >> v2:172.21.15.6:3300/0 conn(0x7f94ec0cf770 0x7f94ec0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.931+0000 7f94f1e00640 1 --2- 172.21.15.6:0/4189862024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 0x7f94ec0cf230 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.931+0000 7f94f1e00640 1 -- 172.21.15.6:0/4189862024 >> 172.21.15.6:0/4189862024 conn(0x7f94ec0b16d0 msgr2=0x7f94ec0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.931+0000 7f94f1e00640 1 -- 172.21.15.6:0/4189862024 shutdown_connections 2024-03-22T13:03:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.931+0000 7f94f1e00640 1 -- 172.21.15.6:0/4189862024 wait complete. 2024-03-22T13:03:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.933+0000 7f94f1e00640 1 Processor -- start 2024-03-22T13:03:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.933+0000 7f94f1e00640 1 -- start start 2024-03-22T13:03:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.934+0000 7f94f1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 0x7f94ec0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.934+0000 7f94f1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec0cccd0 0x7f94ec1f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.934+0000 7f94f1e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94ec0cf770 0x7f94ec154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.934+0000 7f94f1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec15a5d0 con 0x7f94ec0cccd0 2024-03-22T13:03:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.934+0000 7f94f1e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec06dc10 con 0x7f94ec0cc3b0 2024-03-22T13:03:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.934+0000 7f94f1e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec154bb0 con 0x7f94ec0cf770 2024-03-22T13:03:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.935+0000 7f94eb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94ec0cf770 0x7f94ec154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.936+0000 7f94eaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 0x7f94ec0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.936+0000 7f94eb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94ec0cf770 0x7f94ec154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50472/0 (socket says 172.21.15.6:50472) 2024-03-22T13:03:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.936+0000 7f94eb400640 1 -- 172.21.15.6:0/2808944661 learned_addr learned my addr 172.21.15.6:0/2808944661 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:18.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.939+0000 7f94ea000640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec0cccd0 0x7f94ec1f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:18.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.940+0000 7f94eb400640 1 -- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 msgr2=0x7f94ec0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:18.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.940+0000 7f94eb400640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 0x7f94ec0b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:18.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.940+0000 7f94eb400640 1 -- 172.21.15.6:0/2808944661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec0cccd0 msgr2=0x7f94ec1f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:18.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.940+0000 7f94eb400640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec0cccd0 0x7f94ec1f3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:18.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.940+0000 7f94eb400640 1 -- 172.21.15.6:0/2808944661 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec157240 con 0x7f94ec0cf770 2024-03-22T13:03:18.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.943+0000 7f94eb400640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94ec0cf770 0x7f94ec154f40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f94e400cbf0 tx=0x7f94e400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:18.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.947+0000 7f94e3400640 1 -- 172.21.15.6:0/2808944661 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4017070 con 0x7f94ec0cf770 2024-03-22T13:03:18.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.947+0000 7f94e3400640 1 -- 172.21.15.6:0/2808944661 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e400ad40 con 0x7f94ec0cf770 2024-03-22T13:03:18.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.947+0000 7f94e3400640 1 -- 172.21.15.6:0/2808944661 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4012600 con 0x7f94ec0cf770 2024-03-22T13:03:18.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.947+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec155ff0 con 0x7f94ec0cf770 2024-03-22T13:03:18.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.948+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec14e600 con 0x7f94ec0cf770 2024-03-22T13:03:18.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:18.949+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ec0756c0 con 0x7f94ec0cf770 2024-03-22T13:03:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.215+0000 7f94e3400640 1 -- 172.21.15.6:0/2808944661 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f94e401a070 con 0x7f94ec0cf770 2024-03-22T13:03:19.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.218+0000 7f94e3400640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f94cc06bc20 0x7f94cc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:19.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.218+0000 7f94e3400640 1 -- 172.21.15.6:0/2808944661 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f94e408af90 con 0x7f94ec0cf770 2024-03-22T13:03:19.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.218+0000 7f94eaa00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f94cc06bc20 0x7f94cc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:19.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.219+0000 7f94eaa00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f94cc06bc20 0x7f94cc06e0e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94dc036000 tx=0x7f94dc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.228+0000 7f94e3400640 1 -- 172.21.15.6:0/2808944661 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f94e4059110 con 0x7f94ec0cf770 2024-03-22T13:03:19.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.483+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94ec158870 con 0x7f94ec0cf770 2024-03-22T13:03:19.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.495+0000 7f94e3400640 1 -- 172.21.15.6:0/2808944661 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f94e405d1b0 con 0x7f94ec0cf770 2024-03-22T13:03:19.497 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:19.497 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:19.497 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f94cc06bc20 msgr2=0x7f94cc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f94cc06bc20 0x7f94cc06e0e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94dc036000 tx=0x7f94dc03d040 comp rx=0 tx=0).stop 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94ec0cf770 msgr2=0x7f94ec154f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94ec0cf770 0x7f94ec154f40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f94e400cbf0 tx=0x7f94e400a430 comp rx=0 tx=0).stop 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 shutdown_connections 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94ec0cf770 0x7f94ec154f40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec0cccd0 0x7f94ec1f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f94cc06bc20 0x7f94cc06e0e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:19.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 --2- 172.21.15.6:0/2808944661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94ec0cc3b0 0x7f94ec0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:19.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 >> 172.21.15.6:0/2808944661 conn(0x7f94ec0b16d0 msgr2=0x7f94ec159310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:19.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 shutdown_connections 2024-03-22T13:03:19.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:19.498+0000 7f94f1e00640 1 -- 172.21.15.6:0/2808944661 wait complete. 2024-03-22T13:03:20.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:20.122 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:21.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.327+0000 7fb0b0c00640 1 Processor -- start 2024-03-22T13:03:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.327+0000 7fb0b0c00640 1 -- start start 2024-03-22T13:03:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.328+0000 7fb0b0c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0ac071480 0x7fb0ac06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.328+0000 7fb0b0c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac15a1f0 0x7fb0ac15dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.328+0000 7fb0b0c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac15a6a0 con 0x7fb0ac071860 2024-03-22T13:03:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.328+0000 7fb0b0c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ac15a980 con 0x7fb0ac071480 2024-03-22T13:03:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.328+0000 7fb0b0c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac15ac60 con 0x7fb0ac15a1f0 2024-03-22T13:03:21.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.329+0000 7fb0ab400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0ac071480 0x7fb0ac06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:21.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.329+0000 7fb0aaa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac15a1f0 0x7fb0ac15dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:21.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.329+0000 7fb0ab400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0ac071480 0x7fb0ac06cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58850/0 (socket says 172.21.15.6:58850) 2024-03-22T13:03:21.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.329+0000 7fb0ab400640 1 -- 172.21.15.6:0/2869266029 learned_addr learned my addr 172.21.15.6:0/2869266029 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.332+0000 7fb0aaa00640 1 -- 172.21.15.6:0/2869266029 >> v2:172.21.15.6:3300/0 conn(0x7fb0ac071480 msgr2=0x7fb0ac06cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.332+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/2869266029 >> v2:172.21.15.6:3300/0 conn(0x7fb0ac071480 0x7fb0ac06cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.332+0000 7fb0aaa00640 1 -- 172.21.15.6:0/2869266029 >> v1:172.21.15.92:6789/0 conn(0x7fb0ac071860 legacy=0x7fb0ac153fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.332+0000 7fb0aaa00640 1 -- 172.21.15.6:0/2869266029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac15af40 con 0x7fb0ac15a1f0 2024-03-22T13:03:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.332+0000 7fb0ab400640 1 --2- 172.21.15.6:0/2869266029 >> v2:172.21.15.6:3300/0 conn(0x7fb0ac071480 0x7fb0ac06cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:03:21.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.335+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/2869266029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac15a1f0 0x7fb0ac15dea0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb0980097a0 tx=0x7fb09802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b441d9424d389ad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:21.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.340+0000 7fb0aa000640 1 -- 172.21.15.6:0/2869266029 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb098034070 con 0x7fb0ac15a1f0 2024-03-22T13:03:21.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.340+0000 7fb0aa000640 1 -- 172.21.15.6:0/2869266029 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb098030800 con 0x7fb0ac15a1f0 2024-03-22T13:03:21.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.340+0000 7fb0aa000640 1 -- 172.21.15.6:0/2869266029 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb098030bd0 con 0x7fb0ac15a1f0 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2869266029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac15a1f0 msgr2=0x7fb0ac15dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2869266029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac15a1f0 0x7fb0ac15dea0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb0980097a0 tx=0x7fb09802b940 comp rx=0 tx=0).stop 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2869266029 shutdown_connections 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2869266029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac15a1f0 0x7fb0ac15dea0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2869266029 >> v2:172.21.15.6:3300/0 conn(0x7fb0ac071480 0x7fb0ac06cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2869266029 >> 172.21.15.6:0/2869266029 conn(0x7fb0ac06b4a0 msgr2=0x7fb0ac06f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2869266029 shutdown_connections 2024-03-22T13:03:21.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.341+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2869266029 wait complete. 2024-03-22T13:03:21.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.343+0000 7fb0b0c00640 1 Processor -- start 2024-03-22T13:03:21.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.343+0000 7fb0b0c00640 1 -- start start 2024-03-22T13:03:21.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.344+0000 7fb0b0c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac071480 0x7fb0ac0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:21.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.344+0000 7fb0b0c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0ac071860 0x7fb0ac0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:21.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.344+0000 7fb0b0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15a1f0 0x7fb0ac140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:21.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.344+0000 7fb0b0c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac1546c0 con 0x7fb0ac15a1f0 2024-03-22T13:03:21.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.344+0000 7fb0b0c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac06d0e0 con 0x7fb0ac071860 2024-03-22T13:03:21.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.344+0000 7fb0b0c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac154840 con 0x7fb0ac071480 2024-03-22T13:03:21.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.344+0000 7fb0aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0ac071860 0x7fb0ac0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:21.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.345+0000 7fb0abe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15a1f0 0x7fb0ac140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:21.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.345+0000 7fb0ab400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac071480 0x7fb0ac0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:21.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.345+0000 7fb0aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0ac071860 0x7fb0ac0cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:38842/0 (socket says 172.21.15.6:38842) 2024-03-22T13:03:21.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.345+0000 7fb0aaa00640 1 -- 172.21.15.6:0/2183070236 learned_addr learned my addr 172.21.15.6:0/2183070236 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:21.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.349+0000 7fb0ab400640 1 -- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0ac071860 msgr2=0x7fb0ac0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:21.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.349+0000 7fb0ab400640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0ac071860 0x7fb0ac0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.349+0000 7fb0ab400640 1 -- 172.21.15.6:0/2183070236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15a1f0 msgr2=0x7fb0ac140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:21.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.349+0000 7fb0ab400640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15a1f0 0x7fb0ac140220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.349+0000 7fb0ab400640 1 -- 172.21.15.6:0/2183070236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac15d3b0 con 0x7fb0ac071480 2024-03-22T13:03:21.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.352+0000 7fb0ab400640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac071480 0x7fb0ac0cda70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00e900 tx=0x7fb09c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:21.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.357+0000 7fb0a3400640 1 -- 172.21.15.6:0/2183070236 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c00ca00 con 0x7fb0ac071480 2024-03-22T13:03:21.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.357+0000 7fb0a3400640 1 -- 172.21.15.6:0/2183070236 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c00cba0 con 0x7fb0ac071480 2024-03-22T13:03:21.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.357+0000 7fb0a3400640 1 -- 172.21.15.6:0/2183070236 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c004030 con 0x7fb0ac071480 2024-03-22T13:03:21.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.357+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac0cbb90 con 0x7fb0ac071480 2024-03-22T13:03:21.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.357+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac0af3a0 con 0x7fb0ac071480 2024-03-22T13:03:21.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.358+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac075650 con 0x7fb0ac071480 2024-03-22T13:03:21.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.385+0000 7fb0a3400640 1 -- 172.21.15.6:0/2183070236 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb09c0041d0 con 0x7fb0ac071480 2024-03-22T13:03:21.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.386+0000 7fb0a3400640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb08c06bc20 0x7fb08c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:21.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.386+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb08c06bc20 0x7fb08c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:21.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.387+0000 7fb0aaa00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb08c06bc20 0x7fb08c06e0e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac15d7b0 tx=0x7fb09803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:21.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.647+0000 7fb0a3400640 1 -- 172.21.15.6:0/2183070236 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb09c0576b0 con 0x7fb0ac071480 2024-03-22T13:03:21.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.666+0000 7fb0a3400640 1 -- 172.21.15.6:0/2183070236 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb09c017020 con 0x7fb0ac071480 2024-03-22T13:03:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.941+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0ac0c8da0 con 0x7fb0ac071480 2024-03-22T13:03:21.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.957+0000 7fb0a3400640 1 -- 172.21.15.6:0/2183070236 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fb09c059610 con 0x7fb0ac071480 2024-03-22T13:03:21.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:21.959 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:21.959 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:21.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.960+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb08c06bc20 msgr2=0x7fb08c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.960+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb08c06bc20 0x7fb08c06e0e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac15d7b0 tx=0x7fb09803b040 comp rx=0 tx=0).stop 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.960+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac071480 msgr2=0x7fb0ac0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.960+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac071480 0x7fb0ac0cda70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00e900 tx=0x7fb09c00edc0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 shutdown_connections 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0ac15a1f0 0x7fb0ac140220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb08c06bc20 0x7fb08c06e0e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0ac071860 0x7fb0ac0cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 --2- 172.21.15.6:0/2183070236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0ac071480 0x7fb0ac0cda70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 >> 172.21.15.6:0/2183070236 conn(0x7fb0ac06b4a0 msgr2=0x7fb0ac158a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 shutdown_connections 2024-03-22T13:03:21.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:21.961+0000 7fb0b0c00640 1 -- 172.21.15.6:0/2183070236 wait complete. 2024-03-22T13:03:22.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:22.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:23.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:23.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.774+0000 7f23c1800640 1 Processor -- start 2024-03-22T13:03:23.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.775+0000 7f23c1800640 1 -- start start 2024-03-22T13:03:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.776+0000 7f23c1800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f23bc15a1f0 0x7f23bc15dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.776+0000 7f23c1800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f23bc071860 0x7f23bc15e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.776+0000 7f23c1800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23bc15a6a0 con 0x7f23bc071480 2024-03-22T13:03:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.776+0000 7f23c1800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc15a980 con 0x7f23bc071860 2024-03-22T13:03:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.776+0000 7f23c1800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc15ac60 con 0x7f23bc15a1f0 2024-03-22T13:03:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.777+0000 7f23bb400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f23bc15a1f0 0x7f23bc15dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.778+0000 7f23bb400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f23bc15a1f0 0x7f23bc15dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54232/0 (socket says 172.21.15.6:54232) 2024-03-22T13:03:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.778+0000 7f23bb400640 1 -- 172.21.15.6:0/3020363226 learned_addr learned my addr 172.21.15.6:0/3020363226 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.781+0000 7f23bb400640 1 -- 172.21.15.6:0/3020363226 >> v2:172.21.15.6:3300/0 conn(0x7f23bc071860 msgr2=0x7f23bc15e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.781+0000 7f23bb400640 1 --2- 172.21.15.6:0/3020363226 >> v2:172.21.15.6:3300/0 conn(0x7f23bc071860 0x7f23bc15e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.781+0000 7f23bb400640 1 -- 172.21.15.6:0/3020363226 >> v1:172.21.15.6:6789/0 conn(0x7f23bc071480 legacy=0x7f23bc06cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.781+0000 7f23bb400640 1 -- 172.21.15.6:0/3020363226 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc15af40 con 0x7f23bc15a1f0 2024-03-22T13:03:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.784+0000 7f23bb400640 1 --2- 172.21.15.6:0/3020363226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 0x7f23bc15dea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0b3100 tx=0x7f23a802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=597f172d679a5e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.790+0000 7f23baa00640 1 -- 172.21.15.6:0/3020363226 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8034070 con 0x7f23bc15a1f0 2024-03-22T13:03:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.790+0000 7f23baa00640 1 -- 172.21.15.6:0/3020363226 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8030860 con 0x7f23bc15a1f0 2024-03-22T13:03:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.790+0000 7f23baa00640 1 -- 172.21.15.6:0/3020363226 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8030c30 con 0x7f23bc15a1f0 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.791+0000 7f23c1800640 1 -- 172.21.15.6:0/3020363226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 msgr2=0x7f23bc15dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.791+0000 7f23c1800640 1 --2- 172.21.15.6:0/3020363226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 0x7f23bc15dea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0b3100 tx=0x7f23a802b9a0 comp rx=0 tx=0).stop 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.791+0000 7f23c1800640 1 -- 172.21.15.6:0/3020363226 shutdown_connections 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.791+0000 7f23c1800640 1 --2- 172.21.15.6:0/3020363226 >> v2:172.21.15.6:3300/0 conn(0x7f23bc071860 0x7f23bc15e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.791+0000 7f23c1800640 1 --2- 172.21.15.6:0/3020363226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 0x7f23bc15dea0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.791+0000 7f23c1800640 1 -- 172.21.15.6:0/3020363226 >> 172.21.15.6:0/3020363226 conn(0x7f23bc06b4a0 msgr2=0x7f23bc06f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.791+0000 7f23c1800640 1 -- 172.21.15.6:0/3020363226 shutdown_connections 2024-03-22T13:03:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.792+0000 7f23c1800640 1 -- 172.21.15.6:0/3020363226 wait complete. 2024-03-22T13:03:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.793+0000 7f23c1800640 1 Processor -- start 2024-03-22T13:03:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.793+0000 7f23c1800640 1 -- start start 2024-03-22T13:03:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.794+0000 7f23c1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc071480 0x7f23bc0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.794+0000 7f23c1800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc071860 0x7f23bc1448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.794+0000 7f23c1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 0x7f23bc0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.794+0000 7f23c1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0cad90 con 0x7f23bc071480 2024-03-22T13:03:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.794+0000 7f23c1800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0cac10 con 0x7f23bc15a1f0 2024-03-22T13:03:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.794+0000 7f23c1800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0caf10 con 0x7f23bc071860 2024-03-22T13:03:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.795+0000 7f23bb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc071860 0x7f23bc1448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.796+0000 7f23bb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc071860 0x7f23bc1448e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:49526/0 (socket says 172.21.15.6:49526) 2024-03-22T13:03:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.796+0000 7f23bb400640 1 -- 172.21.15.6:0/560372959 learned_addr learned my addr 172.21.15.6:0/560372959 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:23.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.799+0000 7f23c0e00640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 0x7f23bc0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.801+0000 7f23bb400640 1 -- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 msgr2=0x7f23bc0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.801+0000 7f23bb400640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 0x7f23bc0cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.801+0000 7f23bb400640 1 -- 172.21.15.6:0/560372959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc071480 msgr2=0x7f23bc0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.801+0000 7f23bb400640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc071480 0x7f23bc0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.801+0000 7f23bb400640 1 -- 172.21.15.6:0/560372959 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0b1520 con 0x7f23bc071860 2024-03-22T13:03:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.805+0000 7f23bb400640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc071860 0x7f23bc1448e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f23a8000c00 tx=0x7f23a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:23.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.809+0000 7f23b3e00640 1 -- 172.21.15.6:0/560372959 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8034040 con 0x7f23bc071860 2024-03-22T13:03:23.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.809+0000 7f23b3e00640 1 -- 172.21.15.6:0/560372959 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8039d90 con 0x7f23bc071860 2024-03-22T13:03:23.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.809+0000 7f23b3e00640 1 -- 172.21.15.6:0/560372959 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8043630 con 0x7f23bc071860 2024-03-22T13:03:23.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.809+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc0ce6c0 con 0x7f23bc071860 2024-03-22T13:03:23.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.809+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc0b1aa0 con 0x7f23bc071860 2024-03-22T13:03:23.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:23.810+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc0b2560 con 0x7f23bc071860 2024-03-22T13:03:24.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.087+0000 7f23b3e00640 1 -- 172.21.15.6:0/560372959 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f23a804a060 con 0x7f23bc071860 2024-03-22T13:03:24.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.090+0000 7f23b3e00640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f239c06bc20 0x7f239c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:24.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.090+0000 7f23bbe00640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f239c06bc20 0x7f239c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:24.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.091+0000 7f23bbe00640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f239c06bc20 0x7f239c06e0e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001560 tx=0x7f23ac007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:24.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.092+0000 7f23b3e00640 1 -- 172.21.15.6:0/560372959 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f23a8079560 con 0x7f23bc071860 2024-03-22T13:03:24.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.111+0000 7f23b3e00640 1 -- 172.21.15.6:0/560372959 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f23bc0b2560 con 0x7f23bc071860 2024-03-22T13:03:24.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.377+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23bc075650 con 0x7f23bc071860 2024-03-22T13:03:24.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.432+0000 7f23b3e00640 1 -- 172.21.15.6:0/560372959 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f23a803e070 con 0x7f23bc071860 2024-03-22T13:03:24.433 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:24.433 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:24.434 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:24.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f239c06bc20 msgr2=0x7f239c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:24.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f239c06bc20 0x7f239c06e0e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001560 tx=0x7f23ac007990 comp rx=0 tx=0).stop 2024-03-22T13:03:24.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc071860 msgr2=0x7f23bc1448e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:24.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc071860 0x7f23bc1448e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f23a8000c00 tx=0x7f23a80046c0 comp rx=0 tx=0).stop 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 shutdown_connections 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f239c06bc20 0x7f239c06e0e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23bc15a1f0 0x7f23bc0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23bc071860 0x7f23bc1448e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 --2- 172.21.15.6:0/560372959 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23bc071480 0x7f23bc0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 >> 172.21.15.6:0/560372959 conn(0x7f23bc06b4a0 msgr2=0x7f23bc1592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 shutdown_connections 2024-03-22T13:03:24.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:24.435+0000 7f23c1800640 1 -- 172.21.15.6:0/560372959 wait complete. 2024-03-22T13:03:25.057 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:25.057 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:26.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:26.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.241+0000 7fdf61e00640 1 Processor -- start 2024-03-22T13:03:26.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.241+0000 7fdf61e00640 1 -- start start 2024-03-22T13:03:26.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.243+0000 7fdf61e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf5c0cc380 0x7fdf5c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:26.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.243+0000 7fdf61e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdf5c15ae30 0x7fdf5c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:26.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.243+0000 7fdf61e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c15b240 con 0x7fdf5c0ce8a0 2024-03-22T13:03:26.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.243+0000 7fdf61e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c0cee00 con 0x7fdf5c15ae30 2024-03-22T13:03:26.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.243+0000 7fdf61e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c0cf0c0 con 0x7fdf5c0cc380 2024-03-22T13:03:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.244+0000 7fdf60a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf5c0cc380 0x7fdf5c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.244+0000 7fdf60a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf5c0cc380 0x7fdf5c06c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54268/0 (socket says 172.21.15.6:54268) 2024-03-22T13:03:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.244+0000 7fdf60a00640 1 -- 172.21.15.6:0/2552915187 learned_addr learned my addr 172.21.15.6:0/2552915187 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.244+0000 7fdf61400640 1 --2- 172.21.15.6:0/2552915187 >> v2:172.21.15.6:3300/0 conn(0x7fdf5c15ae30 0x7fdf5c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.246+0000 7fdf60a00640 1 -- 172.21.15.6:0/2552915187 >> v2:172.21.15.6:3300/0 conn(0x7fdf5c15ae30 msgr2=0x7fdf5c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.246+0000 7fdf60a00640 1 --2- 172.21.15.6:0/2552915187 >> v2:172.21.15.6:3300/0 conn(0x7fdf5c15ae30 0x7fdf5c0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.246+0000 7fdf60a00640 1 -- 172.21.15.6:0/2552915187 >> v1:172.21.15.6:6789/0 conn(0x7fdf5c0ce8a0 legacy=0x7fdf5c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.246+0000 7fdf60a00640 1 -- 172.21.15.6:0/2552915187 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0cf3a0 con 0x7fdf5c0cc380 2024-03-22T13:03:26.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.248+0000 7fdf60a00640 1 --2- 172.21.15.6:0/2552915187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c06c230 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c08d0e0 tx=0x7fdf5002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=782dbbd4a4580e17 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.251+0000 7fdf5b400640 1 -- 172.21.15.6:0/2552915187 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50034070 con 0x7fdf5c0cc380 2024-03-22T13:03:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.252+0000 7fdf5b400640 1 -- 172.21.15.6:0/2552915187 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50030860 con 0x7fdf5c0cc380 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 -- 172.21.15.6:0/2552915187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 msgr2=0x7fdf5c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 --2- 172.21.15.6:0/2552915187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c06c230 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c08d0e0 tx=0x7fdf5002b9a0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 -- 172.21.15.6:0/2552915187 shutdown_connections 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 --2- 172.21.15.6:0/2552915187 >> v2:172.21.15.6:3300/0 conn(0x7fdf5c15ae30 0x7fdf5c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 --2- 172.21.15.6:0/2552915187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c06c230 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 -- 172.21.15.6:0/2552915187 >> 172.21.15.6:0/2552915187 conn(0x7fdf5c0b1480 msgr2=0x7fdf5c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 -- 172.21.15.6:0/2552915187 shutdown_connections 2024-03-22T13:03:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.253+0000 7fdf61e00640 1 -- 172.21.15.6:0/2552915187 wait complete. 2024-03-22T13:03:26.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.255+0000 7fdf61e00640 1 Processor -- start 2024-03-22T13:03:26.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.255+0000 7fdf61e00640 1 -- start start 2024-03-22T13:03:26.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.256+0000 7fdf61e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c1462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:26.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.256+0000 7fdf61e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c0ce8a0 0x7fdf5c1467e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:26.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.256+0000 7fdf61e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf5c15ae30 0x7fdf5c1f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:26.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.256+0000 7fdf61e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c15a910 con 0x7fdf5c15ae30 2024-03-22T13:03:26.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.256+0000 7fdf61e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c15a630 con 0x7fdf5c0cc380 2024-03-22T13:03:26.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.256+0000 7fdf61e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c1549d0 con 0x7fdf5c0ce8a0 2024-03-22T13:03:26.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.256+0000 7fdf60a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c1462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:26.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.257+0000 7fdf5be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c0ce8a0 0x7fdf5c1467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:26.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.257+0000 7fdf60a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c1462a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54276/0 (socket says 172.21.15.6:54276) 2024-03-22T13:03:26.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.257+0000 7fdf60a00640 1 -- 172.21.15.6:0/3980737364 learned_addr learned my addr 172.21.15.6:0/3980737364 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:26.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.257+0000 7fdf61400640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf5c15ae30 0x7fdf5c1f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:26.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.260+0000 7fdf60a00640 1 -- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c0ce8a0 msgr2=0x7fdf5c1467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:26.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.260+0000 7fdf60a00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c0ce8a0 0x7fdf5c1467e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.260+0000 7fdf60a00640 1 -- 172.21.15.6:0/3980737364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf5c15ae30 msgr2=0x7fdf5c1f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:26.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.260+0000 7fdf60a00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf5c15ae30 0x7fdf5c1f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.260+0000 7fdf60a00640 1 -- 172.21.15.6:0/3980737364 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0b5600 con 0x7fdf5c0cc380 2024-03-22T13:03:26.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.263+0000 7fdf60a00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c1462a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdf5002b9e0 tx=0x7fdf500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.267+0000 7fdf58c00640 1 -- 172.21.15.6:0/3980737364 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50034040 con 0x7fdf5c0cc380 2024-03-22T13:03:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.267+0000 7fdf58c00640 1 -- 172.21.15.6:0/3980737364 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50003820 con 0x7fdf5c0cc380 2024-03-22T13:03:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.267+0000 7fdf58c00640 1 -- 172.21.15.6:0/3980737364 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50003bf0 con 0x7fdf5c0cc380 2024-03-22T13:03:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.267+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c15e060 con 0x7fdf5c0cc380 2024-03-22T13:03:26.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.267+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c146de0 con 0x7fdf5c0cc380 2024-03-22T13:03:26.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.268+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c0cccf0 con 0x7fdf5c0cc380 2024-03-22T13:03:26.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.300+0000 7fdf58c00640 1 -- 172.21.15.6:0/3980737364 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf5004a060 con 0x7fdf5c0cc380 2024-03-22T13:03:26.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.301+0000 7fdf58c00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf3c06bc50 0x7fdf3c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:26.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.302+0000 7fdf5be00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf3c06bc50 0x7fdf3c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:26.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.303+0000 7fdf5be00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf3c06bc50 0x7fdf3c06e110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c06d9b0 tx=0x7fdf48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:26.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.306+0000 7fdf58c00640 1 -- 172.21.15.6:0/3980737364 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdf500793f0 con 0x7fdf5c0cc380 2024-03-22T13:03:26.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.560+0000 7fdf58c00640 1 -- 172.21.15.6:0/3980737364 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf5c0cccf0 con 0x7fdf5c0cc380 2024-03-22T13:03:26.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.809+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf5c074f60 con 0x7fdf5c0cc380 2024-03-22T13:03:26.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.823+0000 7fdf58c00640 1 -- 172.21.15.6:0/3980737364 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fdf5c074f60 con 0x7fdf5c0cc380 2024-03-22T13:03:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:26.824 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf3c06bc50 msgr2=0x7fdf3c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf3c06bc50 0x7fdf3c06e110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c06d9b0 tx=0x7fdf48009040 comp rx=0 tx=0).stop 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 msgr2=0x7fdf5c1462a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c1462a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdf5002b9e0 tx=0x7fdf500046c0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 shutdown_connections 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf5c15ae30 0x7fdf5c1f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c0ce8a0 0x7fdf5c1467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf3c06bc50 0x7fdf3c06e110 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 --2- 172.21.15.6:0/3980737364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c0cc380 0x7fdf5c1462a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 >> 172.21.15.6:0/3980737364 conn(0x7fdf5c0b1480 msgr2=0x7fdf5c153ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 shutdown_connections 2024-03-22T13:03:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:26.826+0000 7fdf61e00640 1 -- 172.21.15.6:0/3980737364 wait complete. 2024-03-22T13:03:27.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:27.449 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:28.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.654+0000 7ff345000640 1 Processor -- start 2024-03-22T13:03:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.654+0000 7ff345000640 1 -- start start 2024-03-22T13:03:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.655+0000 7ff345000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff340154490 0x7ff34006fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.655+0000 7ff345000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff34006c720 0x7ff3400c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.656+0000 7ff345000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3400c75f0 con 0x7ff34006c210 2024-03-22T13:03:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.656+0000 7ff345000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff340070100 con 0x7ff340154490 2024-03-22T13:03:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.656+0000 7ff345000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3400708c0 con 0x7ff34006c720 2024-03-22T13:03:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.657+0000 7ff33d400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff34006c720 0x7ff3400c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.657+0000 7ff33e800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff34006c210 0x7ff34015d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35140/0 (socket says 172.21.15.6:35140) 2024-03-22T13:03:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.657+0000 7ff33e800640 1 -- 172.21.15.6:0/3891876096 learned_addr learned my addr 172.21.15.6:0/3891876096 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.661+0000 7ff33d400640 1 -- 172.21.15.6:0/3891876096 >> v2:172.21.15.6:3300/0 conn(0x7ff340154490 msgr2=0x7ff34006fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.661+0000 7ff33d400640 1 --2- 172.21.15.6:0/3891876096 >> v2:172.21.15.6:3300/0 conn(0x7ff340154490 0x7ff34006fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.661+0000 7ff33d400640 1 -- 172.21.15.6:0/3891876096 >> v1:172.21.15.92:6789/0 conn(0x7ff34006c210 legacy=0x7ff34015d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.661+0000 7ff33d400640 1 -- 172.21.15.6:0/3891876096 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34006b1b0 con 0x7ff34006c720 2024-03-22T13:03:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.665+0000 7ff33d400640 1 --2- 172.21.15.6:0/3891876096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff34006c720 0x7ff3400c70b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff340154b70 tx=0x7ff32c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f38a03a205f84b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.669+0000 7ff33ca00640 1 -- 172.21.15.6:0/3891876096 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c035020 con 0x7ff34006c720 2024-03-22T13:03:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.669+0000 7ff33ca00640 1 -- 172.21.15.6:0/3891876096 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c031830 con 0x7ff34006c720 2024-03-22T13:03:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.669+0000 7ff33ca00640 1 -- 172.21.15.6:0/3891876096 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c031c00 con 0x7ff34006c720 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 -- 172.21.15.6:0/3891876096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff34006c720 msgr2=0x7ff3400c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 --2- 172.21.15.6:0/3891876096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff34006c720 0x7ff3400c70b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff340154b70 tx=0x7ff32c02b8d0 comp rx=0 tx=0).stop 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 -- 172.21.15.6:0/3891876096 shutdown_connections 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 --2- 172.21.15.6:0/3891876096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff34006c720 0x7ff3400c70b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 --2- 172.21.15.6:0/3891876096 >> v2:172.21.15.6:3300/0 conn(0x7ff340154490 0x7ff34006fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 -- 172.21.15.6:0/3891876096 >> 172.21.15.6:0/3891876096 conn(0x7ff3400c8cf0 msgr2=0x7ff3400c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 -- 172.21.15.6:0/3891876096 shutdown_connections 2024-03-22T13:03:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.670+0000 7ff345000640 1 -- 172.21.15.6:0/3891876096 wait complete. 2024-03-22T13:03:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.672+0000 7ff345000640 1 Processor -- start 2024-03-22T13:03:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.672+0000 7ff345000640 1 -- start start 2024-03-22T13:03:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.673+0000 7ff345000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34006c210 0x7ff34014f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.673+0000 7ff345000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff34006c720 0x7ff34015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.673+0000 7ff345000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff340154490 0x7ff3401f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.673+0000 7ff345000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34006ba10 con 0x7ff34006c210 2024-03-22T13:03:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.673+0000 7ff345000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34006b730 con 0x7ff340154490 2024-03-22T13:03:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.673+0000 7ff345000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34006bcf0 con 0x7ff34006c720 2024-03-22T13:03:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.674+0000 7ff33d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff34006c720 0x7ff34015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:28.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.674+0000 7ff33d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff34006c720 0x7ff34015b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:49540/0 (socket says 172.21.15.6:49540) 2024-03-22T13:03:28.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.674+0000 7ff33d400640 1 -- 172.21.15.6:0/846031691 learned_addr learned my addr 172.21.15.6:0/846031691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:28.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.674+0000 7ff33e800640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff340154490 0x7ff3401f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:28.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.680+0000 7ff33e800640 1 -- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff34006c720 msgr2=0x7ff34015b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:28.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.680+0000 7ff33e800640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff34006c720 0x7ff34015b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:28.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.680+0000 7ff33e800640 1 -- 172.21.15.6:0/846031691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34006c210 msgr2=0x7ff34014f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:28.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.680+0000 7ff33e800640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34006c210 0x7ff34014f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:28.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.680+0000 7ff33e800640 1 -- 172.21.15.6:0/846031691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3401e3b20 con 0x7ff340154490 2024-03-22T13:03:28.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.683+0000 7ff33e800640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff340154490 0x7ff3401f3a80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff33800c3f0 tx=0x7ff33800c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:28.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.689+0000 7ff336000640 1 -- 172.21.15.6:0/846031691 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338017070 con 0x7ff340154490 2024-03-22T13:03:28.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.689+0000 7ff336000640 1 -- 172.21.15.6:0/846031691 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff338005ce0 con 0x7ff340154490 2024-03-22T13:03:28.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.689+0000 7ff336000640 1 -- 172.21.15.6:0/846031691 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3380072d0 con 0x7ff340154490 2024-03-22T13:03:28.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.689+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3401e3e00 con 0x7ff340154490 2024-03-22T13:03:28.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.689+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3401e5860 con 0x7ff340154490 2024-03-22T13:03:28.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.690+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3400756c0 con 0x7ff340154490 2024-03-22T13:03:28.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.724+0000 7ff336000640 1 -- 172.21.15.6:0/846031691 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff33801a070 con 0x7ff340154490 2024-03-22T13:03:28.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.725+0000 7ff336000640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff32006bc20 0x7ff32006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:28.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.726+0000 7ff33de00640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff32006bc20 0x7ff32006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:28.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.727+0000 7ff33de00640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff32006bc20 0x7ff32006e0e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff33000ab90 tx=0x7ff330009250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:28.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:28.728+0000 7ff336000640 1 -- 172.21.15.6:0/846031691 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff3380566e0 con 0x7ff340154490 2024-03-22T13:03:29.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.094+0000 7ff336000640 1 -- 172.21.15.6:0/846031691 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff338058640 con 0x7ff340154490 2024-03-22T13:03:29.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.342+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3400c4530 con 0x7ff340154490 2024-03-22T13:03:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.354+0000 7ff336000640 1 -- 172.21.15.6:0/846031691 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7ff33805c6e0 con 0x7ff340154490 2024-03-22T13:03:29.354 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:29.355 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff32006bc20 msgr2=0x7ff32006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff32006bc20 0x7ff32006e0e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff33000ab90 tx=0x7ff330009250 comp rx=0 tx=0).stop 2024-03-22T13:03:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff340154490 msgr2=0x7ff3401f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff340154490 0x7ff3401f3a80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff33800c3f0 tx=0x7ff33800c8b0 comp rx=0 tx=0).stop 2024-03-22T13:03:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 shutdown_connections 2024-03-22T13:03:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff32006bc20 0x7ff32006e0e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff340154490 0x7ff3401f3a80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff34006c720 0x7ff34015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 --2- 172.21.15.6:0/846031691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff34006c210 0x7ff34014f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 >> 172.21.15.6:0/846031691 conn(0x7ff3400c8cf0 msgr2=0x7ff3400c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 shutdown_connections 2024-03-22T13:03:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:29.357+0000 7ff345000640 1 -- 172.21.15.6:0/846031691 wait complete. 2024-03-22T13:03:29.977 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:29.978 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:30.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:31.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:03:31.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:03:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.164+0000 7f3f58a00640 1 Processor -- start 2024-03-22T13:03:31.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.164+0000 7f3f58a00640 1 -- start start 2024-03-22T13:03:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.165+0000 7f3f58a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f540cc380 0x7f3f5406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.165+0000 7f3f58a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5415b240 con 0x7f3f540ce8a0 2024-03-22T13:03:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.165+0000 7f3f58a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f540cee00 con 0x7f3f5415ae30 2024-03-22T13:03:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.165+0000 7f3f58a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f540cf0c0 con 0x7f3f540cc380 2024-03-22T13:03:31.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.166+0000 7f3f53200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3f5415ae30 0x7f3f54164b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35152/0 (socket says 172.21.15.6:35152) 2024-03-22T13:03:31.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.166+0000 7f3f53200640 1 -- 172.21.15.6:0/3239196982 learned_addr learned my addr 172.21.15.6:0/3239196982 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:31.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.166+0000 7f3f52800640 1 --2- 172.21.15.6:0/3239196982 >> v2:172.21.15.6:3300/0 conn(0x7f3f540cc380 0x7f3f5406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:31.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.169+0000 7f3f51400640 1 -- 172.21.15.6:0/3239196982 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699849841 0 0) 0x7f3f540cee00 con 0x7f3f5415ae30 2024-03-22T13:03:31.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.170+0000 7f3f51400640 1 -- 172.21.15.6:0/3239196982 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f38003520 con 0x7f3f5415ae30 2024-03-22T13:03:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.171+0000 7f3f52800640 1 -- 172.21.15.6:0/3239196982 >> v1:172.21.15.92:6789/0 conn(0x7f3f5415ae30 legacy=0x7f3f54164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.171+0000 7f3f52800640 1 -- 172.21.15.6:0/3239196982 >> v1:172.21.15.6:6789/0 conn(0x7f3f540ce8a0 legacy=0x7f3f540cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:31.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.172+0000 7f3f52800640 1 -- 172.21.15.6:0/3239196982 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540cf3a0 con 0x7f3f540cc380 2024-03-22T13:03:31.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.176+0000 7f3f52800640 1 --2- 172.21.15.6:0/3239196982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 0x7f3f5406c230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f440097a0 tx=0x7f3f4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585a18ed24ce1e61 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:31.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.180+0000 7f3f51400640 1 -- 172.21.15.6:0/3239196982 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44035020 con 0x7f3f540cc380 2024-03-22T13:03:31.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.180+0000 7f3f51400640 1 -- 172.21.15.6:0/3239196982 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44031830 con 0x7f3f540cc380 2024-03-22T13:03:31.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.180+0000 7f3f51400640 1 -- 172.21.15.6:0/3239196982 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44031c00 con 0x7f3f540cc380 2024-03-22T13:03:31.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.181+0000 7f3f58a00640 1 -- 172.21.15.6:0/3239196982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 msgr2=0x7f3f5406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:31.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.181+0000 7f3f58a00640 1 --2- 172.21.15.6:0/3239196982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 0x7f3f5406c230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f440097a0 tx=0x7f3f4402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.181+0000 7f3f58a00640 1 -- 172.21.15.6:0/3239196982 shutdown_connections 2024-03-22T13:03:31.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.181+0000 7f3f58a00640 1 --2- 172.21.15.6:0/3239196982 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 0x7f3f5406c230 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.181+0000 7f3f58a00640 1 -- 172.21.15.6:0/3239196982 >> 172.21.15.6:0/3239196982 conn(0x7f3f540b1480 msgr2=0x7f3f540b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:31.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.182+0000 7f3f58a00640 1 -- 172.21.15.6:0/3239196982 shutdown_connections 2024-03-22T13:03:31.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.182+0000 7f3f58a00640 1 -- 172.21.15.6:0/3239196982 wait complete. 2024-03-22T13:03:31.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.184+0000 7f3f58a00640 1 Processor -- start 2024-03-22T13:03:31.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.184+0000 7f3f58a00640 1 -- start start 2024-03-22T13:03:31.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.185+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 0x7f3f54149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:31.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.185+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f540ce8a0 0x7f3f540c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:31.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.185+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f5415ae30 0x7f3f54146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:31.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.185+0000 7f3f58a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5415a910 con 0x7f3f540cc380 2024-03-22T13:03:31.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.185+0000 7f3f58a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5415a630 con 0x7f3f540ce8a0 2024-03-22T13:03:31.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.185+0000 7f3f58a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f541549d0 con 0x7f3f5415ae30 2024-03-22T13:03:31.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.186+0000 7f3f51e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f540ce8a0 0x7f3f540c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:31.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.186+0000 7f3f53200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f5415ae30 0x7f3f54146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:31.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.186+0000 7f3f52800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 0x7f3f54149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:31.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.187+0000 7f3f51e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f540ce8a0 0x7f3f540c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54302/0 (socket says 172.21.15.6:54302) 2024-03-22T13:03:31.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.187+0000 7f3f51e00640 1 -- 172.21.15.6:0/1939627448 learned_addr learned my addr 172.21.15.6:0/1939627448 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.192+0000 7f3f53200640 1 -- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f540ce8a0 msgr2=0x7f3f540c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.192+0000 7f3f53200640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f540ce8a0 0x7f3f540c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.192+0000 7f3f53200640 1 -- 172.21.15.6:0/1939627448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 msgr2=0x7f3f54149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.192+0000 7f3f53200640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 0x7f3f54149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.192+0000 7f3f53200640 1 -- 172.21.15.6:0/1939627448 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540c9440 con 0x7f3f5415ae30 2024-03-22T13:03:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.195+0000 7f3f53200640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f5415ae30 0x7f3f54146ce0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00caa0 tx=0x7f3f4c00cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:31.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.199+0000 7f3f4aa00640 1 -- 172.21.15.6:0/1939627448 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c017070 con 0x7f3f5415ae30 2024-03-22T13:03:31.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.200+0000 7f3f4aa00640 1 -- 172.21.15.6:0/1939627448 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4c00aba0 con 0x7f3f5415ae30 2024-03-22T13:03:31.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.200+0000 7f3f4aa00640 1 -- 172.21.15.6:0/1939627448 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c012400 con 0x7f3f5415ae30 2024-03-22T13:03:31.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.200+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f540c9170 con 0x7f3f5415ae30 2024-03-22T13:03:31.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.200+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f540c99e0 con 0x7f3f5415ae30 2024-03-22T13:03:31.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.202+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f54075650 con 0x7f3f5415ae30 2024-03-22T13:03:31.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.234+0000 7f3f4aa00640 1 -- 172.21.15.6:0/1939627448 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f4c01a070 con 0x7f3f5415ae30 2024-03-22T13:03:31.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.236+0000 7f3f4aa00640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f3406bc20 0x7f3f3406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:31.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.236+0000 7f3f52800640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f3406bc20 0x7f3f3406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:31.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.237+0000 7f3f52800640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f3406bc20 0x7f3f3406e0e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f44036000 tx=0x7f3f4403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:31.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.476+0000 7f3f4aa00640 1 -- 172.21.15.6:0/1939627448 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3f4c0570a0 con 0x7f3f5415ae30 2024-03-22T13:03:31.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.497+0000 7f3f4aa00640 1 -- 172.21.15.6:0/1939627448 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f4c021020 con 0x7f3f5415ae30 2024-03-22T13:03:31.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.744+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f540cc760 con 0x7f3f5415ae30 2024-03-22T13:03:31.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.764+0000 7f3f4aa00640 1 -- 172.21.15.6:0/1939627448 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f3f4c059000 con 0x7f3f5415ae30 2024-03-22T13:03:31.764 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:31.764 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:31.765 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:31.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f3406bc20 msgr2=0x7f3f3406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:31.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f3406bc20 0x7f3f3406e0e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f44036000 tx=0x7f3f4403c040 comp rx=0 tx=0).stop 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f5415ae30 msgr2=0x7f3f54146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f5415ae30 0x7f3f54146ce0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00caa0 tx=0x7f3f4c00cf60 comp rx=0 tx=0).stop 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 shutdown_connections 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f5415ae30 0x7f3f54146ce0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f3406bc20 0x7f3f3406e0e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f540ce8a0 0x7f3f540c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 --2- 172.21.15.6:0/1939627448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f540cc380 0x7f3f54149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 >> 172.21.15.6:0/1939627448 conn(0x7f3f540b1480 msgr2=0x7f3f5406c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.767+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 shutdown_connections 2024-03-22T13:03:31.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:31.768+0000 7f3f58a00640 1 -- 172.21.15.6:0/1939627448 wait complete. 2024-03-22T13:03:32.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:32.390 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:33.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.584+0000 7f5354400640 1 Processor -- start 2024-03-22T13:03:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.584+0000 7f5354400640 1 -- start start 2024-03-22T13:03:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.586+0000 7f5354400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c06f8c0 0x7f534c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.586+0000 7f5354400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f534c071c10 0x7f534c0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.586+0000 7f5354400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c1590b0 con 0x7f534c152400 2024-03-22T13:03:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.586+0000 7f5354400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534c0724d0 con 0x7f534c071c10 2024-03-22T13:03:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.586+0000 7f5354400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0721f0 con 0x7f534c06f8c0 2024-03-22T13:03:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.587+0000 7f5351000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c06f8c0 0x7f534c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.587+0000 7f5352400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f534c071c10 0x7f534c0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.587+0000 7f5351a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f534c152400 0x7f534c15b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:58508/0 (socket says 172.21.15.6:58508) 2024-03-22T13:03:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.587+0000 7f5351a00640 1 -- 172.21.15.6:0/2169340750 learned_addr learned my addr 172.21.15.6:0/2169340750 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:33.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.590+0000 7f5351000640 1 -- 172.21.15.6:0/2169340750 >> v2:172.21.15.92:3300/0 conn(0x7f534c071c10 msgr2=0x7f534c0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:33.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.590+0000 7f5351000640 1 --2- 172.21.15.6:0/2169340750 >> v2:172.21.15.92:3300/0 conn(0x7f534c071c10 0x7f534c0c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.590+0000 7f5351000640 1 -- 172.21.15.6:0/2169340750 >> v1:172.21.15.92:6789/0 conn(0x7f534c152400 legacy=0x7f534c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.590+0000 7f5351000640 1 -- 172.21.15.6:0/2169340750 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c1582f0 con 0x7f534c06f8c0 2024-03-22T13:03:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.593+0000 7f5351000640 1 --2- 172.21.15.6:0/2169340750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c06f8c0 0x7f534c0716d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f53380097a0 tx=0x7f533802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d0e54debb940b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.596+0000 7f534be00640 1 -- 172.21.15.6:0/2169340750 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338034070 con 0x7f534c06f8c0 2024-03-22T13:03:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.598+0000 7f534be00640 1 -- 172.21.15.6:0/2169340750 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5338030610 con 0x7f534c06f8c0 2024-03-22T13:03:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.598+0000 7f534be00640 1 -- 172.21.15.6:0/2169340750 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53380309e0 con 0x7f534c06f8c0 2024-03-22T13:03:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.599+0000 7f5354400640 1 -- 172.21.15.6:0/2169340750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c06f8c0 msgr2=0x7f534c0716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.599+0000 7f5354400640 1 --2- 172.21.15.6:0/2169340750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c06f8c0 0x7f534c0716d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f53380097a0 tx=0x7f533802b960 comp rx=0 tx=0).stop 2024-03-22T13:03:33.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.599+0000 7f5354400640 1 -- 172.21.15.6:0/2169340750 shutdown_connections 2024-03-22T13:03:33.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.599+0000 7f5354400640 1 --2- 172.21.15.6:0/2169340750 >> v2:172.21.15.92:3300/0 conn(0x7f534c071c10 0x7f534c0c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:33.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.599+0000 7f5354400640 1 --2- 172.21.15.6:0/2169340750 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c06f8c0 0x7f534c0716d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:33.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.599+0000 7f5354400640 1 -- 172.21.15.6:0/2169340750 >> 172.21.15.6:0/2169340750 conn(0x7f534c0cb930 msgr2=0x7f534c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:33.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.600+0000 7f5354400640 1 -- 172.21.15.6:0/2169340750 shutdown_connections 2024-03-22T13:03:33.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.600+0000 7f5354400640 1 -- 172.21.15.6:0/2169340750 wait complete. 2024-03-22T13:03:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.602+0000 7f5354400640 1 Processor -- start 2024-03-22T13:03:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.602+0000 7f5354400640 1 -- start start 2024-03-22T13:03:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.604+0000 7f5354400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f534c06f8c0 0x7f534c14c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.604+0000 7f5354400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f534c071c10 0x7f534c141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.604+0000 7f5354400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c152400 0x7f534c1417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.604+0000 7f5354400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c158b90 con 0x7f534c06f8c0 2024-03-22T13:03:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.604+0000 7f5354400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c1588b0 con 0x7f534c071c10 2024-03-22T13:03:33.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.604+0000 7f5354400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c1527e0 con 0x7f534c152400 2024-03-22T13:03:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.605+0000 7f5351000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f534c071c10 0x7f534c141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.605+0000 7f5352400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c152400 0x7f534c1417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.605+0000 7f5351a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f534c06f8c0 0x7f534c14c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.605+0000 7f5351000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f534c071c10 0x7f534c141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60096/0 (socket says 172.21.15.6:60096) 2024-03-22T13:03:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.605+0000 7f5351000640 1 -- 172.21.15.6:0/1403965914 learned_addr learned my addr 172.21.15.6:0/1403965914 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:33.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.609+0000 7f5352400640 1 -- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f534c071c10 msgr2=0x7f534c141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:33.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.609+0000 7f5352400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f534c071c10 0x7f534c141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:33.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.609+0000 7f5352400640 1 -- 172.21.15.6:0/1403965914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f534c06f8c0 msgr2=0x7f534c14c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:33.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.609+0000 7f5352400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f534c06f8c0 0x7f534c14c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:33.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.609+0000 7f5352400640 1 -- 172.21.15.6:0/1403965914 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c144cf0 con 0x7f534c152400 2024-03-22T13:03:33.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.611+0000 7f5352400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c152400 0x7f534c1417c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f534400e7f0 tx=0x7f534400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:33.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.614+0000 7f5349600640 1 -- 172.21.15.6:0/1403965914 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344017070 con 0x7f534c152400 2024-03-22T13:03:33.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.614+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534c156ff0 con 0x7f534c152400 2024-03-22T13:03:33.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.614+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534c1575b0 con 0x7f534c152400 2024-03-22T13:03:33.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.616+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534c072750 con 0x7f534c152400 2024-03-22T13:03:33.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.617+0000 7f5349600640 1 -- 172.21.15.6:0/1403965914 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5344004020 con 0x7f534c152400 2024-03-22T13:03:33.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.617+0000 7f5349600640 1 -- 172.21.15.6:0/1403965914 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344021060 con 0x7f534c152400 2024-03-22T13:03:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.907+0000 7f5349600640 1 -- 172.21.15.6:0/1403965914 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f534401a070 con 0x7f534c152400 2024-03-22T13:03:33.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.909+0000 7f5349600640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f532c06bbd0 0x7f532c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:33.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.909+0000 7f5351a00640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f532c06bbd0 0x7f532c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:33.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.911+0000 7f5349600640 1 -- 172.21.15.6:0/1403965914 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f53440572d0 con 0x7f534c152400 2024-03-22T13:03:33.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.911+0000 7f5351a00640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f532c06bbd0 0x7f532c06e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f53400012e0 tx=0x7f5340001990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:33.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:33.931+0000 7f5349600640 1 -- 172.21.15.6:0/1403965914 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f534c072750 con 0x7f534c152400 2024-03-22T13:03:34.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.214+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f534c0756c0 con 0x7f534c152400 2024-03-22T13:03:34.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.230+0000 7f5349600640 1 -- 172.21.15.6:0/1403965914 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f5344059230 con 0x7f534c152400 2024-03-22T13:03:34.231 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:34.231 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:34.231 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.232+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f532c06bbd0 msgr2=0x7f532c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.232+0000 7f5354400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f532c06bbd0 0x7f532c06e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f53400012e0 tx=0x7f5340001990 comp rx=0 tx=0).stop 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.232+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c152400 msgr2=0x7f534c1417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.232+0000 7f5354400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c152400 0x7f534c1417c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f534400e7f0 tx=0x7f534400ecb0 comp rx=0 tx=0).stop 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 shutdown_connections 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f534c152400 0x7f534c1417c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f532c06bbd0 0x7f532c06e090 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f534c071c10 0x7f534c141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:34.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 --2- 172.21.15.6:0/1403965914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f534c06f8c0 0x7f534c14c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:34.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 >> 172.21.15.6:0/1403965914 conn(0x7f534c0cb930 msgr2=0x7f534c0c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:34.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 shutdown_connections 2024-03-22T13:03:34.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:34.233+0000 7f5354400640 1 -- 172.21.15.6:0/1403965914 wait complete. 2024-03-22T13:03:34.855 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:34.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:35.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.050+0000 7f7f0c000640 1 Processor -- start 2024-03-22T13:03:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.050+0000 7f7f0c000640 1 -- start start 2024-03-22T13:03:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.051+0000 7f7f0c000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f0406f8c0 0x7f7f040716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.051+0000 7f7f0c000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7f04071c10 0x7f7f040c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.051+0000 7f7f0c000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f041590b0 con 0x7f7f04152400 2024-03-22T13:03:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.051+0000 7f7f0c000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f040724d0 con 0x7f7f04071c10 2024-03-22T13:03:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.051+0000 7f7f0c000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040721f0 con 0x7f7f0406f8c0 2024-03-22T13:03:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.052+0000 7f7f09600640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7f04152400 0x7f7f0415b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:58520/0 (socket says 172.21.15.6:58520) 2024-03-22T13:03:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.052+0000 7f7f09600640 1 -- 172.21.15.6:0/606587989 learned_addr learned my addr 172.21.15.6:0/606587989 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.052+0000 7f7f08c00640 1 --2- 172.21.15.6:0/606587989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f0406f8c0 0x7f7f040716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.052+0000 7f7f0a000640 1 --2- 172.21.15.6:0/606587989 >> v2:172.21.15.6:3300/0 conn(0x7f7f04071c10 0x7f7f040c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.055+0000 7f7f03e00640 1 -- 172.21.15.6:0/606587989 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168028359 0 0) 0x7f7f041590b0 con 0x7f7f04152400 2024-03-22T13:03:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.055+0000 7f7f03e00640 1 -- 172.21.15.6:0/606587989 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec003520 con 0x7f7f04152400 2024-03-22T13:03:36.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.056+0000 7f7f08c00640 1 -- 172.21.15.6:0/606587989 >> v2:172.21.15.6:3300/0 conn(0x7f7f04071c10 msgr2=0x7f7f040c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:36.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.056+0000 7f7f08c00640 1 --2- 172.21.15.6:0/606587989 >> v2:172.21.15.6:3300/0 conn(0x7f7f04071c10 0x7f7f040c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.056+0000 7f7f08c00640 1 -- 172.21.15.6:0/606587989 >> v1:172.21.15.92:6789/0 conn(0x7f7f04152400 legacy=0x7f7f0415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:36.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.056+0000 7f7f08c00640 1 -- 172.21.15.6:0/606587989 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f041582f0 con 0x7f7f0406f8c0 2024-03-22T13:03:36.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.059+0000 7f7f08c00640 1 --2- 172.21.15.6:0/606587989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f0406f8c0 0x7f7f040716d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00097a0 tx=0x7f7ef002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afd638e2730b1a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:36.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.062+0000 7f7f03e00640 1 -- 172.21.15.6:0/606587989 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0034070 con 0x7f7f0406f8c0 2024-03-22T13:03:36.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.064+0000 7f7f03e00640 1 -- 172.21.15.6:0/606587989 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef0030610 con 0x7f7f0406f8c0 2024-03-22T13:03:36.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.064+0000 7f7f03e00640 1 -- 172.21.15.6:0/606587989 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00309c0 con 0x7f7f0406f8c0 2024-03-22T13:03:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.065+0000 7f7f0c000640 1 -- 172.21.15.6:0/606587989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f0406f8c0 msgr2=0x7f7f040716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.065+0000 7f7f0c000640 1 --2- 172.21.15.6:0/606587989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f0406f8c0 0x7f7f040716d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00097a0 tx=0x7f7ef002b960 comp rx=0 tx=0).stop 2024-03-22T13:03:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.065+0000 7f7f0c000640 1 -- 172.21.15.6:0/606587989 shutdown_connections 2024-03-22T13:03:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.065+0000 7f7f0c000640 1 --2- 172.21.15.6:0/606587989 >> v2:172.21.15.6:3300/0 conn(0x7f7f04071c10 0x7f7f040c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.065+0000 7f7f0c000640 1 --2- 172.21.15.6:0/606587989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f0406f8c0 0x7f7f040716d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.065+0000 7f7f0c000640 1 -- 172.21.15.6:0/606587989 >> 172.21.15.6:0/606587989 conn(0x7f7f040cb930 msgr2=0x7f7f040cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.065+0000 7f7f0c000640 1 -- 172.21.15.6:0/606587989 shutdown_connections 2024-03-22T13:03:36.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.066+0000 7f7f0c000640 1 -- 172.21.15.6:0/606587989 wait complete. 2024-03-22T13:03:36.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.068+0000 7f7f0c000640 1 Processor -- start 2024-03-22T13:03:36.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.069+0000 7f7f0c000640 1 -- start start 2024-03-22T13:03:36.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.070+0000 7f7f0c000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f0406f8c0 0x7f7f040ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:36.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.070+0000 7f7f0c000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f04071c10 0x7f7f040c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:36.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.070+0000 7f7f0c000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f04152400 0x7f7f040c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:36.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.070+0000 7f7f0c000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f04158b90 con 0x7f7f0406f8c0 2024-03-22T13:03:36.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.070+0000 7f7f0c000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f041588b0 con 0x7f7f04152400 2024-03-22T13:03:36.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.070+0000 7f7f0c000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f041527e0 con 0x7f7f04071c10 2024-03-22T13:03:36.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.071+0000 7f7f0a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f04152400 0x7f7f040c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:36.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.071+0000 7f7f08c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f04071c10 0x7f7f040c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:36.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.071+0000 7f7f0a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f04152400 0x7f7f040c4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60108/0 (socket says 172.21.15.6:60108) 2024-03-22T13:03:36.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.071+0000 7f7f0a000640 1 -- 172.21.15.6:0/718011888 learned_addr learned my addr 172.21.15.6:0/718011888 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:36.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.071+0000 7f7f09600640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f0406f8c0 0x7f7f040ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.076+0000 7f7f0a000640 1 -- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f04071c10 msgr2=0x7f7f040c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.076+0000 7f7f0a000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f04071c10 0x7f7f040c3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.076+0000 7f7f0a000640 1 -- 172.21.15.6:0/718011888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f0406f8c0 msgr2=0x7f7f040ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.076+0000 7f7f0a000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f0406f8c0 0x7f7f040ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.076+0000 7f7f0a000640 1 -- 172.21.15.6:0/718011888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040c4e50 con 0x7f7f04152400 2024-03-22T13:03:36.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.080+0000 7f7f0a000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f04152400 0x7f7f040c4160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00e960 tx=0x7f7efc00ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:36.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.087+0000 7f7f01600640 1 -- 172.21.15.6:0/718011888 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc017070 con 0x7f7f04152400 2024-03-22T13:03:36.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.087+0000 7f7f01600640 1 -- 172.21.15.6:0/718011888 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc00c980 con 0x7f7f04152400 2024-03-22T13:03:36.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.087+0000 7f7f01600640 1 -- 172.21.15.6:0/718011888 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc00cd70 con 0x7f7f04152400 2024-03-22T13:03:36.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.087+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f040c32a0 con 0x7f7f04152400 2024-03-22T13:03:36.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.087+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f040ca1e0 con 0x7f7f04152400 2024-03-22T13:03:36.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.089+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f040756c0 con 0x7f7f04152400 2024-03-22T13:03:36.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.121+0000 7f7f01600640 1 -- 172.21.15.6:0/718011888 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7efc01a070 con 0x7f7f04152400 2024-03-22T13:03:36.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.123+0000 7f7f01600640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ee406bc20 0x7f7ee406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:36.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.123+0000 7f7f09600640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ee406bc20 0x7f7ee406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:36.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.124+0000 7f7f09600640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ee406bc20 0x7f7ee406e0e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005c00 tx=0x7f7ef8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:36.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.125+0000 7f7f01600640 1 -- 172.21.15.6:0/718011888 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f7efc0576d0 con 0x7f7f04152400 2024-03-22T13:03:36.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.398+0000 7f7f01600640 1 -- 172.21.15.6:0/718011888 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7efc059630 con 0x7f7f04152400 2024-03-22T13:03:36.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.654+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f04073080 con 0x7f7f04152400 2024-03-22T13:03:36.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.670+0000 7f7f01600640 1 -- 172.21.15.6:0/718011888 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f7efc05d6d0 con 0x7f7f04152400 2024-03-22T13:03:36.671 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:36.671 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:36.671 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:36.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ee406bc20 msgr2=0x7f7ee406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:36.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ee406bc20 0x7f7ee406e0e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005c00 tx=0x7f7ef8009040 comp rx=0 tx=0).stop 2024-03-22T13:03:36.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f04152400 msgr2=0x7f7f040c4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:36.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f04152400 0x7f7f040c4160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00e960 tx=0x7f7efc00ee20 comp rx=0 tx=0).stop 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 shutdown_connections 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7ee406bc20 0x7f7ee406e0e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f04152400 0x7f7f040c4160 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f04071c10 0x7f7f040c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 --2- 172.21.15.6:0/718011888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f0406f8c0 0x7f7f040ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 >> 172.21.15.6:0/718011888 conn(0x7f7f040cb930 msgr2=0x7f7f0414f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 shutdown_connections 2024-03-22T13:03:36.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:36.673+0000 7f7f0c000640 1 -- 172.21.15.6:0/718011888 wait complete. 2024-03-22T13:03:37.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:37.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:38.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:38.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.493+0000 7f5dcb000640 1 Processor -- start 2024-03-22T13:03:38.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.494+0000 7f5dcb000640 1 -- start start 2024-03-22T13:03:38.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.495+0000 7f5dcb000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5dc40ce8a0 0x7f5dc406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:38.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.496+0000 7f5dcb000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5dc415ae30 0x7f5dc40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:38.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.496+0000 7f5dcb000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc415b240 con 0x7f5dc40cc380 2024-03-22T13:03:38.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.496+0000 7f5dcb000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc40cee00 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.496+0000 7f5dcb000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc40cf0c0 con 0x7f5dc415ae30 2024-03-22T13:03:38.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.497+0000 7f5dca600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5dc415ae30 0x7f5dc40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:38.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.497+0000 7f5dc9c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5dc40cc380 0x7f5dc40cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:58536/0 (socket says 172.21.15.6:58536) 2024-03-22T13:03:38.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.497+0000 7f5dc9c00640 1 -- 172.21.15.6:0/4249449897 learned_addr learned my addr 172.21.15.6:0/4249449897 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:38.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.501+0000 7f5dca600640 1 -- 172.21.15.6:0/4249449897 >> v2:172.21.15.6:3300/0 conn(0x7f5dc40ce8a0 msgr2=0x7f5dc406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:38.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.502+0000 7f5dca600640 1 --2- 172.21.15.6:0/4249449897 >> v2:172.21.15.6:3300/0 conn(0x7f5dc40ce8a0 0x7f5dc406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:38.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.502+0000 7f5dca600640 1 -- 172.21.15.6:0/4249449897 >> v1:172.21.15.92:6789/0 conn(0x7f5dc40cc380 legacy=0x7f5dc40cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:38.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.502+0000 7f5dca600640 1 -- 172.21.15.6:0/4249449897 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc40cf3a0 con 0x7f5dc415ae30 2024-03-22T13:03:38.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.511+0000 7f5dca600640 1 --2- 172.21.15.6:0/4249449897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 0x7f5dc40cc760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40c9f50 tx=0x7f5dbc02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e40e8ebe5f4438fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.595+0000 7f5dc3e00640 1 -- 172.21.15.6:0/4249449897 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc039070 con 0x7f5dc415ae30 2024-03-22T13:03:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.595+0000 7f5dc3e00640 1 -- 172.21.15.6:0/4249449897 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc02cc80 con 0x7f5dc415ae30 2024-03-22T13:03:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.595+0000 7f5dc3e00640 1 -- 172.21.15.6:0/4249449897 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc034660 con 0x7f5dc415ae30 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 -- 172.21.15.6:0/4249449897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 msgr2=0x7f5dc40cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 --2- 172.21.15.6:0/4249449897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 0x7f5dc40cc760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40c9f50 tx=0x7f5dbc02bed0 comp rx=0 tx=0).stop 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 -- 172.21.15.6:0/4249449897 shutdown_connections 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 --2- 172.21.15.6:0/4249449897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 0x7f5dc40cc760 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 --2- 172.21.15.6:0/4249449897 >> v2:172.21.15.6:3300/0 conn(0x7f5dc40ce8a0 0x7f5dc406c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 -- 172.21.15.6:0/4249449897 >> 172.21.15.6:0/4249449897 conn(0x7f5dc40b1480 msgr2=0x7f5dc40b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 -- 172.21.15.6:0/4249449897 shutdown_connections 2024-03-22T13:03:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.596+0000 7f5dcb000640 1 -- 172.21.15.6:0/4249449897 wait complete. 2024-03-22T13:03:38.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.599+0000 7f5dcb000640 1 Processor -- start 2024-03-22T13:03:38.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.599+0000 7f5dcb000640 1 -- start start 2024-03-22T13:03:38.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dcb000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc40cc380 0x7f5dc40ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:38.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dcb000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5dc40ce8a0 0x7f5dc414e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:38.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dcb000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 0x7f5dc41f1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:38.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dcb000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc415a910 con 0x7f5dc40cc380 2024-03-22T13:03:38.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dcb000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc415a630 con 0x7f5dc415ae30 2024-03-22T13:03:38.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dcb000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc41549d0 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dc9200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5dc40ce8a0 0x7f5dc414e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.601+0000 7f5dc9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc40cc380 0x7f5dc40ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.602+0000 7f5dc9200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5dc40ce8a0 0x7f5dc414e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:52082/0 (socket says 172.21.15.6:52082) 2024-03-22T13:03:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.602+0000 7f5dc9200640 1 -- 172.21.15.6:0/3042439692 learned_addr learned my addr 172.21.15.6:0/3042439692 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.602+0000 7f5dca600640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 0x7f5dc41f1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:38.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.607+0000 7f5dc9200640 1 -- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 msgr2=0x7f5dc41f1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:38.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.607+0000 7f5dc9200640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 0x7f5dc41f1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:38.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.607+0000 7f5dc9200640 1 -- 172.21.15.6:0/3042439692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc40cc380 msgr2=0x7f5dc40ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:38.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.607+0000 7f5dc9200640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc40cc380 0x7f5dc40ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:38.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.607+0000 7f5dc9200640 1 -- 172.21.15.6:0/3042439692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc40b2fa0 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.611+0000 7f5dc9200640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5dc40ce8a0 0x7f5dc414e190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5db000b4a0 tx=0x7f5db000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.616+0000 7f5dc1600640 1 -- 172.21.15.6:0/3042439692 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0014070 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.616+0000 7f5dc1600640 1 -- 172.21.15.6:0/3042439692 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db0010610 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.616+0000 7f5dc1600640 1 -- 172.21.15.6:0/3042439692 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0010a00 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.616+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc41f4300 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.616+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc40b3540 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.618+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc40ca520 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.890+0000 7f5dc1600640 1 -- 172.21.15.6:0/3042439692 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5db0017070 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.892+0000 7f5dc1600640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5da406bc20 0x7f5da406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.893+0000 7f5dc9c00640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5da406bc20 0x7f5da406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.893+0000 7f5dc1600640 1 -- 172.21.15.6:0/3042439692 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5db008ac30 con 0x7f5dc40ce8a0 2024-03-22T13:03:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.893+0000 7f5dc9c00640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5da406bc20 0x7f5da406e0e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5db8007fd0 tx=0x7f5db8007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:38.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:38.911+0000 7f5dc1600640 1 -- 172.21.15.6:0/3042439692 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5dc40ca520 con 0x7f5dc40ce8a0 2024-03-22T13:03:39.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.170+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc4075650 con 0x7f5dc40ce8a0 2024-03-22T13:03:39.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.185+0000 7f5dc1600640 1 -- 172.21.15.6:0/3042439692 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f5db0058d20 con 0x7f5dc40ce8a0 2024-03-22T13:03:39.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:39.185 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:39.185 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5da406bc20 msgr2=0x7f5da406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5da406bc20 0x7f5da406e0e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5db8007fd0 tx=0x7f5db8007e80 comp rx=0 tx=0).stop 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5dc40ce8a0 msgr2=0x7f5dc414e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5dc40ce8a0 0x7f5dc414e190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5db000b4a0 tx=0x7f5db000b960 comp rx=0 tx=0).stop 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 shutdown_connections 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5da406bc20 0x7f5da406e0e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dc415ae30 0x7f5dc41f1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5dc40ce8a0 0x7f5dc414e190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 --2- 172.21.15.6:0/3042439692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc40cc380 0x7f5dc40ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 >> 172.21.15.6:0/3042439692 conn(0x7f5dc40b1480 msgr2=0x7f5dc4158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 shutdown_connections 2024-03-22T13:03:39.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:39.188+0000 7f5dcb000640 1 -- 172.21.15.6:0/3042439692 wait complete. 2024-03-22T13:03:39.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:39.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:40.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:41.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.000+0000 7f11daa00640 1 Processor -- start 2024-03-22T13:03:41.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.000+0000 7f11daa00640 1 -- start start 2024-03-22T13:03:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.002+0000 7f11daa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f11d4152400 0x7f11d415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.002+0000 7f11daa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d415bc80 0x7f11d40716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.002+0000 7f11daa00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d41590b0 con 0x7f11d406f8c0 2024-03-22T13:03:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.002+0000 7f11daa00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d40724d0 con 0x7f11d4152400 2024-03-22T13:03:41.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.002+0000 7f11daa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40721f0 con 0x7f11d415bc80 2024-03-22T13:03:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.003+0000 7f11d3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d415bc80 0x7f11d40716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.003+0000 7f11d8a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f11d406f8c0 0x7f11d40c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:58548/0 (socket says 172.21.15.6:58548) 2024-03-22T13:03:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.003+0000 7f11d8a00640 1 -- 172.21.15.6:0/2326304070 learned_addr learned my addr 172.21.15.6:0/2326304070 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:41.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.003+0000 7f11d3e00640 1 --2- 172.21.15.6:0/2326304070 >> v2:172.21.15.92:3300/0 conn(0x7f11d4152400 0x7f11d415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:41.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.007+0000 7f11d3e00640 1 -- 172.21.15.6:0/2326304070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d415bc80 msgr2=0x7f11d40716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:41.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.007+0000 7f11d3e00640 1 --2- 172.21.15.6:0/2326304070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d415bc80 0x7f11d40716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.007+0000 7f11d3e00640 1 -- 172.21.15.6:0/2326304070 >> v1:172.21.15.92:6789/0 conn(0x7f11d406f8c0 legacy=0x7f11d40c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:41.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.007+0000 7f11d3e00640 1 -- 172.21.15.6:0/2326304070 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d41582f0 con 0x7f11d4152400 2024-03-22T13:03:41.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.012+0000 7f11d3e00640 1 --2- 172.21.15.6:0/2326304070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d4152400 0x7f11d415b740 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f11c40097a0 tx=0x7f11c402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e6a30e981eec41 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:41.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.017+0000 7f11d2a00640 1 -- 172.21.15.6:0/2326304070 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4034070 con 0x7f11d4152400 2024-03-22T13:03:41.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.017+0000 7f11d2a00640 1 -- 172.21.15.6:0/2326304070 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c4030610 con 0x7f11d4152400 2024-03-22T13:03:41.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.017+0000 7f11d2a00640 1 -- 172.21.15.6:0/2326304070 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40309e0 con 0x7f11d4152400 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.017+0000 7f11daa00640 1 -- 172.21.15.6:0/2326304070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d4152400 msgr2=0x7f11d415b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.017+0000 7f11daa00640 1 --2- 172.21.15.6:0/2326304070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d4152400 0x7f11d415b740 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f11c40097a0 tx=0x7f11c402b960 comp rx=0 tx=0).stop 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.018+0000 7f11daa00640 1 -- 172.21.15.6:0/2326304070 shutdown_connections 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.018+0000 7f11daa00640 1 --2- 172.21.15.6:0/2326304070 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d415bc80 0x7f11d40716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.018+0000 7f11daa00640 1 --2- 172.21.15.6:0/2326304070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d4152400 0x7f11d415b740 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.018+0000 7f11daa00640 1 -- 172.21.15.6:0/2326304070 >> 172.21.15.6:0/2326304070 conn(0x7f11d40cb930 msgr2=0x7f11d40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.018+0000 7f11daa00640 1 -- 172.21.15.6:0/2326304070 shutdown_connections 2024-03-22T13:03:41.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.018+0000 7f11daa00640 1 -- 172.21.15.6:0/2326304070 wait complete. 2024-03-22T13:03:41.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.019+0000 7f11daa00640 1 Processor -- start 2024-03-22T13:03:41.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.019+0000 7f11daa00640 1 -- start start 2024-03-22T13:03:41.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.021+0000 7f11daa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11d406f8c0 0x7f11d41f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.021+0000 7f11daa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d4152400 0x7f11d40c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.021+0000 7f11daa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d415bc80 0x7f11d4148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.021+0000 7f11daa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4158b90 con 0x7f11d406f8c0 2024-03-22T13:03:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.021+0000 7f11daa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41588b0 con 0x7f11d415bc80 2024-03-22T13:03:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.021+0000 7f11daa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41527e0 con 0x7f11d4152400 2024-03-22T13:03:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.021+0000 7f11d8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d415bc80 0x7f11d4148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:41.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.022+0000 7f11d3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d4152400 0x7f11d40c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:41.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.022+0000 7f11d8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d415bc80 0x7f11d4148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60150/0 (socket says 172.21.15.6:60150) 2024-03-22T13:03:41.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.022+0000 7f11d8a00640 1 -- 172.21.15.6:0/2247835291 learned_addr learned my addr 172.21.15.6:0/2247835291 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:41.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.022+0000 7f11d3e00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11d406f8c0 0x7f11d41f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:41.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.027+0000 7f11d8a00640 1 -- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d4152400 msgr2=0x7f11d40c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:41.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.027+0000 7f11d8a00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d4152400 0x7f11d40c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.027+0000 7f11d8a00640 1 -- 172.21.15.6:0/2247835291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11d406f8c0 msgr2=0x7f11d41f0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:41.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.027+0000 7f11d8a00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11d406f8c0 0x7f11d41f0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.027+0000 7f11d8a00640 1 -- 172.21.15.6:0/2247835291 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d41f03d0 con 0x7f11d415bc80 2024-03-22T13:03:41.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.030+0000 7f11d8a00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d415bc80 0x7f11d4148130 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00cc20 tx=0x7f11cc00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:41.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.035+0000 7f11cbe00640 1 -- 172.21.15.6:0/2247835291 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc017070 con 0x7f11d415bc80 2024-03-22T13:03:41.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.035+0000 7f11cbe00640 1 -- 172.21.15.6:0/2247835291 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11cc00ad40 con 0x7f11d415bc80 2024-03-22T13:03:41.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.035+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d41f44f0 con 0x7f11d415bc80 2024-03-22T13:03:41.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.035+0000 7f11cbe00640 1 -- 172.21.15.6:0/2247835291 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc012600 con 0x7f11d415bc80 2024-03-22T13:03:41.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.035+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d40c4e00 con 0x7f11d415bc80 2024-03-22T13:03:41.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.036+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d41f01c0 con 0x7f11d415bc80 2024-03-22T13:03:41.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.061+0000 7f11cbe00640 1 -- 172.21.15.6:0/2247835291 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11cc01a070 con 0x7f11d415bc80 2024-03-22T13:03:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.062+0000 7f11cbe00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11b406bc20 0x7f11b406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.062+0000 7f11d3e00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11b406bc20 0x7f11b406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.063+0000 7f11d3e00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11b406bc20 0x7f11b406e0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f11c4036000 tx=0x7f11c403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:41.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.304+0000 7f11cbe00640 1 -- 172.21.15.6:0/2247835291 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f11cc057140 con 0x7f11d415bc80 2024-03-22T13:03:41.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.323+0000 7f11cbe00640 1 -- 172.21.15.6:0/2247835291 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11cc0590a0 con 0x7f11d415bc80 2024-03-22T13:03:41.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.574+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11d40756c0 con 0x7f11d415bc80 2024-03-22T13:03:41.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.587+0000 7f11cbe00640 1 -- 172.21.15.6:0/2247835291 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f11cc05d140 con 0x7f11d415bc80 2024-03-22T13:03:41.587 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:41.587 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:41.587 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.589+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11b406bc20 msgr2=0x7f11b406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11b406bc20 0x7f11b406e0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f11c4036000 tx=0x7f11c403b040 comp rx=0 tx=0).stop 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d415bc80 msgr2=0x7f11d4148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d415bc80 0x7f11d4148130 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00cc20 tx=0x7f11cc00a430 comp rx=0 tx=0).stop 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 shutdown_connections 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11b406bc20 0x7f11b406e0e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d415bc80 0x7f11d4148130 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11d4152400 0x7f11d40c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 --2- 172.21.15.6:0/2247835291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11d406f8c0 0x7f11d41f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 >> 172.21.15.6:0/2247835291 conn(0x7f11d40cb930 msgr2=0x7f11d41501a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 shutdown_connections 2024-03-22T13:03:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:41.590+0000 7f11daa00640 1 -- 172.21.15.6:0/2247835291 wait complete. 2024-03-22T13:03:42.211 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:42.211 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:43.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:43.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.410+0000 7f2871800640 1 Processor -- start 2024-03-22T13:03:43.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.410+0000 7f2871800640 1 -- start start 2024-03-22T13:03:43.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.411+0000 7f2871800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c152400 0x7f286c15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:43.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.411+0000 7f2871800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f286c15bc80 0x7f286c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:43.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.411+0000 7f2871800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f286c06f8c0 0x7f286c071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:43.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.411+0000 7f2871800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c0724d0 con 0x7f286c15bc80 2024-03-22T13:03:43.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.412+0000 7f2871800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c1590b0 con 0x7f286c06f8c0 2024-03-22T13:03:43.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.412+0000 7f2871800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c0721f0 con 0x7f286c152400 2024-03-22T13:03:43.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.412+0000 7f286b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f286c06f8c0 0x7f286c071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:43.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.412+0000 7f286a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f286c15bc80 0x7f286c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:43.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.412+0000 7f286aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c152400 0x7f286c15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:43.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.413+0000 7f286a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f286c15bc80 0x7f286c0716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35748/0 (socket says 172.21.15.6:35748) 2024-03-22T13:03:43.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.413+0000 7f286a000640 1 -- 172.21.15.6:0/1963863859 learned_addr learned my addr 172.21.15.6:0/1963863859 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.416+0000 7f286aa00640 1 -- 172.21.15.6:0/1963863859 >> v2:172.21.15.92:3300/0 conn(0x7f286c06f8c0 msgr2=0x7f286c071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.416+0000 7f286aa00640 1 --2- 172.21.15.6:0/1963863859 >> v2:172.21.15.92:3300/0 conn(0x7f286c06f8c0 0x7f286c071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.416+0000 7f286aa00640 1 -- 172.21.15.6:0/1963863859 >> v2:172.21.15.6:3300/0 conn(0x7f286c15bc80 msgr2=0x7f286c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.416+0000 7f286aa00640 1 --2- 172.21.15.6:0/1963863859 >> v2:172.21.15.6:3300/0 conn(0x7f286c15bc80 0x7f286c0716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.416+0000 7f286aa00640 1 -- 172.21.15.6:0/1963863859 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c1582f0 con 0x7f286c152400 2024-03-22T13:03:43.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.419+0000 7f286aa00640 1 --2- 172.21.15.6:0/1963863859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c152400 0x7f286c15b740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f286c06e7f0 tx=0x7f285c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56b9a0188e963023 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:43.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.421+0000 7f2869600640 1 -- 172.21.15.6:0/1963863859 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c035020 con 0x7f286c152400 2024-03-22T13:03:43.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.423+0000 7f2869600640 1 -- 172.21.15.6:0/1963863859 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c031830 con 0x7f286c152400 2024-03-22T13:03:43.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.423+0000 7f2869600640 1 -- 172.21.15.6:0/1963863859 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c031c00 con 0x7f286c152400 2024-03-22T13:03:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 -- 172.21.15.6:0/1963863859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c152400 msgr2=0x7f286c15b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 --2- 172.21.15.6:0/1963863859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c152400 0x7f286c15b740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f286c06e7f0 tx=0x7f285c02b8d0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 -- 172.21.15.6:0/1963863859 shutdown_connections 2024-03-22T13:03:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 --2- 172.21.15.6:0/1963863859 >> v2:172.21.15.92:3300/0 conn(0x7f286c06f8c0 0x7f286c071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 --2- 172.21.15.6:0/1963863859 >> v2:172.21.15.6:3300/0 conn(0x7f286c15bc80 0x7f286c0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 --2- 172.21.15.6:0/1963863859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c152400 0x7f286c15b740 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 -- 172.21.15.6:0/1963863859 >> 172.21.15.6:0/1963863859 conn(0x7f286c0cb930 msgr2=0x7f286c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 -- 172.21.15.6:0/1963863859 shutdown_connections 2024-03-22T13:03:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.424+0000 7f2871800640 1 -- 172.21.15.6:0/1963863859 wait complete. 2024-03-22T13:03:43.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.426+0000 7f2871800640 1 Processor -- start 2024-03-22T13:03:43.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.426+0000 7f2871800640 1 -- start start 2024-03-22T13:03:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.427+0000 7f2871800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c06f8c0 0x7f286c148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.427+0000 7f2871800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c152400 0x7f286c156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.427+0000 7f2871800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c15bc80 0x7f286c157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.427+0000 7f2871800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c158b90 con 0x7f286c06f8c0 2024-03-22T13:03:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.427+0000 7f2871800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1588b0 con 0x7f286c152400 2024-03-22T13:03:43.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.427+0000 7f2871800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1527e0 con 0x7f286c15bc80 2024-03-22T13:03:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.428+0000 7f286a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c152400 0x7f286c156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.428+0000 7f286aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c06f8c0 0x7f286c148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.428+0000 7f286b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c15bc80 0x7f286c157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.428+0000 7f286aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c06f8c0 0x7f286c148380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35762/0 (socket says 172.21.15.6:35762) 2024-03-22T13:03:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.428+0000 7f286aa00640 1 -- 172.21.15.6:0/892635208 learned_addr learned my addr 172.21.15.6:0/892635208 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:43.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.432+0000 7f286b400640 1 -- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c152400 msgr2=0x7f286c156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:43.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.432+0000 7f286b400640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c152400 0x7f286c156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.432+0000 7f286b400640 1 -- 172.21.15.6:0/892635208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c06f8c0 msgr2=0x7f286c148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:43.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.432+0000 7f286b400640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c06f8c0 0x7f286c148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:43.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.432+0000 7f286b400640 1 -- 172.21.15.6:0/892635208 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c152ad0 con 0x7f286c15bc80 2024-03-22T13:03:43.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.434+0000 7f286b400640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c15bc80 0x7f286c157040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f286400ef00 tx=0x7f286400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:43.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.437+0000 7f2862a00640 1 -- 172.21.15.6:0/892635208 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864018020 con 0x7f286c15bc80 2024-03-22T13:03:43.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.437+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c157620 con 0x7f286c15bc80 2024-03-22T13:03:43.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.437+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c1594b0 con 0x7f286c15bc80 2024-03-22T13:03:43.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.439+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c0756c0 con 0x7f286c15bc80 2024-03-22T13:03:43.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.462+0000 7f2862a00640 1 -- 172.21.15.6:0/892635208 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286401d400 con 0x7f286c15bc80 2024-03-22T13:03:43.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.462+0000 7f2862a00640 1 -- 172.21.15.6:0/892635208 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864009e60 con 0x7f286c15bc80 2024-03-22T13:03:43.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.465+0000 7f2862a00640 1 -- 172.21.15.6:0/892635208 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2864005d10 con 0x7f286c15bc80 2024-03-22T13:03:43.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.466+0000 7f2862a00640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f284c06bc00 0x7f284c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:43.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.467+0000 7f2862a00640 1 -- 172.21.15.6:0/892635208 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f286408bb30 con 0x7f286c15bc80 2024-03-22T13:03:43.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.467+0000 7f286aa00640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f284c06bc00 0x7f284c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:43.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.468+0000 7f286aa00640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f284c06bc00 0x7f284c06e0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f285c009560 tx=0x7f285c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:43.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.723+0000 7f2862a00640 1 -- 172.21.15.6:0/892635208 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2864022030 con 0x7f286c15bc80 2024-03-22T13:03:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:43.992+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286c0c6400 con 0x7f286c15bc80 2024-03-22T13:03:44.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.005+0000 7f2862a00640 1 -- 172.21.15.6:0/892635208 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f2864059cb0 con 0x7f286c15bc80 2024-03-22T13:03:44.005 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:44.005 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:44.005 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f284c06bc00 msgr2=0x7f284c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f284c06bc00 0x7f284c06e0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f285c009560 tx=0x7f285c008030 comp rx=0 tx=0).stop 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c15bc80 msgr2=0x7f286c157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c15bc80 0x7f286c157040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f286400ef00 tx=0x7f286400b480 comp rx=0 tx=0).stop 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 shutdown_connections 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f286c15bc80 0x7f286c157040 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f284c06bc00 0x7f284c06e0c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f286c152400 0x7f286c156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 --2- 172.21.15.6:0/892635208 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f286c06f8c0 0x7f286c148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 >> 172.21.15.6:0/892635208 conn(0x7f286c0cb930 msgr2=0x7f286c14fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 shutdown_connections 2024-03-22T13:03:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:44.008+0000 7f2871800640 1 -- 172.21.15.6:0/892635208 wait complete. 2024-03-22T13:03:44.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:44.639 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:45.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:45.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.866+0000 7fcf3d800640 1 Processor -- start 2024-03-22T13:03:45.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.866+0000 7fcf3d800640 1 -- start start 2024-03-22T13:03:45.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.867+0000 7fcf3d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcf3806c230 0x7fcf3815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.867+0000 7fcf3d800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3806c640 con 0x7fcf380cc380 2024-03-22T13:03:45.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.867+0000 7fcf3d800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf380cee00 con 0x7fcf380ce8a0 2024-03-22T13:03:45.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.867+0000 7fcf3d800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf380cf0c0 con 0x7fcf3806c230 2024-03-22T13:03:45.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.881+0000 7fcf37400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fcf380ce8a0 0x7fcf38162b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:57336/0 (socket says 172.21.15.6:57336) 2024-03-22T13:03:45.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.881+0000 7fcf37400640 1 -- 172.21.15.6:0/2434311283 learned_addr learned my addr 172.21.15.6:0/2434311283 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.884+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180729769 0 0) 0x7fcf380cee00 con 0x7fcf380ce8a0 2024-03-22T13:03:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.884+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c003520 con 0x7fcf380ce8a0 2024-03-22T13:03:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127056806 0 0) 0x7fcf1c003520 con 0x7fcf380ce8a0 2024-03-22T13:03:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 >> v2:172.21.15.6:3300/0 conn(0x7fcf3806c230 msgr2=0x7fcf3815ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 --2- 172.21.15.6:0/2434311283 >> v2:172.21.15.6:3300/0 conn(0x7fcf3806c230 0x7fcf3815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 >> v1:172.21.15.6:6789/0 conn(0x7fcf380cc380 legacy=0x7fcf380cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf380cf3a0 con 0x7fcf380ce8a0 2024-03-22T13:03:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7fcf24001670 con 0x7fcf380ce8a0 2024-03-22T13:03:45.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 >> v1:172.21.15.92:6789/0 conn(0x7fcf380ce8a0 legacy=0x7fcf38162b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:45.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf1c0039b0 0x7fcf1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c004800 0x7fcf1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf1c0052e0 0x7fcf1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380cf0c0 con 0x7fcf1c0039b0 2024-03-22T13:03:45.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c00c1e0 con 0x7fcf1c004800 2024-03-22T13:03:45.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.888+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c00c3e0 con 0x7fcf1c0052e0 2024-03-22T13:03:45.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.889+0000 7fcf3ce00640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf1c0052e0 0x7fcf1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:45.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.892+0000 7fcf37400640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c004800 0x7fcf1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:45.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.894+0000 7fcf3ce00640 1 -- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c004800 msgr2=0x7fcf1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:45.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.894+0000 7fcf3ce00640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c004800 0x7fcf1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:45.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.894+0000 7fcf3ce00640 1 -- 172.21.15.6:0/2434311283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf1c0039b0 msgr2=0x7fcf1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:45.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.894+0000 7fcf3ce00640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf1c0039b0 0x7fcf1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:45.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.894+0000 7fcf3ce00640 1 -- 172.21.15.6:0/2434311283 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c003520 con 0x7fcf1c0052e0 2024-03-22T13:03:45.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.896+0000 7fcf3ce00640 1 -- 172.21.15.6:0/2434311283 reap_dead start 2024-03-22T13:03:45.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.898+0000 7fcf3ce00640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf1c0052e0 0x7fcf1c00b9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e40 tx=0x7fcf1c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6d5eb2651628aa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:45.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.902+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30009a10 con 0x7fcf1c0052e0 2024-03-22T13:03:45.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.902+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf30006e50 con 0x7fcf1c0052e0 2024-03-22T13:03:45.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.902+0000 7fcf36a00640 1 -- 172.21.15.6:0/2434311283 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30003970 con 0x7fcf1c0052e0 2024-03-22T13:03:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.903+0000 7fcf3d800640 1 -- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf1c0052e0 msgr2=0x7fcf1c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.903+0000 7fcf3d800640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf1c0052e0 0x7fcf1c00b9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e40 tx=0x7fcf1c005830 comp rx=0 tx=0).stop 2024-03-22T13:03:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.904+0000 7fcf3d800640 1 -- 172.21.15.6:0/2434311283 shutdown_connections 2024-03-22T13:03:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.904+0000 7fcf3d800640 1 --2- 172.21.15.6:0/2434311283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf1c0052e0 0x7fcf1c00b9c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.904+0000 7fcf3d800640 1 -- 172.21.15.6:0/2434311283 >> 172.21.15.6:0/2434311283 conn(0x7fcf380b1480 msgr2=0x7fcf380b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.904+0000 7fcf3d800640 1 -- 172.21.15.6:0/2434311283 shutdown_connections 2024-03-22T13:03:45.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.904+0000 7fcf3d800640 1 -- 172.21.15.6:0/2434311283 wait complete. 2024-03-22T13:03:45.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.906+0000 7fcf3d800640 1 Processor -- start 2024-03-22T13:03:45.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.906+0000 7fcf3d800640 1 -- start start 2024-03-22T13:03:45.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.907+0000 7fcf3d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c0052e0 0x7fcf38150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.907+0000 7fcf3d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf38151310 0x7fcf3814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.907+0000 7fcf3d800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf3814e260 0x7fcf380b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.907+0000 7fcf3d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3815a910 con 0x7fcf38151310 2024-03-22T13:03:45.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.907+0000 7fcf3d800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3815a630 con 0x7fcf1c0052e0 2024-03-22T13:03:45.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.907+0000 7fcf3d800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf381549d0 con 0x7fcf3814e260 2024-03-22T13:03:45.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.908+0000 7fcf37e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c0052e0 0x7fcf38150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:45.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.908+0000 7fcf3ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf3814e260 0x7fcf380b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:45.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.908+0000 7fcf3ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf3814e260 0x7fcf380b4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55174/0 (socket says 172.21.15.6:55174) 2024-03-22T13:03:45.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.908+0000 7fcf3ce00640 1 -- 172.21.15.6:0/53649418 learned_addr learned my addr 172.21.15.6:0/53649418 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:45.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.913+0000 7fcf3ce00640 1 -- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c0052e0 msgr2=0x7fcf38150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:45.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.913+0000 7fcf3ce00640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c0052e0 0x7fcf38150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:45.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.913+0000 7fcf3ce00640 1 -- 172.21.15.6:0/53649418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf38151310 msgr2=0x7fcf3814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:45.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.913+0000 7fcf3ce00640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf38151310 0x7fcf3814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:45.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.913+0000 7fcf3ce00640 1 -- 172.21.15.6:0/53649418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf381556f0 con 0x7fcf3814e260 2024-03-22T13:03:45.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.916+0000 7fcf3ce00640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf3814e260 0x7fcf380b4a30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009f60 tx=0x7fcf30004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.921+0000 7fcf2fe00640 1 -- 172.21.15.6:0/53649418 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30003e10 con 0x7fcf3814e260 2024-03-22T13:03:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.921+0000 7fcf2fe00640 1 -- 172.21.15.6:0/53649418 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf30042070 con 0x7fcf3814e260 2024-03-22T13:03:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.921+0000 7fcf2fe00640 1 -- 172.21.15.6:0/53649418 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf300047c0 con 0x7fcf3814e260 2024-03-22T13:03:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.921+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3814f720 con 0x7fcf3814e260 2024-03-22T13:03:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.921+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf381505d0 con 0x7fcf3814e260 2024-03-22T13:03:45.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.922+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf38075650 con 0x7fcf3814e260 2024-03-22T13:03:45.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.953+0000 7fcf2fe00640 1 -- 172.21.15.6:0/53649418 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcf3003c070 con 0x7fcf3814e260 2024-03-22T13:03:45.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.955+0000 7fcf2fe00640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf1806bc50 0x7fcf1806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:45.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.955+0000 7fcf37e00640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf1806bc50 0x7fcf1806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:45.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:45.956+0000 7fcf37e00640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf1806bc50 0x7fcf1806e110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf28001770 tx=0x7fcf28009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:46.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.197+0000 7fcf2fe00640 1 -- 172.21.15.6:0/53649418 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fcf30079400 con 0x7fcf3814e260 2024-03-22T13:03:46.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.209+0000 7fcf2fe00640 1 -- 172.21.15.6:0/53649418 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcf381556f0 con 0x7fcf3814e260 2024-03-22T13:03:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.465+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf380b2510 con 0x7fcf3814e260 2024-03-22T13:03:46.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.486+0000 7fcf2fe00640 1 -- 172.21.15.6:0/53649418 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fcf3003a040 con 0x7fcf3814e260 2024-03-22T13:03:46.487 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:46.487 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:46.487 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:46.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf1806bc50 msgr2=0x7fcf1806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:46.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf1806bc50 0x7fcf1806e110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf28001770 tx=0x7fcf28009040 comp rx=0 tx=0).stop 2024-03-22T13:03:46.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf3814e260 msgr2=0x7fcf380b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf3814e260 0x7fcf380b4a30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009f60 tx=0x7fcf30004b50 comp rx=0 tx=0).stop 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 shutdown_connections 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf3814e260 0x7fcf380b4a30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf38151310 0x7fcf3814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf1806bc50 0x7fcf1806e110 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 --2- 172.21.15.6:0/53649418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf1c0052e0 0x7fcf38150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 >> 172.21.15.6:0/53649418 conn(0x7fcf380b1480 msgr2=0x7fcf380cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 shutdown_connections 2024-03-22T13:03:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:46.489+0000 7fcf3d800640 1 -- 172.21.15.6:0/53649418 wait complete. 2024-03-22T13:03:47.112 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:47.112 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:48.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:48.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.303+0000 7f1fbcc00640 1 Processor -- start 2024-03-22T13:03:48.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.303+0000 7f1fbcc00640 1 -- start start 2024-03-22T13:03:48.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.305+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:48.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.305+0000 7f1fbcc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1fb806f8c0 0x7f1fb80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:48.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.305+0000 7f1fbcc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb81590b0 con 0x7f1fb815bc80 2024-03-22T13:03:48.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.305+0000 7f1fbcc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb80724d0 con 0x7f1fb806f8c0 2024-03-22T13:03:48.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.305+0000 7f1fbcc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb80721f0 con 0x7f1fb8152400 2024-03-22T13:03:48.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.306+0000 7f1fb5e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:48.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.306+0000 7f1fb6800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1fb806f8c0 0x7f1fb80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:48.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.306+0000 7f1fb5e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb815b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55190/0 (socket says 172.21.15.6:55190) 2024-03-22T13:03:48.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.306+0000 7f1fb5e00640 1 -- 172.21.15.6:0/1791267475 learned_addr learned my addr 172.21.15.6:0/1791267475 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:48.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.309+0000 7f1fb5e00640 1 -- 172.21.15.6:0/1791267475 >> v2:172.21.15.92:3300/0 conn(0x7f1fb806f8c0 msgr2=0x7f1fb80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:48.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.309+0000 7f1fb5e00640 1 --2- 172.21.15.6:0/1791267475 >> v2:172.21.15.92:3300/0 conn(0x7f1fb806f8c0 0x7f1fb80c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.309+0000 7f1fb5e00640 1 -- 172.21.15.6:0/1791267475 >> v1:172.21.15.92:6789/0 conn(0x7f1fb815bc80 legacy=0x7f1fb80716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:48.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.309+0000 7f1fb5e00640 1 -- 172.21.15.6:0/1791267475 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb81582f0 con 0x7f1fb8152400 2024-03-22T13:03:48.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.311+0000 7f1fb5e00640 1 --2- 172.21.15.6:0/1791267475 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb815b740 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009870 tx=0x7f1fa802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ebb2b072945dd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:48.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.315+0000 7f1fb4a00640 1 -- 172.21.15.6:0/1791267475 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa802f840 con 0x7f1fb8152400 2024-03-22T13:03:48.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.315+0000 7f1fb4a00640 1 -- 172.21.15.6:0/1791267475 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa802f9e0 con 0x7f1fb8152400 2024-03-22T13:03:48.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.315+0000 7f1fb4a00640 1 -- 172.21.15.6:0/1791267475 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa802fdb0 con 0x7f1fb8152400 2024-03-22T13:03:48.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 -- 172.21.15.6:0/1791267475 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 msgr2=0x7f1fb815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:48.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/1791267475 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb815b740 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009870 tx=0x7f1fa802ba20 comp rx=0 tx=0).stop 2024-03-22T13:03:48.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 -- 172.21.15.6:0/1791267475 shutdown_connections 2024-03-22T13:03:48.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/1791267475 >> v2:172.21.15.92:3300/0 conn(0x7f1fb806f8c0 0x7f1fb80c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/1791267475 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb815b740 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 -- 172.21.15.6:0/1791267475 >> 172.21.15.6:0/1791267475 conn(0x7f1fb80cb930 msgr2=0x7f1fb80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:48.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 -- 172.21.15.6:0/1791267475 shutdown_connections 2024-03-22T13:03:48.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.316+0000 7f1fbcc00640 1 -- 172.21.15.6:0/1791267475 wait complete. 2024-03-22T13:03:48.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.318+0000 7f1fbcc00640 1 Processor -- start 2024-03-22T13:03:48.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.318+0000 7f1fbcc00640 1 -- start start 2024-03-22T13:03:48.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.319+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fb806f8c0 0x7f1fb8145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:48.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.319+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb8146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:48.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.319+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb815bc80 0x7f1fb8146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:48.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.319+0000 7f1fbcc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb8158b90 con 0x7f1fb806f8c0 2024-03-22T13:03:48.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.319+0000 7f1fbcc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81588b0 con 0x7f1fb815bc80 2024-03-22T13:03:48.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.319+0000 7f1fbcc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81527e0 con 0x7f1fb8152400 2024-03-22T13:03:48.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.320+0000 7f1fb6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb815bc80 0x7f1fb8146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:48.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.320+0000 7f1fb5400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb8146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:48.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.320+0000 7f1fb5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fb806f8c0 0x7f1fb8145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:48.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.320+0000 7f1fb5400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb8146760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55194/0 (socket says 172.21.15.6:55194) 2024-03-22T13:03:48.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.320+0000 7f1fb5400640 1 -- 172.21.15.6:0/2209342789 learned_addr learned my addr 172.21.15.6:0/2209342789 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:48.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.324+0000 7f1fb5400640 1 -- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb815bc80 msgr2=0x7f1fb8146ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:48.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.324+0000 7f1fb5400640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb815bc80 0x7f1fb8146ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.324+0000 7f1fb5400640 1 -- 172.21.15.6:0/2209342789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fb806f8c0 msgr2=0x7f1fb8145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:48.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.324+0000 7f1fb5400640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fb806f8c0 0x7f1fb8145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.324+0000 7f1fb5400640 1 -- 172.21.15.6:0/2209342789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb81ec050 con 0x7f1fb8152400 2024-03-22T13:03:48.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.326+0000 7f1fb5400640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb8146760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c470 tx=0x7f1fa400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:48.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.328+0000 7f1fae000640 1 -- 172.21.15.6:0/2209342789 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40052d0 con 0x7f1fb8152400 2024-03-22T13:03:48.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.329+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb81ec2d0 con 0x7f1fb8152400 2024-03-22T13:03:48.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.329+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb81ec890 con 0x7f1fb8152400 2024-03-22T13:03:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.330+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb80756c0 con 0x7f1fb8152400 2024-03-22T13:03:48.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.331+0000 7f1fae000640 1 -- 172.21.15.6:0/2209342789 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa4005470 con 0x7f1fb8152400 2024-03-22T13:03:48.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.331+0000 7f1fae000640 1 -- 172.21.15.6:0/2209342789 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4020060 con 0x7f1fb8152400 2024-03-22T13:03:48.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.612+0000 7f1fae000640 1 -- 172.21.15.6:0/2209342789 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1fa4007220 con 0x7f1fb8152400 2024-03-22T13:03:48.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.616+0000 7f1fae000640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9806bbd0 0x7f1f9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:48.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.616+0000 7f1fae000640 1 -- 172.21.15.6:0/2209342789 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1fa408a680 con 0x7f1fb8152400 2024-03-22T13:03:48.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.617+0000 7f1fb5e00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9806bbd0 0x7f1f9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:48.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.618+0000 7f1fb5e00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9806bbd0 0x7f1f9806e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8036000 tx=0x7f1fa8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:48.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.624+0000 7f1fae000640 1 -- 172.21.15.6:0/2209342789 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1fb81ec050 con 0x7f1fb8152400 2024-03-22T13:03:48.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.870+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fb80c6400 con 0x7f1fb8152400 2024-03-22T13:03:48.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.894+0000 7f1fae000640 1 -- 172.21.15.6:0/2209342789 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f1fa4058800 con 0x7f1fb8152400 2024-03-22T13:03:48.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:48.894 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:48.894 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.896+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9806bbd0 msgr2=0x7f1f9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.896+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9806bbd0 0x7f1f9806e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8036000 tx=0x7f1fa8008030 comp rx=0 tx=0).stop 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.896+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 msgr2=0x7f1fb8146760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.896+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb8146760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c470 tx=0x7f1fa400c930 comp rx=0 tx=0).stop 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 shutdown_connections 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1f9806bbd0 0x7f1f9806e090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb815bc80 0x7f1fb8146ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fb8152400 0x7f1fb8146760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 --2- 172.21.15.6:0/2209342789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1fb806f8c0 0x7f1fb8145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:48.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 >> 172.21.15.6:0/2209342789 conn(0x7f1fb80cb930 msgr2=0x7f1fb80716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:48.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 shutdown_connections 2024-03-22T13:03:48.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:48.897+0000 7f1fbcc00640 1 -- 172.21.15.6:0/2209342789 wait complete. 2024-03-22T13:03:49.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:49.519 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:50.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:50.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.723+0000 7fed49400640 1 Processor -- start 2024-03-22T13:03:50.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.723+0000 7fed49400640 1 -- start start 2024-03-22T13:03:50.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.724+0000 7fed49400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed440cca20 0x7fed4415e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:50.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.724+0000 7fed49400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:50.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.724+0000 7fed49400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4406c440 con 0x7fed440cc640 2024-03-22T13:03:50.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.724+0000 7fed49400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4406d330 con 0x7fed440cca20 2024-03-22T13:03:50.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.724+0000 7fed49400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4406c720 con 0x7fed4415b7a0 2024-03-22T13:03:50.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.725+0000 7fed43400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed440cca20 0x7fed4415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:50.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.726+0000 7fed43400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed440cca20 0x7fed4415e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:59410/0 (socket says 172.21.15.6:59410) 2024-03-22T13:03:50.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.726+0000 7fed43400640 1 -- 172.21.15.6:0/2834972742 learned_addr learned my addr 172.21.15.6:0/2834972742 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.727+0000 7fed48a00640 1 --2- 172.21.15.6:0/2834972742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.730+0000 7fed48a00640 1 -- 172.21.15.6:0/2834972742 >> v2:172.21.15.92:3300/0 conn(0x7fed440cca20 msgr2=0x7fed4415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:50.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.730+0000 7fed48a00640 1 --2- 172.21.15.6:0/2834972742 >> v2:172.21.15.92:3300/0 conn(0x7fed440cca20 0x7fed4415e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:50.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.730+0000 7fed48a00640 1 -- 172.21.15.6:0/2834972742 >> v1:172.21.15.92:6789/0 conn(0x7fed440cc640 legacy=0x7fed4406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:50.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.730+0000 7fed48a00640 1 -- 172.21.15.6:0/2834972742 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4406ca00 con 0x7fed4415b7a0 2024-03-22T13:03:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.732+0000 7fed48a00640 1 --2- 172.21.15.6:0/2834972742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440ca340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009d90 tx=0x7fed3c02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e423735f964cf441 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.736+0000 7fed42a00640 1 -- 172.21.15.6:0/2834972742 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c039070 con 0x7fed4415b7a0 2024-03-22T13:03:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.736+0000 7fed42a00640 1 -- 172.21.15.6:0/2834972742 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c02cce0 con 0x7fed4415b7a0 2024-03-22T13:03:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.736+0000 7fed42a00640 1 -- 172.21.15.6:0/2834972742 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c034660 con 0x7fed4415b7a0 2024-03-22T13:03:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 -- 172.21.15.6:0/2834972742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 msgr2=0x7fed440ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 --2- 172.21.15.6:0/2834972742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440ca340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009d90 tx=0x7fed3c02bf30 comp rx=0 tx=0).stop 2024-03-22T13:03:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 -- 172.21.15.6:0/2834972742 shutdown_connections 2024-03-22T13:03:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 --2- 172.21.15.6:0/2834972742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440ca340 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 --2- 172.21.15.6:0/2834972742 >> v2:172.21.15.92:3300/0 conn(0x7fed440cca20 0x7fed4415e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 -- 172.21.15.6:0/2834972742 >> 172.21.15.6:0/2834972742 conn(0x7fed440cf9c0 msgr2=0x7fed440cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 -- 172.21.15.6:0/2834972742 shutdown_connections 2024-03-22T13:03:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.737+0000 7fed49400640 1 -- 172.21.15.6:0/2834972742 wait complete. 2024-03-22T13:03:50.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.739+0000 7fed49400640 1 Processor -- start 2024-03-22T13:03:50.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.739+0000 7fed49400640 1 -- start start 2024-03-22T13:03:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.740+0000 7fed49400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed440cc640 0x7fed441f0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.740+0000 7fed49400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed440cca20 0x7fed440b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.740+0000 7fed49400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.740+0000 7fed49400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4415b000 con 0x7fed440cc640 2024-03-22T13:03:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.740+0000 7fed49400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4406cfc0 con 0x7fed440cca20 2024-03-22T13:03:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.740+0000 7fed49400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4415b180 con 0x7fed4415b7a0 2024-03-22T13:03:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.741+0000 7fed43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed440cca20 0x7fed440b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.741+0000 7fed43e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed440cc640 0x7fed441f0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.741+0000 7fed48a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.741+0000 7fed43e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed440cc640 0x7fed441f0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35802/0 (socket says 172.21.15.6:35802) 2024-03-22T13:03:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.741+0000 7fed43e00640 1 -- 172.21.15.6:0/93118451 learned_addr learned my addr 172.21.15.6:0/93118451 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:50.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.745+0000 7fed48a00640 1 -- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed440cca20 msgr2=0x7fed440b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:50.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.745+0000 7fed48a00640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed440cca20 0x7fed440b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:50.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.745+0000 7fed48a00640 1 -- 172.21.15.6:0/93118451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed440cc640 msgr2=0x7fed441f0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:50.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.745+0000 7fed48a00640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed440cc640 0x7fed441f0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:50.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.745+0000 7fed48a00640 1 -- 172.21.15.6:0/93118451 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed440cd7a0 con 0x7fed4415b7a0 2024-03-22T13:03:50.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.747+0000 7fed48a00640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440afd00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed3c033000 tx=0x7fed3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:50.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.750+0000 7fed3be00640 1 -- 172.21.15.6:0/93118451 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c039050 con 0x7fed4415b7a0 2024-03-22T13:03:50.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.750+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed440cda70 con 0x7fed4415b7a0 2024-03-22T13:03:50.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.750+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed440ce030 con 0x7fed4415b7a0 2024-03-22T13:03:50.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.750+0000 7fed3be00640 1 -- 172.21.15.6:0/93118451 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c045070 con 0x7fed4415b7a0 2024-03-22T13:03:50.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.750+0000 7fed3be00640 1 -- 172.21.15.6:0/93118451 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c009450 con 0x7fed4415b7a0 2024-03-22T13:03:50.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:50.751+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c005290 con 0x7fed4415b7a0 2024-03-22T13:03:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.015+0000 7fed3be00640 1 -- 172.21.15.6:0/93118451 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fed3c03e810 con 0x7fed4415b7a0 2024-03-22T13:03:51.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.017+0000 7fed3be00640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed2406bbd0 0x7fed2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.018+0000 7fed43e00640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed2406bbd0 0x7fed2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.018+0000 7fed3be00640 1 -- 172.21.15.6:0/93118451 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fed3c07a4a0 con 0x7fed4415b7a0 2024-03-22T13:03:51.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.019+0000 7fed43e00640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed2406bbd0 0x7fed2406e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fed340017a0 tx=0x7fed34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:51.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.029+0000 7fed3be00640 1 -- 172.21.15.6:0/93118451 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fed3c07c400 con 0x7fed4415b7a0 2024-03-22T13:03:51.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.281+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed1c005580 con 0x7fed4415b7a0 2024-03-22T13:03:51.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.355+0000 7fed3be00640 1 -- 172.21.15.6:0/93118451 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fed3c0804a0 con 0x7fed4415b7a0 2024-03-22T13:03:51.356 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:51.356 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:51.356 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed2406bbd0 msgr2=0x7fed2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed2406bbd0 0x7fed2406e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fed340017a0 tx=0x7fed34009040 comp rx=0 tx=0).stop 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 msgr2=0x7fed440afd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440afd00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed3c033000 tx=0x7fed3c004a40 comp rx=0 tx=0).stop 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 shutdown_connections 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed4415b7a0 0x7fed440afd00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed2406bbd0 0x7fed2406e090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed440cca20 0x7fed440b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 --2- 172.21.15.6:0/93118451 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed440cc640 0x7fed441f0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 >> 172.21.15.6:0/93118451 conn(0x7fed440cf9c0 msgr2=0x7fed44158110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:51.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 shutdown_connections 2024-03-22T13:03:51.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:51.359+0000 7fed49400640 1 -- 172.21.15.6:0/93118451 wait complete. 2024-03-22T13:03:51.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:51.981 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:52.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:53.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.182+0000 7fdf8fe00640 1 Processor -- start 2024-03-22T13:03:53.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.182+0000 7fdf8fe00640 1 -- start start 2024-03-22T13:03:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.184+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf900cdb00 0x7fdf9006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.184+0000 7fdf8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdf9006dd60 0x7fdf900cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.184+0000 7fdf8fe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf900cdf40 con 0x7fdf9015af60 2024-03-22T13:03:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.184+0000 7fdf8fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf900cc3d0 con 0x7fdf9006dd60 2024-03-22T13:03:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.184+0000 7fdf8fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900ce380 con 0x7fdf900cdb00 2024-03-22T13:03:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.185+0000 7fdf8ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdf9015af60 0x7fdf900cc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53074/0 (socket says 172.21.15.6:53074) 2024-03-22T13:03:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.185+0000 7fdf8ea00640 1 -- 172.21.15.6:0/735722323 learned_addr learned my addr 172.21.15.6:0/735722323 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:53.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.185+0000 7fdf87e00640 1 --2- 172.21.15.6:0/735722323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf900cdb00 0x7fdf9006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:53.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.185+0000 7fdf8f400640 1 --2- 172.21.15.6:0/735722323 >> v2:172.21.15.6:3300/0 conn(0x7fdf9006dd60 0x7fdf900cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:53.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.187+0000 7fdf87e00640 1 -- 172.21.15.6:0/735722323 >> v2:172.21.15.6:3300/0 conn(0x7fdf9006dd60 msgr2=0x7fdf900cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:53.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.187+0000 7fdf87e00640 1 --2- 172.21.15.6:0/735722323 >> v2:172.21.15.6:3300/0 conn(0x7fdf9006dd60 0x7fdf900cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.187+0000 7fdf87e00640 1 -- 172.21.15.6:0/735722323 >> v1:172.21.15.92:6789/0 conn(0x7fdf9015af60 legacy=0x7fdf900cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:53.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.187+0000 7fdf87e00640 1 -- 172.21.15.6:0/735722323 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900ce660 con 0x7fdf900cdb00 2024-03-22T13:03:53.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.190+0000 7fdf87e00640 1 --2- 172.21.15.6:0/735722323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf900cdb00 0x7fdf9006d820 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdf780097a0 tx=0x7fdf7802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cd3f19c69e8a49d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:53.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.195+0000 7fdf8e000640 1 -- 172.21.15.6:0/735722323 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78034070 con 0x7fdf900cdb00 2024-03-22T13:03:53.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.195+0000 7fdf8e000640 1 -- 172.21.15.6:0/735722323 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf78030800 con 0x7fdf900cdb00 2024-03-22T13:03:53.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.195+0000 7fdf8e000640 1 -- 172.21.15.6:0/735722323 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78030bd0 con 0x7fdf900cdb00 2024-03-22T13:03:53.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 -- 172.21.15.6:0/735722323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf900cdb00 msgr2=0x7fdf9006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/735722323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf900cdb00 0x7fdf9006d820 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdf780097a0 tx=0x7fdf7802b940 comp rx=0 tx=0).stop 2024-03-22T13:03:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 -- 172.21.15.6:0/735722323 shutdown_connections 2024-03-22T13:03:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/735722323 >> v2:172.21.15.6:3300/0 conn(0x7fdf9006dd60 0x7fdf900cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/735722323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf900cdb00 0x7fdf9006d820 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 -- 172.21.15.6:0/735722323 >> 172.21.15.6:0/735722323 conn(0x7fdf900cf9d0 msgr2=0x7fdf900cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 -- 172.21.15.6:0/735722323 shutdown_connections 2024-03-22T13:03:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.196+0000 7fdf8fe00640 1 -- 172.21.15.6:0/735722323 wait complete. 2024-03-22T13:03:53.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.198+0000 7fdf8fe00640 1 Processor -- start 2024-03-22T13:03:53.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.199+0000 7fdf8fe00640 1 -- start start 2024-03-22T13:03:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.200+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf9006dd60 0x7fdf90159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.200+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf900cdb00 0x7fdf901efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.200+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf9015af60 0x7fdf901f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.200+0000 7fdf8fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9015a4c0 con 0x7fdf9015af60 2024-03-22T13:03:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.200+0000 7fdf8fe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9015a340 con 0x7fdf900cdb00 2024-03-22T13:03:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.200+0000 7fdf8fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900cf160 con 0x7fdf9006dd60 2024-03-22T13:03:53.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.201+0000 7fdf87e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf900cdb00 0x7fdf901efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:53.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.201+0000 7fdf8ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf9006dd60 0x7fdf90159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:53.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.201+0000 7fdf8f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf9015af60 0x7fdf901f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:53.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.201+0000 7fdf8ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf9006dd60 0x7fdf90159470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55588/0 (socket says 172.21.15.6:55588) 2024-03-22T13:03:53.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.201+0000 7fdf8ea00640 1 -- 172.21.15.6:0/774039195 learned_addr learned my addr 172.21.15.6:0/774039195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:53.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.205+0000 7fdf8ea00640 1 -- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf900cdb00 msgr2=0x7fdf901efdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:53.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.205+0000 7fdf8ea00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf900cdb00 0x7fdf901efdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.205+0000 7fdf8ea00640 1 -- 172.21.15.6:0/774039195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf9015af60 msgr2=0x7fdf901f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:53.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.205+0000 7fdf8ea00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf9015af60 0x7fdf901f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.205+0000 7fdf8ea00640 1 -- 172.21.15.6:0/774039195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900b51b0 con 0x7fdf9006dd60 2024-03-22T13:03:53.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.208+0000 7fdf8ea00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf9006dd60 0x7fdf90159470 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000c4b0 tx=0x7fdf8000c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:53.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.212+0000 7fdf86a00640 1 -- 172.21.15.6:0/774039195 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80017070 con 0x7fdf9006dd60 2024-03-22T13:03:53.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.213+0000 7fdf86a00640 1 -- 172.21.15.6:0/774039195 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80005ce0 con 0x7fdf9006dd60 2024-03-22T13:03:53.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.213+0000 7fdf86a00640 1 -- 172.21.15.6:0/774039195 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80007270 con 0x7fdf9006dd60 2024-03-22T13:03:53.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.213+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf901f34d0 con 0x7fdf9006dd60 2024-03-22T13:03:53.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.213+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf901f3a90 con 0x7fdf9006dd60 2024-03-22T13:03:53.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.214+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf901f3dc0 con 0x7fdf9006dd60 2024-03-22T13:03:53.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.241+0000 7fdf86a00640 1 -- 172.21.15.6:0/774039195 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf8001a070 con 0x7fdf9006dd60 2024-03-22T13:03:53.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.242+0000 7fdf86a00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf6c06bc20 0x7fdf6c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:53.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.243+0000 7fdf87e00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf6c06bc20 0x7fdf6c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.244+0000 7fdf87e00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf6c06bc20 0x7fdf6c06e0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdf78036000 tx=0x7fdf7803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:53.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.487+0000 7fdf86a00640 1 -- 172.21.15.6:0/774039195 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdf80056670 con 0x7fdf9006dd60 2024-03-22T13:03:53.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.499+0000 7fdf86a00640 1 -- 172.21.15.6:0/774039195 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf800585d0 con 0x7fdf9006dd60 2024-03-22T13:03:53.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.776+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf90075650 con 0x7fdf9006dd60 2024-03-22T13:03:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.802+0000 7fdf86a00640 1 -- 172.21.15.6:0/774039195 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fdf8005c670 con 0x7fdf9006dd60 2024-03-22T13:03:53.802 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:53.802 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:53.802 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf6c06bc20 msgr2=0x7fdf6c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf6c06bc20 0x7fdf6c06e0e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdf78036000 tx=0x7fdf7803b040 comp rx=0 tx=0).stop 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf9006dd60 msgr2=0x7fdf90159470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf9006dd60 0x7fdf90159470 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000c4b0 tx=0x7fdf8000c970 comp rx=0 tx=0).stop 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 shutdown_connections 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf9015af60 0x7fdf901f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf6c06bc20 0x7fdf6c06e0e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf900cdb00 0x7fdf901efdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 --2- 172.21.15.6:0/774039195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf9006dd60 0x7fdf90159470 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 >> 172.21.15.6:0/774039195 conn(0x7fdf900cf9d0 msgr2=0x7fdf901536a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 shutdown_connections 2024-03-22T13:03:53.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:53.805+0000 7fdf8fe00640 1 -- 172.21.15.6:0/774039195 wait complete. 2024-03-22T13:03:54.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:54.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:55.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:55.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.628+0000 7f1ac0400640 1 Processor -- start 2024-03-22T13:03:55.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.628+0000 7f1ac0400640 1 -- start start 2024-03-22T13:03:55.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.630+0000 7f1ac0400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ab80ccd80 0x7f1ab80cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:55.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.630+0000 7f1ac0400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ab80cac20 0x7f1ab806c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:55.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.630+0000 7f1ac0400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab815a6b0 con 0x7f1ab80cf5c0 2024-03-22T13:03:55.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.630+0000 7f1ac0400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab8154d60 con 0x7f1ab80cac20 2024-03-22T13:03:55.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.630+0000 7f1ac0400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab815b4d0 con 0x7f1ab80ccd80 2024-03-22T13:03:55.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.631+0000 7f1abe400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ab80cac20 0x7f1ab806c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:55.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.631+0000 7f1abda00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ab80ccd80 0x7f1ab80cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:55.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.632+0000 7f1abd000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1ab80cf5c0 0x7f1ab80ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53088/0 (socket says 172.21.15.6:53088) 2024-03-22T13:03:55.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.632+0000 7f1abd000640 1 -- 172.21.15.6:0/4092032362 learned_addr learned my addr 172.21.15.6:0/4092032362 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:03:55.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.635+0000 7f1abe400640 1 -- 172.21.15.6:0/4092032362 >> v2:172.21.15.92:3300/0 conn(0x7f1ab80ccd80 msgr2=0x7f1ab80cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:55.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.635+0000 7f1abe400640 1 --2- 172.21.15.6:0/4092032362 >> v2:172.21.15.92:3300/0 conn(0x7f1ab80ccd80 0x7f1ab80cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:55.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.635+0000 7f1abe400640 1 -- 172.21.15.6:0/4092032362 >> v1:172.21.15.92:6789/0 conn(0x7f1ab80cf5c0 legacy=0x7f1ab80ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:55.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.635+0000 7f1abe400640 1 -- 172.21.15.6:0/4092032362 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab815a950 con 0x7f1ab80cac20 2024-03-22T13:03:55.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.637+0000 7f1abe400640 1 --2- 172.21.15.6:0/4092032362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80cac20 0x7f1ab806c9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80cc570 tx=0x7f1ab002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c51173b2e569596 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:55.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.643+0000 7f1ab7e00640 1 -- 172.21.15.6:0/4092032362 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0039070 con 0x7f1ab80cac20 2024-03-22T13:03:55.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.643+0000 7f1ab7e00640 1 -- 172.21.15.6:0/4092032362 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab002cc70 con 0x7f1ab80cac20 2024-03-22T13:03:55.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.643+0000 7f1ab7e00640 1 -- 172.21.15.6:0/4092032362 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0034660 con 0x7f1ab80cac20 2024-03-22T13:03:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.643+0000 7f1ac0400640 1 -- 172.21.15.6:0/4092032362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80cac20 msgr2=0x7f1ab806c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.643+0000 7f1ac0400640 1 --2- 172.21.15.6:0/4092032362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80cac20 0x7f1ab806c9c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80cc570 tx=0x7f1ab002bec0 comp rx=0 tx=0).stop 2024-03-22T13:03:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.644+0000 7f1ac0400640 1 -- 172.21.15.6:0/4092032362 shutdown_connections 2024-03-22T13:03:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.644+0000 7f1ac0400640 1 --2- 172.21.15.6:0/4092032362 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80cac20 0x7f1ab806c9c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.644+0000 7f1ac0400640 1 --2- 172.21.15.6:0/4092032362 >> v2:172.21.15.92:3300/0 conn(0x7f1ab80ccd80 0x7f1ab80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.644+0000 7f1ac0400640 1 -- 172.21.15.6:0/4092032362 >> 172.21.15.6:0/4092032362 conn(0x7f1ab80b16c0 msgr2=0x7f1ab80b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.644+0000 7f1ac0400640 1 -- 172.21.15.6:0/4092032362 shutdown_connections 2024-03-22T13:03:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.644+0000 7f1ac0400640 1 -- 172.21.15.6:0/4092032362 wait complete. 2024-03-22T13:03:55.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.645+0000 7f1ac0400640 1 Processor -- start 2024-03-22T13:03:55.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.646+0000 7f1ac0400640 1 -- start start 2024-03-22T13:03:55.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.646+0000 7f1ac0400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ab80cac20 0x7f1ab80c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:55.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.646+0000 7f1ac0400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80ccd80 0x7f1ab8153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:55.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.647+0000 7f1ac0400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ab80cf5c0 0x7f1ab80c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:55.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.647+0000 7f1ac0400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab815b1f0 con 0x7f1ab80ccd80 2024-03-22T13:03:55.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.647+0000 7f1ac0400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab815af10 con 0x7f1ab80cf5c0 2024-03-22T13:03:55.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.647+0000 7f1ac0400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab806db10 con 0x7f1ab80cac20 2024-03-22T13:03:55.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.647+0000 7f1abd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80ccd80 0x7f1ab8153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.647+0000 7f1abda00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ab80cac20 0x7f1ab80c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.647+0000 7f1abe400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ab80cf5c0 0x7f1ab80c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.648+0000 7f1abe400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ab80cf5c0 0x7f1ab80c73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46380/0 (socket says 172.21.15.6:46380) 2024-03-22T13:03:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.648+0000 7f1abe400640 1 -- 172.21.15.6:0/3525729390 learned_addr learned my addr 172.21.15.6:0/3525729390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:55.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.653+0000 7f1abd000640 1 -- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ab80cac20 msgr2=0x7f1ab80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:55.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.653+0000 7f1abd000640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ab80cac20 0x7f1ab80c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:55.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.653+0000 7f1abd000640 1 -- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ab80cf5c0 msgr2=0x7f1ab80c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:55.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.653+0000 7f1abd000640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ab80cf5c0 0x7f1ab80c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:55.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.653+0000 7f1abd000640 1 -- 172.21.15.6:0/3525729390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab81f4420 con 0x7f1ab80ccd80 2024-03-22T13:03:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.656+0000 7f1abd000640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80ccd80 0x7f1ab8153900 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c390 tx=0x7f1aa400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:55.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.662+0000 7f1ab5600640 1 -- 172.21.15.6:0/3525729390 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4017070 con 0x7f1ab80ccd80 2024-03-22T13:03:55.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.662+0000 7f1ab5600640 1 -- 172.21.15.6:0/3525729390 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa4005ce0 con 0x7f1ab80ccd80 2024-03-22T13:03:55.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.662+0000 7f1ab5600640 1 -- 172.21.15.6:0/3525729390 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4007310 con 0x7f1ab80ccd80 2024-03-22T13:03:55.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.662+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab81f4700 con 0x7f1ab80ccd80 2024-03-22T13:03:55.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.662+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab81f09c0 con 0x7f1ab80ccd80 2024-03-22T13:03:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.663+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab80756c0 con 0x7f1ab80ccd80 2024-03-22T13:03:55.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.981+0000 7f1ab5600640 1 -- 172.21.15.6:0/3525729390 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1aa401a070 con 0x7f1ab80ccd80 2024-03-22T13:03:55.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.983+0000 7f1ab5600640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a9806bc50 0x7f1a9806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:55.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.983+0000 7f1abda00640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a9806bc50 0x7f1a9806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:55.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:55.984+0000 7f1abda00640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a9806bc50 0x7f1a9806e110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009b40 tx=0x7f1aac009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:56.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.002+0000 7f1ab5600640 1 -- 172.21.15.6:0/3525729390 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1aa4056620 con 0x7f1ab80ccd80 2024-03-22T13:03:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.004+0000 7f1ab5600640 1 -- 172.21.15.6:0/3525729390 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1aa4021020 con 0x7f1ab80ccd80 2024-03-22T13:03:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.280+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ab80af6a0 con 0x7f1ab80ccd80 2024-03-22T13:03:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.379+0000 7f1ab5600640 1 -- 172.21.15.6:0/3525729390 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f1aa4058580 con 0x7f1ab80ccd80 2024-03-22T13:03:56.379 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:56.379 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:56.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a9806bc50 msgr2=0x7f1a9806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:56.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a9806bc50 0x7f1a9806e110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009b40 tx=0x7f1aac009190 comp rx=0 tx=0).stop 2024-03-22T13:03:56.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80ccd80 msgr2=0x7f1ab8153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:56.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80ccd80 0x7f1ab8153900 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c390 tx=0x7f1aa400c790 comp rx=0 tx=0).stop 2024-03-22T13:03:56.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 shutdown_connections 2024-03-22T13:03:56.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a9806bc50 0x7f1a9806e110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:56.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ab80cf5c0 0x7f1ab80c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ab80ccd80 0x7f1ab8153900 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 --2- 172.21.15.6:0/3525729390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ab80cac20 0x7f1ab80c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 >> 172.21.15.6:0/3525729390 conn(0x7f1ab80b16c0 msgr2=0x7f1ab8158820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 shutdown_connections 2024-03-22T13:03:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:56.382+0000 7f1ac0400640 1 -- 172.21.15.6:0/3525729390 wait complete. 2024-03-22T13:03:57.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:57.005 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:03:58.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:03:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.196+0000 7f6477200640 1 Processor -- start 2024-03-22T13:03:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.196+0000 7f6477200640 1 -- start start 2024-03-22T13:03:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.198+0000 7f6477200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f64780cc380 0x7f647806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.198+0000 7f6477200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64780ce8a0 0x7f647815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.198+0000 7f6477200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64780cee00 con 0x7f64780cc760 2024-03-22T13:03:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.198+0000 7f6477200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64780cf0c0 con 0x7f64780ce8a0 2024-03-22T13:03:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.198+0000 7f6477200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64780cf3a0 con 0x7f64780cc380 2024-03-22T13:03:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.199+0000 7f6475e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f64780cc380 0x7f647806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.199+0000 7f6475e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f64780cc380 0x7f647806c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46388/0 (socket says 172.21.15.6:46388) 2024-03-22T13:03:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.199+0000 7f6475e00640 1 -- 172.21.15.6:0/291003178 learned_addr learned my addr 172.21.15.6:0/291003178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.199+0000 7f6475400640 1 --2- 172.21.15.6:0/291003178 >> v2:172.21.15.6:3300/0 conn(0x7f64780ce8a0 0x7f647815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:58.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.202+0000 7f6475e00640 1 -- 172.21.15.6:0/291003178 >> v2:172.21.15.6:3300/0 conn(0x7f64780ce8a0 msgr2=0x7f647815ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.202+0000 7f6475e00640 1 --2- 172.21.15.6:0/291003178 >> v2:172.21.15.6:3300/0 conn(0x7f64780ce8a0 0x7f647815ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.202+0000 7f6475e00640 1 -- 172.21.15.6:0/291003178 >> v1:172.21.15.6:6789/0 conn(0x7f64780cc760 legacy=0x7f64780cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:03:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.202+0000 7f6475e00640 1 -- 172.21.15.6:0/291003178 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64780cf680 con 0x7f64780cc380 2024-03-22T13:03:58.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.204+0000 7f6475e00640 1 --2- 172.21.15.6:0/291003178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f647806c230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f647808d0e0 tx=0x7f646802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b72f4b70275222d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.208+0000 7f6474a00640 1 -- 172.21.15.6:0/291003178 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468035020 con 0x7f64780cc380 2024-03-22T13:03:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.208+0000 7f6474a00640 1 -- 172.21.15.6:0/291003178 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468031870 con 0x7f64780cc380 2024-03-22T13:03:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.208+0000 7f6474a00640 1 -- 172.21.15.6:0/291003178 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468031c40 con 0x7f64780cc380 2024-03-22T13:03:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.209+0000 7f6477200640 1 -- 172.21.15.6:0/291003178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 msgr2=0x7f647806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.209+0000 7f6477200640 1 --2- 172.21.15.6:0/291003178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f647806c230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f647808d0e0 tx=0x7f646802b910 comp rx=0 tx=0).stop 2024-03-22T13:03:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.209+0000 7f6477200640 1 -- 172.21.15.6:0/291003178 shutdown_connections 2024-03-22T13:03:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.209+0000 7f6477200640 1 --2- 172.21.15.6:0/291003178 >> v2:172.21.15.6:3300/0 conn(0x7f64780ce8a0 0x7f647815ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.209+0000 7f6477200640 1 --2- 172.21.15.6:0/291003178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f647806c230 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.209+0000 7f6477200640 1 -- 172.21.15.6:0/291003178 >> 172.21.15.6:0/291003178 conn(0x7f64780b1480 msgr2=0x7f64780b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.210+0000 7f6477200640 1 -- 172.21.15.6:0/291003178 shutdown_connections 2024-03-22T13:03:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.210+0000 7f6477200640 1 -- 172.21.15.6:0/291003178 wait complete. 2024-03-22T13:03:58.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.212+0000 7f6477200640 1 Processor -- start 2024-03-22T13:03:58.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.212+0000 7f6477200640 1 -- start start 2024-03-22T13:03:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.213+0000 7f6477200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f64780bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.213+0000 7f6477200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64780cc760 0x7f6478149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.213+0000 7f6477200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64780ce8a0 0x7f64780b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.213+0000 7f6477200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64781549d0 con 0x7f64780ce8a0 2024-03-22T13:03:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.213+0000 7f6477200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647815a910 con 0x7f64780cc380 2024-03-22T13:03:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.213+0000 7f6477200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64780ccb70 con 0x7f64780cc760 2024-03-22T13:03:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.214+0000 7f6476800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64780ce8a0 0x7f64780b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.214+0000 7f6476800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64780ce8a0 0x7f64780b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59934/0 (socket says 172.21.15.6:59934) 2024-03-22T13:03:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.214+0000 7f6476800640 1 -- 172.21.15.6:0/584160018 learned_addr learned my addr 172.21.15.6:0/584160018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:03:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.215+0000 7f6475400640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64780cc760 0x7f6478149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.217+0000 7f6475e00640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f64780bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.220+0000 7f6475e00640 1 -- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64780cc760 msgr2=0x7f6478149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.220+0000 7f6475e00640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64780cc760 0x7f6478149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.220+0000 7f6475e00640 1 -- 172.21.15.6:0/584160018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64780ce8a0 msgr2=0x7f64780b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.220+0000 7f6475e00640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64780ce8a0 0x7f64780b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.220+0000 7f6475e00640 1 -- 172.21.15.6:0/584160018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64781485e0 con 0x7f64780cc380 2024-03-22T13:03:58.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.223+0000 7f6475e00640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f64780bffa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6468033000 tx=0x7f64680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.228+0000 7f646e000640 1 -- 172.21.15.6:0/584160018 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468035040 con 0x7f64780cc380 2024-03-22T13:03:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.228+0000 7f646e000640 1 -- 172.21.15.6:0/584160018 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468030d00 con 0x7f64780cc380 2024-03-22T13:03:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.228+0000 7f646e000640 1 -- 172.21.15.6:0/584160018 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680456e0 con 0x7f64780cc380 2024-03-22T13:03:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.228+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6478152890 con 0x7f64780cc380 2024-03-22T13:03:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.228+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6478148860 con 0x7f64780cc380 2024-03-22T13:03:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.229+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6478075650 con 0x7f64780cc380 2024-03-22T13:03:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.249+0000 7f646e000640 1 -- 172.21.15.6:0/584160018 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6468038020 con 0x7f64780cc380 2024-03-22T13:03:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.250+0000 7f646e000640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f645806bc20 0x7f645806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:03:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.251+0000 7f6475400640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f645806bc20 0x7f645806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:03:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.251+0000 7f6475400640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f645806bc20 0x7f645806e0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f64780c2d90 tx=0x7f6464009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:03:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.251+0000 7f646e000640 1 -- 172.21.15.6:0/584160018 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f64680793c0 con 0x7f64780cc380 2024-03-22T13:03:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.509+0000 7f646e000640 1 -- 172.21.15.6:0/584160018 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f64781485e0 con 0x7f64780cc380 2024-03-22T13:03:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.799+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6478158c60 con 0x7f64780cc380 2024-03-22T13:03:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.815+0000 7f646e000640 1 -- 172.21.15.6:0/584160018 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f646807b320 con 0x7f64780cc380 2024-03-22T13:03:58.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:03:58.815 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:03:58.815 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:03:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f645806bc20 msgr2=0x7f645806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f645806bc20 0x7f645806e0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f64780c2d90 tx=0x7f6464009190 comp rx=0 tx=0).stop 2024-03-22T13:03:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 msgr2=0x7f64780bffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:03:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f64780bffa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6468033000 tx=0x7f64680046c0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 shutdown_connections 2024-03-22T13:03:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64780ce8a0 0x7f64780b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64780cc760 0x7f6478149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f645806bc20 0x7f645806e0e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 --2- 172.21.15.6:0/584160018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64780cc380 0x7f64780bffa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:03:58.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 >> 172.21.15.6:0/584160018 conn(0x7f64780b1480 msgr2=0x7f6478149e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:03:58.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.817+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 shutdown_connections 2024-03-22T13:03:58.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:03:58.818+0000 7f6477200640 1 -- 172.21.15.6:0/584160018 wait complete. 2024-03-22T13:03:59.438 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:03:59.439 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:00.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:00.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.633+0000 7fb881200640 1 Processor -- start 2024-03-22T13:04:00.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.633+0000 7fb881200640 1 -- start start 2024-03-22T13:04:00.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.634+0000 7fb881200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb87c0cca20 0x7fb87c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.635+0000 7fb881200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c06c440 con 0x7fb87c15b7a0 2024-03-22T13:04:00.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.635+0000 7fb881200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c06c720 con 0x7fb87c0cc640 2024-03-22T13:04:00.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.635+0000 7fb881200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c06d330 con 0x7fb87c0cca20 2024-03-22T13:04:00.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.635+0000 7fb87b400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb87c0cc640 0x7fb87c06d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53090/0 (socket says 172.21.15.6:53090) 2024-03-22T13:04:00.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.635+0000 7fb87b400640 1 -- 172.21.15.6:0/3381651289 learned_addr learned my addr 172.21.15.6:0/3381651289 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:00.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.653+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213962798 0 0) 0x7fb87c06c720 con 0x7fb87c0cc640 2024-03-22T13:04:00.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.653+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb860003520 con 0x7fb87c0cc640 2024-03-22T13:04:00.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.656+0000 7fb87aa00640 1 --2- 172.21.15.6:0/3381651289 >> v2:172.21.15.6:3300/0 conn(0x7fb87c0cca20 0x7fb87c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409601961 0 0) 0x7fb860003520 con 0x7fb87c0cc640 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 >> v2:172.21.15.6:3300/0 conn(0x7fb87c0cca20 msgr2=0x7fb87c15e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 --2- 172.21.15.6:0/3381651289 >> v2:172.21.15.6:3300/0 conn(0x7fb87c0cca20 0x7fb87c15e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 >> v1:172.21.15.6:6789/0 conn(0x7fb87c15b7a0 legacy=0x7fb87c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c06ca00 con 0x7fb87c0cc640 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7fb86c001670 con 0x7fb87c0cc640 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 >> v1:172.21.15.92:6789/0 conn(0x7fb87c0cc640 legacy=0x7fb87c06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb860003a00 0x7fb860003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c06d330 con 0x7fb860004800 2024-03-22T13:04:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86000c0a0 con 0x7fb8600051c0 2024-03-22T13:04:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.659+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86000c2a0 con 0x7fb860003a00 2024-03-22T13:04:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.662+0000 7fb87be00640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.665+0000 7fb87be00640 1 -- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb860003a00 msgr2=0x7fb860003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.665+0000 7fb87be00640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb860003a00 0x7fb860003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.665+0000 7fb87be00640 1 -- 172.21.15.6:0/3381651289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb860004800 msgr2=0x7fb860004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.665+0000 7fb87be00640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.665+0000 7fb87be00640 1 -- 172.21.15.6:0/3381651289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb860003520 con 0x7fb8600051c0 2024-03-22T13:04:00.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.667+0000 7fb87be00640 1 -- 172.21.15.6:0/3381651289 reap_dead start 2024-03-22T13:04:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.668+0000 7fb87be00640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb874004480 tx=0x7fb860005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74ec09cf15418303 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:00.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.671+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874004080 con 0x7fb8600051c0 2024-03-22T13:04:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.674+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874030400 con 0x7fb8600051c0 2024-03-22T13:04:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.674+0000 7fb87a000640 1 -- 172.21.15.6:0/3381651289 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8740307d0 con 0x7fb8600051c0 2024-03-22T13:04:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.675+0000 7fb881200640 1 -- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8600051c0 msgr2=0x7fb86000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.675+0000 7fb881200640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb874004480 tx=0x7fb860005830 comp rx=0 tx=0).stop 2024-03-22T13:04:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.675+0000 7fb881200640 1 -- 172.21.15.6:0/3381651289 shutdown_connections 2024-03-22T13:04:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.675+0000 7fb881200640 1 --2- 172.21.15.6:0/3381651289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8600051c0 0x7fb86000b9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.675+0000 7fb881200640 1 -- 172.21.15.6:0/3381651289 >> 172.21.15.6:0/3381651289 conn(0x7fb87c0cf9c0 msgr2=0x7fb87c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.675+0000 7fb881200640 1 -- 172.21.15.6:0/3381651289 shutdown_connections 2024-03-22T13:04:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.675+0000 7fb881200640 1 -- 172.21.15.6:0/3381651289 wait complete. 2024-03-22T13:04:00.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.678+0000 7fb881200640 1 Processor -- start 2024-03-22T13:04:00.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.678+0000 7fb881200640 1 -- start start 2024-03-22T13:04:00.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.680+0000 7fb881200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8600051c0 0x7fb87c0ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.680+0000 7fb881200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb87c1f1030 0x7fb87c156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.680+0000 7fb881200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c154ca0 0x7fb87c157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.680+0000 7fb881200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c15b000 con 0x7fb87c154ca0 2024-03-22T13:04:00.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.680+0000 7fb881200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c06cfc0 con 0x7fb87c1f1030 2024-03-22T13:04:00.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.680+0000 7fb881200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c15b180 con 0x7fb8600051c0 2024-03-22T13:04:00.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.681+0000 7fb87aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb87c1f1030 0x7fb87c156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:00.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.681+0000 7fb87aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb87c1f1030 0x7fb87c156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46426/0 (socket says 172.21.15.6:46426) 2024-03-22T13:04:00.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.681+0000 7fb87aa00640 1 -- 172.21.15.6:0/949219103 learned_addr learned my addr 172.21.15.6:0/949219103 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:00.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.681+0000 7fb87be00640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c154ca0 0x7fb87c157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:00.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.684+0000 7fb87aa00640 1 -- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8600051c0 msgr2=0x7fb87c0ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:00.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.684+0000 7fb87aa00640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8600051c0 0x7fb87c0ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:00.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.684+0000 7fb87aa00640 1 -- 172.21.15.6:0/949219103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c154ca0 msgr2=0x7fb87c157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:00.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.684+0000 7fb87aa00640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c154ca0 0x7fb87c157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:00.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.684+0000 7fb87aa00640 1 -- 172.21.15.6:0/949219103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0cf060 con 0x7fb87c1f1030 2024-03-22T13:04:00.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.686+0000 7fb87aa00640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb87c1f1030 0x7fb87c156880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb86800e7c0 tx=0x7fb86800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.689+0000 7fb873400640 1 -- 172.21.15.6:0/949219103 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868017070 con 0x7fb87c1f1030 2024-03-22T13:04:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.689+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c0cd7a0 con 0x7fb87c1f1030 2024-03-22T13:04:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.689+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c0cdd60 con 0x7fb87c1f1030 2024-03-22T13:04:00.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.690+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c075650 con 0x7fb87c1f1030 2024-03-22T13:04:00.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.691+0000 7fb873400640 1 -- 172.21.15.6:0/949219103 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868005990 con 0x7fb87c1f1030 2024-03-22T13:04:00.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.691+0000 7fb873400640 1 -- 172.21.15.6:0/949219103 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868021060 con 0x7fb87c1f1030 2024-03-22T13:04:00.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.986+0000 7fb873400640 1 -- 172.21.15.6:0/949219103 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb86801a070 con 0x7fb87c1f1030 2024-03-22T13:04:00.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.987+0000 7fb873400640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb85c06bc00 0x7fb85c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:00.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.988+0000 7fb87b400640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb85c06bc00 0x7fb85c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:00.988+0000 7fb87b400640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb85c06bc00 0x7fb85c06e0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0079b0 tx=0x7fb86c0015e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.011+0000 7fb873400640 1 -- 172.21.15.6:0/949219103 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb868057520 con 0x7fb87c1f1030 2024-03-22T13:04:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.032+0000 7fb873400640 1 -- 172.21.15.6:0/949219103 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb868059480 con 0x7fb87c1f1030 2024-03-22T13:04:01.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:04:01.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:04:01.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.297+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb87c0af100 con 0x7fb87c1f1030 2024-03-22T13:04:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.318+0000 7fb873400640 1 -- 172.21.15.6:0/949219103 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fb86805d520 con 0x7fb87c1f1030 2024-03-22T13:04:01.318 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:01.318 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:01.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb85c06bc00 msgr2=0x7fb85c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb85c06bc00 0x7fb85c06e0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0079b0 tx=0x7fb86c0015e0 comp rx=0 tx=0).stop 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb87c1f1030 msgr2=0x7fb87c156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb87c1f1030 0x7fb87c156880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb86800e7c0 tx=0x7fb86800ec80 comp rx=0 tx=0).stop 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 shutdown_connections 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c154ca0 0x7fb87c157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb85c06bc00 0x7fb85c06e0c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb87c1f1030 0x7fb87c156880 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 --2- 172.21.15.6:0/949219103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8600051c0 0x7fb87c0ceb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.320+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 >> 172.21.15.6:0/949219103 conn(0x7fb87c0cf9c0 msgr2=0x7fb87c1582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.321+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 shutdown_connections 2024-03-22T13:04:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:01.321+0000 7fb881200640 1 -- 172.21.15.6:0/949219103 wait complete. 2024-03-22T13:04:01.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:01.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:02.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:03.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.143+0000 7f6069400640 1 Processor -- start 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.143+0000 7f6069400640 1 -- start start 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.145+0000 7f6069400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f606406f8c0 0x7f60640716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.145+0000 7f6069400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f60640c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.145+0000 7f6069400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60641590b0 con 0x7f6064152400 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.145+0000 7f6069400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60640724d0 con 0x7f606406f8c0 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.145+0000 7f6069400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60640721f0 con 0x7f6064071c10 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.146+0000 7f6062800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6064152400 0x7f606415b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53460/0 (socket says 172.21.15.6:53460) 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.146+0000 7f6062800640 1 -- 172.21.15.6:0/340952276 learned_addr learned my addr 172.21.15.6:0/340952276 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.146+0000 7f6063200640 1 --2- 172.21.15.6:0/340952276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f60640c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.146+0000 7f6061e00640 1 --2- 172.21.15.6:0/340952276 >> v2:172.21.15.6:3300/0 conn(0x7f606406f8c0 0x7f60640716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.149+0000 7f6061400640 1 -- 172.21.15.6:0/340952276 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349446365 0 0) 0x7f60641590b0 con 0x7f6064152400 2024-03-22T13:04:03.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.149+0000 7f6061400640 1 -- 172.21.15.6:0/340952276 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6048003520 con 0x7f6064152400 2024-03-22T13:04:03.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.150+0000 7f6063200640 1 -- 172.21.15.6:0/340952276 >> v2:172.21.15.6:3300/0 conn(0x7f606406f8c0 msgr2=0x7f60640716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:03.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.150+0000 7f6063200640 1 --2- 172.21.15.6:0/340952276 >> v2:172.21.15.6:3300/0 conn(0x7f606406f8c0 0x7f60640716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.150+0000 7f6063200640 1 -- 172.21.15.6:0/340952276 >> v1:172.21.15.92:6789/0 conn(0x7f6064152400 legacy=0x7f606415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:03.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.150+0000 7f6063200640 1 -- 172.21.15.6:0/340952276 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60641582f0 con 0x7f6064071c10 2024-03-22T13:04:03.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.153+0000 7f6063200640 1 --2- 172.21.15.6:0/340952276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f60640c5040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6064072650 tx=0x7f605c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5854945ed096bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.157+0000 7f6061400640 1 -- 172.21.15.6:0/340952276 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c039070 con 0x7f6064071c10 2024-03-22T13:04:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.157+0000 7f6061400640 1 -- 172.21.15.6:0/340952276 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c02ccd0 con 0x7f6064071c10 2024-03-22T13:04:03.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.157+0000 7f6061400640 1 -- 172.21.15.6:0/340952276 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c034660 con 0x7f6064071c10 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 -- 172.21.15.6:0/340952276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 msgr2=0x7f60640c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 --2- 172.21.15.6:0/340952276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f60640c5040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6064072650 tx=0x7f605c02bf20 comp rx=0 tx=0).stop 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 -- 172.21.15.6:0/340952276 shutdown_connections 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 --2- 172.21.15.6:0/340952276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f60640c5040 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 --2- 172.21.15.6:0/340952276 >> v2:172.21.15.6:3300/0 conn(0x7f606406f8c0 0x7f60640716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 -- 172.21.15.6:0/340952276 >> 172.21.15.6:0/340952276 conn(0x7f60640cb930 msgr2=0x7f60640cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 -- 172.21.15.6:0/340952276 shutdown_connections 2024-03-22T13:04:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.158+0000 7f6069400640 1 -- 172.21.15.6:0/340952276 wait complete. 2024-03-22T13:04:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.160+0000 7f6069400640 1 Processor -- start 2024-03-22T13:04:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.160+0000 7f6069400640 1 -- start start 2024-03-22T13:04:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.161+0000 7f6069400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f606406f8c0 0x7f606415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.161+0000 7f6069400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f606415b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.161+0000 7f6069400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064152400 0x7f6064145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.161+0000 7f6069400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6064158b90 con 0x7f606406f8c0 2024-03-22T13:04:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.161+0000 7f6069400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60641588b0 con 0x7f6064152400 2024-03-22T13:04:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.161+0000 7f6069400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60641527e0 con 0x7f6064071c10 2024-03-22T13:04:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.162+0000 7f6063200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064152400 0x7f6064145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.162+0000 7f6061e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f606415b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.162+0000 7f6062800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f606406f8c0 0x7f606415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.162+0000 7f6061e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f606415b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33956/0 (socket says 172.21.15.6:33956) 2024-03-22T13:04:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.162+0000 7f6061e00640 1 -- 172.21.15.6:0/1817284574 learned_addr learned my addr 172.21.15.6:0/1817284574 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.162+0000 7f6063200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064152400 0x7f6064145630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36750/0 (socket says 172.21.15.6:36750) 2024-03-22T13:04:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.166+0000 7f6062800640 1 -- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 msgr2=0x7f606415b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.166+0000 7f6062800640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f606415b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.166+0000 7f6062800640 1 -- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064152400 msgr2=0x7f6064145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.166+0000 7f6062800640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064152400 0x7f6064145630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.166+0000 7f6062800640 1 -- 172.21.15.6:0/1817284574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606414f7b0 con 0x7f606406f8c0 2024-03-22T13:04:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.170+0000 7f6062800640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f606406f8c0 0x7f606415b330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f605400c3c0 tx=0x7f6054001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.172+0000 7f605aa00640 1 -- 172.21.15.6:0/1817284574 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054017070 con 0x7f606406f8c0 2024-03-22T13:04:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.172+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60641579a0 con 0x7f606406f8c0 2024-03-22T13:04:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.172+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6064157240 con 0x7f606406f8c0 2024-03-22T13:04:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.173+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60640756c0 con 0x7f606406f8c0 2024-03-22T13:04:03.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.176+0000 7f605aa00640 1 -- 172.21.15.6:0/1817284574 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6054005ce0 con 0x7f606406f8c0 2024-03-22T13:04:03.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.176+0000 7f605aa00640 1 -- 172.21.15.6:0/1817284574 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054021060 con 0x7f606406f8c0 2024-03-22T13:04:03.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.483+0000 7f605aa00640 1 -- 172.21.15.6:0/1817284574 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f605401a070 con 0x7f606406f8c0 2024-03-22T13:04:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.484+0000 7f605aa00640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f604406bbd0 0x7f604406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.485+0000 7f6061e00640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f604406bbd0 0x7f604406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.486+0000 7f6061e00640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f604406bbd0 0x7f604406e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6064072ec0 tx=0x7f6050009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.507+0000 7f605aa00640 1 -- 172.21.15.6:0/1817284574 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6054056a00 con 0x7f606406f8c0 2024-03-22T13:04:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.509+0000 7f605aa00640 1 -- 172.21.15.6:0/1817284574 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6054007e20 con 0x7f606406f8c0 2024-03-22T13:04:03.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.781+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60641481a0 con 0x7f606406f8c0 2024-03-22T13:04:03.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.799+0000 7f605aa00640 1 -- 172.21.15.6:0/1817284574 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f6054058960 con 0x7f606406f8c0 2024-03-22T13:04:03.799 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:03.800 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:03.800 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:03.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f604406bbd0 msgr2=0x7f604406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:03.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f604406bbd0 0x7f604406e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6064072ec0 tx=0x7f6050009190 comp rx=0 tx=0).stop 2024-03-22T13:04:03.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f606406f8c0 msgr2=0x7f606415b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:03.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f606406f8c0 0x7f606415b330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f605400c3c0 tx=0x7f6054001e70 comp rx=0 tx=0).stop 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 shutdown_connections 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f604406bbd0 0x7f604406e090 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064152400 0x7f6064145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6064071c10 0x7f606415b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 --2- 172.21.15.6:0/1817284574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f606406f8c0 0x7f606415b330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 >> 172.21.15.6:0/1817284574 conn(0x7f60640cb930 msgr2=0x7f606414f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 shutdown_connections 2024-03-22T13:04:03.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:03.802+0000 7f6069400640 1 -- 172.21.15.6:0/1817284574 wait complete. 2024-03-22T13:04:04.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:04.423 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:05.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.628+0000 7f1b6ee00640 1 Processor -- start 2024-03-22T13:04:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.628+0000 7f1b6ee00640 1 -- start start 2024-03-22T13:04:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.629+0000 7f1b6ee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b6806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.629+0000 7f1b6ee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b6815b0e0 0x7f1b6815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.629+0000 7f1b6ee00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6806c7a0 con 0x7f1b680ca520 2024-03-22T13:04:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.629+0000 7f1b6ee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b680cd000 con 0x7f1b6815b0e0 2024-03-22T13:04:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.629+0000 7f1b6ee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6806ca80 con 0x7f1b680ccad0 2024-03-22T13:04:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.630+0000 7f1b67e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1b680ca520 0x7f1b680cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53464/0 (socket says 172.21.15.6:53464) 2024-03-22T13:04:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.630+0000 7f1b67e00640 1 -- 172.21.15.6:0/443359928 learned_addr learned my addr 172.21.15.6:0/443359928 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.630+0000 7f1b5f400640 1 --2- 172.21.15.6:0/443359928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b6806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.631+0000 7f1b6ce00640 1 --2- 172.21.15.6:0/443359928 >> v2:172.21.15.6:3300/0 conn(0x7f1b6815b0e0 0x7f1b6815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.634+0000 7f1b67400640 1 -- 172.21.15.6:0/443359928 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094695304 0 0) 0x7f1b6806c7a0 con 0x7f1b680ca520 2024-03-22T13:04:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.634+0000 7f1b67400640 1 -- 172.21.15.6:0/443359928 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b4c003520 con 0x7f1b680ca520 2024-03-22T13:04:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.634+0000 7f1b6ce00640 1 -- 172.21.15.6:0/443359928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 msgr2=0x7f1b6806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.634+0000 7f1b6ce00640 1 --2- 172.21.15.6:0/443359928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b6806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.634+0000 7f1b6ce00640 1 -- 172.21.15.6:0/443359928 >> v1:172.21.15.92:6789/0 conn(0x7f1b680ca520 legacy=0x7f1b680cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.634+0000 7f1b6ce00640 1 -- 172.21.15.6:0/443359928 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6806cd60 con 0x7f1b6815b0e0 2024-03-22T13:04:05.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.636+0000 7f1b6ce00640 1 --2- 172.21.15.6:0/443359928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 0x7f1b6815b980 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1b600097a0 tx=0x7f1b6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a476b030467a33c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:05.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.641+0000 7f1b67400640 1 -- 172.21.15.6:0/443359928 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b600315a0 con 0x7f1b6815b0e0 2024-03-22T13:04:05.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.641+0000 7f1b67400640 1 -- 172.21.15.6:0/443359928 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b60037070 con 0x7f1b6815b0e0 2024-03-22T13:04:05.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.641+0000 7f1b67400640 1 -- 172.21.15.6:0/443359928 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60031960 con 0x7f1b6815b0e0 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 -- 172.21.15.6:0/443359928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 msgr2=0x7f1b6815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/443359928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 0x7f1b6815b980 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1b600097a0 tx=0x7f1b6002b840 comp rx=0 tx=0).stop 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 -- 172.21.15.6:0/443359928 shutdown_connections 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/443359928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 0x7f1b6815b980 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/443359928 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b6806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 -- 172.21.15.6:0/443359928 >> 172.21.15.6:0/443359928 conn(0x7f1b680b16b0 msgr2=0x7f1b680b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 -- 172.21.15.6:0/443359928 shutdown_connections 2024-03-22T13:04:05.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.642+0000 7f1b6ee00640 1 -- 172.21.15.6:0/443359928 wait complete. 2024-03-22T13:04:05.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.644+0000 7f1b6ee00640 1 Processor -- start 2024-03-22T13:04:05.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.644+0000 7f1b6ee00640 1 -- start start 2024-03-22T13:04:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.645+0000 7f1b6ee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b680ca520 0x7f1b681f0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.645+0000 7f1b6ee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b680b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.645+0000 7f1b6ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 0x7f1b680c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.645+0000 7f1b6ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b68154c20 con 0x7f1b6815b0e0 2024-03-22T13:04:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.645+0000 7f1b6ee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6806d320 con 0x7f1b680ca520 2024-03-22T13:04:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.645+0000 7f1b6ee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b680ca900 con 0x7f1b680ccad0 2024-03-22T13:04:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.646+0000 7f1b67e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b680ca520 0x7f1b681f0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.646+0000 7f1b5f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b680b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.646+0000 7f1b6ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 0x7f1b680c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.646+0000 7f1b67e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b680ca520 0x7f1b681f0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36762/0 (socket says 172.21.15.6:36762) 2024-03-22T13:04:05.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.646+0000 7f1b67e00640 1 -- 172.21.15.6:0/1479086910 learned_addr learned my addr 172.21.15.6:0/1479086910 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:05.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.650+0000 7f1b67e00640 1 -- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 msgr2=0x7f1b680b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:05.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.650+0000 7f1b67e00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b680b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:05.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.650+0000 7f1b67e00640 1 -- 172.21.15.6:0/1479086910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 msgr2=0x7f1b680c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:05.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.650+0000 7f1b67e00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 0x7f1b680c34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:05.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.650+0000 7f1b67e00640 1 -- 172.21.15.6:0/1479086910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68155f10 con 0x7f1b680ca520 2024-03-22T13:04:05.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.652+0000 7f1b67e00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b680ca520 0x7f1b681f0d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c8f0 tx=0x7f1b5800cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:05.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.655+0000 7f1b64c00640 1 -- 172.21.15.6:0/1479086910 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b58018070 con 0x7f1b680ca520 2024-03-22T13:04:05.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.655+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b681561f0 con 0x7f1b680ca520 2024-03-22T13:04:05.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.655+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b68156be0 con 0x7f1b680ca520 2024-03-22T13:04:05.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.656+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b680756c0 con 0x7f1b680ca520 2024-03-22T13:04:05.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.657+0000 7f1b64c00640 1 -- 172.21.15.6:0/1479086910 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5800a640 con 0x7f1b680ca520 2024-03-22T13:04:05.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.657+0000 7f1b64c00640 1 -- 172.21.15.6:0/1479086910 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b58022060 con 0x7f1b680ca520 2024-03-22T13:04:05.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.946+0000 7f1b64c00640 1 -- 172.21.15.6:0/1479086910 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b5801b070 con 0x7f1b680ca520 2024-03-22T13:04:05.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.948+0000 7f1b64c00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b4406bbd0 0x7f1b4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:05.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.949+0000 7f1b5f400640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b4406bbd0 0x7f1b4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:05.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.950+0000 7f1b5f400640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b4406bbd0 0x7f1b4406e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b681f48d0 tx=0x7f1b50009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:05.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.951+0000 7f1b64c00640 1 -- 172.21.15.6:0/1479086910 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1b58056f20 con 0x7f1b680ca520 2024-03-22T13:04:05.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:05.962+0000 7f1b64c00640 1 -- 172.21.15.6:0/1479086910 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b58058e80 con 0x7f1b680ca520 2024-03-22T13:04:06.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.219+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b680c2fe0 con 0x7f1b680ca520 2024-03-22T13:04:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.238+0000 7f1b64c00640 1 -- 172.21.15.6:0/1479086910 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f1b5805cf20 con 0x7f1b680ca520 2024-03-22T13:04:06.238 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:06.238 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:06.238 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.241+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b4406bbd0 msgr2=0x7f1b4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.241+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b4406bbd0 0x7f1b4406e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b681f48d0 tx=0x7f1b50009190 comp rx=0 tx=0).stop 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.241+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b680ca520 msgr2=0x7f1b681f0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.241+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b680ca520 0x7f1b681f0d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c8f0 tx=0x7f1b5800cdb0 comp rx=0 tx=0).stop 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 shutdown_connections 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b6815b0e0 0x7f1b680c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b680ccad0 0x7f1b680b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b4406bbd0 0x7f1b4406e090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 --2- 172.21.15.6:0/1479086910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b680ca520 0x7f1b681f0d20 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c8f0 tx=0x7f1b5800cdb0 comp rx=0 tx=0).stop 2024-03-22T13:04:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 >> 172.21.15.6:0/1479086910 conn(0x7f1b680b16b0 msgr2=0x7f1b680b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 shutdown_connections 2024-03-22T13:04:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:06.242+0000 7f1b6ee00640 1 -- 172.21.15.6:0/1479086910 wait complete. 2024-03-22T13:04:06.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:06.864 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:07.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.058+0000 7f0e2da00640 1 Processor -- start 2024-03-22T13:04:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.058+0000 7f0e2da00640 1 -- start start 2024-03-22T13:04:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.060+0000 7f0e2da00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e28152400 0x7f0e2815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.060+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e2815bc80 0x7f0e280716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.060+0000 7f0e2da00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e281590b0 con 0x7f0e2806f8c0 2024-03-22T13:04:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.060+0000 7f0e2da00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e280724d0 con 0x7f0e28152400 2024-03-22T13:04:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.060+0000 7f0e2da00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e280721f0 con 0x7f0e2815bc80 2024-03-22T13:04:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.061+0000 7f0e26000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e2815bc80 0x7f0e280716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.061+0000 7f0e26000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e2815bc80 0x7f0e280716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33982/0 (socket says 172.21.15.6:33982) 2024-03-22T13:04:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.061+0000 7f0e26000640 1 -- 172.21.15.6:0/2634751864 learned_addr learned my addr 172.21.15.6:0/2634751864 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.061+0000 7f0e26a00640 1 --2- 172.21.15.6:0/2634751864 >> v2:172.21.15.92:3300/0 conn(0x7f0e28152400 0x7f0e2815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.063+0000 7f0e26000640 1 -- 172.21.15.6:0/2634751864 >> v2:172.21.15.92:3300/0 conn(0x7f0e28152400 msgr2=0x7f0e2815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.063+0000 7f0e26000640 1 --2- 172.21.15.6:0/2634751864 >> v2:172.21.15.92:3300/0 conn(0x7f0e28152400 0x7f0e2815b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.064+0000 7f0e26000640 1 -- 172.21.15.6:0/2634751864 >> v1:172.21.15.92:6789/0 conn(0x7f0e2806f8c0 legacy=0x7f0e280c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.064+0000 7f0e26000640 1 -- 172.21.15.6:0/2634751864 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e281582f0 con 0x7f0e2815bc80 2024-03-22T13:04:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.066+0000 7f0e26000640 1 --2- 172.21.15.6:0/2634751864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e2815bc80 0x7f0e280716d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009870 tx=0x7f0e1402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c4762b754272043 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.069+0000 7f0e25600640 1 -- 172.21.15.6:0/2634751864 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1402f840 con 0x7f0e2815bc80 2024-03-22T13:04:08.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.071+0000 7f0e25600640 1 -- 172.21.15.6:0/2634751864 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1402f9e0 con 0x7f0e2815bc80 2024-03-22T13:04:08.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.071+0000 7f0e25600640 1 -- 172.21.15.6:0/2634751864 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1402fdb0 con 0x7f0e2815bc80 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 -- 172.21.15.6:0/2634751864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e2815bc80 msgr2=0x7f0e280716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 --2- 172.21.15.6:0/2634751864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e2815bc80 0x7f0e280716d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009870 tx=0x7f0e1402ba20 comp rx=0 tx=0).stop 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 -- 172.21.15.6:0/2634751864 shutdown_connections 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 --2- 172.21.15.6:0/2634751864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e2815bc80 0x7f0e280716d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 --2- 172.21.15.6:0/2634751864 >> v2:172.21.15.92:3300/0 conn(0x7f0e28152400 0x7f0e2815b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 -- 172.21.15.6:0/2634751864 >> 172.21.15.6:0/2634751864 conn(0x7f0e280cb930 msgr2=0x7f0e280cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 -- 172.21.15.6:0/2634751864 shutdown_connections 2024-03-22T13:04:08.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.072+0000 7f0e2da00640 1 -- 172.21.15.6:0/2634751864 wait complete. 2024-03-22T13:04:08.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.075+0000 7f0e2da00640 1 Processor -- start 2024-03-22T13:04:08.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.075+0000 7f0e2da00640 1 -- start start 2024-03-22T13:04:08.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.076+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f8c0 0x7f0e2815b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:08.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.076+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e28152400 0x7f0e280c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:08.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.076+0000 7f0e2da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e2815bc80 0x7f0e28148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:08.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.076+0000 7f0e2da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e28158b90 con 0x7f0e2815bc80 2024-03-22T13:04:08.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.076+0000 7f0e2da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e281588b0 con 0x7f0e2806f8c0 2024-03-22T13:04:08.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.076+0000 7f0e2da00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e281527e0 con 0x7f0e28152400 2024-03-22T13:04:08.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.077+0000 7f0e26000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e28152400 0x7f0e280c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:08.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.077+0000 7f0e26a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f8c0 0x7f0e2815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:08.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.077+0000 7f0e26000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e28152400 0x7f0e280c40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33986/0 (socket says 172.21.15.6:33986) 2024-03-22T13:04:08.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.077+0000 7f0e26000640 1 -- 172.21.15.6:0/3367366971 learned_addr learned my addr 172.21.15.6:0/3367366971 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:08.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.077+0000 7f0e27400640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e2815bc80 0x7f0e28148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.080+0000 7f0e26000640 1 -- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f8c0 msgr2=0x7f0e2815b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.080+0000 7f0e26000640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f8c0 0x7f0e2815b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.080+0000 7f0e26000640 1 -- 172.21.15.6:0/3367366971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e2815bc80 msgr2=0x7f0e28148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.080+0000 7f0e26000640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e2815bc80 0x7f0e28148260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.080+0000 7f0e26000640 1 -- 172.21.15.6:0/3367366971 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2814de30 con 0x7f0e28152400 2024-03-22T13:04:08.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.083+0000 7f0e26000640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e28152400 0x7f0e280c40d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e140090b0 tx=0x7f0e14004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:08.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.086+0000 7f0e1ea00640 1 -- 172.21.15.6:0/3367366971 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14046070 con 0x7f0e28152400 2024-03-22T13:04:08.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.086+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e280cb1b0 con 0x7f0e28152400 2024-03-22T13:04:08.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.086+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2814e0b0 con 0x7f0e28152400 2024-03-22T13:04:08.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.087+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e28072750 con 0x7f0e28152400 2024-03-22T13:04:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.297+0000 7f0e1ea00640 1 -- 172.21.15.6:0/3367366971 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1402f940 con 0x7f0e28152400 2024-03-22T13:04:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.373+0000 7f0e1ea00640 1 -- 172.21.15.6:0/3367366971 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14050060 con 0x7f0e28152400 2024-03-22T13:04:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.377+0000 7f0e1ea00640 1 -- 172.21.15.6:0/3367366971 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0e14049070 con 0x7f0e28152400 2024-03-22T13:04:08.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.379+0000 7f0e1ea00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e0806bbd0 0x7f0e0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:08.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.379+0000 7f0e1ea00640 1 -- 172.21.15.6:0/3367366971 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0e140ad4a0 con 0x7f0e28152400 2024-03-22T13:04:08.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.379+0000 7f0e26a00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e0806bbd0 0x7f0e0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.380+0000 7f0e26a00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e0806bbd0 0x7f0e0806e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009b40 tx=0x7f0e18009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:08.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.390+0000 7f0e1ea00640 1 -- 172.21.15.6:0/3367366971 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0e28072750 con 0x7f0e28152400 2024-03-22T13:04:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.641+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e280756c0 con 0x7f0e28152400 2024-03-22T13:04:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.656+0000 7f0e1ea00640 1 -- 172.21.15.6:0/3367366971 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f0e2814de30 con 0x7f0e28152400 2024-03-22T13:04:08.656 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:08.656 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e0806bbd0 msgr2=0x7f0e0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e0806bbd0 0x7f0e0806e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009b40 tx=0x7f0e18009190 comp rx=0 tx=0).stop 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e28152400 msgr2=0x7f0e280c40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e28152400 0x7f0e280c40d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e140090b0 tx=0x7f0e14004730 comp rx=0 tx=0).stop 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 shutdown_connections 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e2815bc80 0x7f0e28148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e28152400 0x7f0e280c40d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e0806bbd0 0x7f0e0806e090 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 --2- 172.21.15.6:0/3367366971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2806f8c0 0x7f0e2815b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.659+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 >> 172.21.15.6:0/3367366971 conn(0x7f0e280cb930 msgr2=0x7f0e2814ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.660+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 shutdown_connections 2024-03-22T13:04:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:08.660+0000 7f0e2da00640 1 -- 172.21.15.6:0/3367366971 wait complete. 2024-03-22T13:04:09.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:09.280 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:10.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.477+0000 7f3b5a400640 1 Processor -- start 2024-03-22T13:04:10.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.477+0000 7f3b5a400640 1 -- start start 2024-03-22T13:04:10.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.478+0000 7f3b5a400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3b540c5210 0x7f3b54158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.478+0000 7f3b5a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b5406f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.478+0000 7f3b5a400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5406e0f0 con 0x7f3b540c55f0 2024-03-22T13:04:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.478+0000 7f3b5a400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b540c6f90 con 0x7f3b540c5210 2024-03-22T13:04:10.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.478+0000 7f3b5a400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5406d350 con 0x7f3b5406ede0 2024-03-22T13:04:10.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.479+0000 7f3b59a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b5406f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:10.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.480+0000 7f3b59a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b5406f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33994/0 (socket says 172.21.15.6:33994) 2024-03-22T13:04:10.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.480+0000 7f3b59a00640 1 -- 172.21.15.6:0/1435368686 learned_addr learned my addr 172.21.15.6:0/1435368686 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.483+0000 7f3b59a00640 1 -- 172.21.15.6:0/1435368686 >> v2:172.21.15.92:3300/0 conn(0x7f3b540c5210 msgr2=0x7f3b54158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.483+0000 7f3b59a00640 1 --2- 172.21.15.6:0/1435368686 >> v2:172.21.15.92:3300/0 conn(0x7f3b540c5210 0x7f3b54158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.483+0000 7f3b59a00640 1 -- 172.21.15.6:0/1435368686 >> v1:172.21.15.6:6789/0 conn(0x7f3b540c55f0 legacy=0x7f3b540c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:10.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.483+0000 7f3b59a00640 1 -- 172.21.15.6:0/1435368686 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b540c6cb0 con 0x7f3b5406ede0 2024-03-22T13:04:10.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.486+0000 7f3b59a00640 1 --2- 172.21.15.6:0/1435368686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b5406f210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009e50 tx=0x7f3b4c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69016605919e020b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:10.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.491+0000 7f3b53400640 1 -- 172.21.15.6:0/1435368686 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c02ca70 con 0x7f3b5406ede0 2024-03-22T13:04:10.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b53400640 1 -- 172.21.15.6:0/1435368686 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c02cc10 con 0x7f3b5406ede0 2024-03-22T13:04:10.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b53400640 1 -- 172.21.15.6:0/1435368686 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0343f0 con 0x7f3b5406ede0 2024-03-22T13:04:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b5a400640 1 -- 172.21.15.6:0/1435368686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 msgr2=0x7f3b5406f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b5a400640 1 --2- 172.21.15.6:0/1435368686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b5406f210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009e50 tx=0x7f3b4c02bed0 comp rx=0 tx=0).stop 2024-03-22T13:04:10.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b5a400640 1 -- 172.21.15.6:0/1435368686 shutdown_connections 2024-03-22T13:04:10.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b5a400640 1 --2- 172.21.15.6:0/1435368686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b5406f210 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:10.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b5a400640 1 --2- 172.21.15.6:0/1435368686 >> v2:172.21.15.92:3300/0 conn(0x7f3b540c5210 0x7f3b54158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:10.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b5a400640 1 -- 172.21.15.6:0/1435368686 >> 172.21.15.6:0/1435368686 conn(0x7f3b540cb890 msgr2=0x7f3b540cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:10.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.492+0000 7f3b5a400640 1 -- 172.21.15.6:0/1435368686 shutdown_connections 2024-03-22T13:04:10.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.493+0000 7f3b5a400640 1 -- 172.21.15.6:0/1435368686 wait complete. 2024-03-22T13:04:10.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.496+0000 7f3b5a400640 1 Processor -- start 2024-03-22T13:04:10.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.497+0000 7f3b5a400640 1 -- start start 2024-03-22T13:04:10.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.498+0000 7f3b5a400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b540c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:10.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.498+0000 7f3b5a400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540c5210 0x7f3b54152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:10.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.499+0000 7f3b5a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540c55f0 0x7f3b541e5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:10.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.499+0000 7f3b5a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5406dbd0 con 0x7f3b540c55f0 2024-03-22T13:04:10.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.499+0000 7f3b5a400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5406d8f0 con 0x7f3b540c5210 2024-03-22T13:04:10.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.499+0000 7f3b5a400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b540c69a0 con 0x7f3b5406ede0 2024-03-22T13:04:10.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.499+0000 7f3b53e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540c5210 0x7f3b54152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:10.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.500+0000 7f3b59000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b540c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:10.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.500+0000 7f3b53e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540c5210 0x7f3b54152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36792/0 (socket says 172.21.15.6:36792) 2024-03-22T13:04:10.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.500+0000 7f3b53e00640 1 -- 172.21.15.6:0/4075147129 learned_addr learned my addr 172.21.15.6:0/4075147129 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:10.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.503+0000 7f3b59000640 1 -- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540c5210 msgr2=0x7f3b54152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:10.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.503+0000 7f3b59000640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540c5210 0x7f3b54152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:10.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.503+0000 7f3b59000640 1 -- 172.21.15.6:0/4075147129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540c55f0 msgr2=0x7f3b541e5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:10.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.503+0000 7f3b59000640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540c55f0 0x7f3b541e5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:10.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.503+0000 7f3b59000640 1 -- 172.21.15.6:0/4075147129 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b541e18b0 con 0x7f3b5406ede0 2024-03-22T13:04:10.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.506+0000 7f3b59000640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b540c4a50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800b420 tx=0x7f3b4800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.508+0000 7f3b50c00640 1 -- 172.21.15.6:0/4075147129 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48011830 con 0x7f3b5406ede0 2024-03-22T13:04:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.508+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b54153460 con 0x7f3b5406ede0 2024-03-22T13:04:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.508+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b54153a20 con 0x7f3b5406ede0 2024-03-22T13:04:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.510+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b54075650 con 0x7f3b5406ede0 2024-03-22T13:04:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.510+0000 7f3b50c00640 1 -- 172.21.15.6:0/4075147129 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b48016070 con 0x7f3b5406ede0 2024-03-22T13:04:10.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.510+0000 7f3b50c00640 1 -- 172.21.15.6:0/4075147129 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48009200 con 0x7f3b5406ede0 2024-03-22T13:04:10.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.535+0000 7f3b50c00640 1 -- 172.21.15.6:0/4075147129 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3b48011bf0 con 0x7f3b5406ede0 2024-03-22T13:04:10.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.537+0000 7f3b50c00640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3406bbd0 0x7f3b3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:10.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.538+0000 7f3b53e00640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3406bbd0 0x7f3b3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:10.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.539+0000 7f3b53e00640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3406bbd0 0x7f3b3406e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3b54154150 tx=0x7f3b400090e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:10.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.539+0000 7f3b50c00640 1 -- 172.21.15.6:0/4075147129 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3b480575f0 con 0x7f3b5406ede0 2024-03-22T13:04:10.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:10.864+0000 7f3b50c00640 1 -- 172.21.15.6:0/4075147129 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3b48059c50 con 0x7f3b5406ede0 2024-03-22T13:04:11.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.114+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b540ca520 con 0x7f3b5406ede0 2024-03-22T13:04:11.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.134+0000 7f3b50c00640 1 -- 172.21.15.6:0/4075147129 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f3b48059550 con 0x7f3b5406ede0 2024-03-22T13:04:11.134 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:11.134 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:11.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3406bbd0 msgr2=0x7f3b3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:11.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3406bbd0 0x7f3b3406e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3b54154150 tx=0x7f3b400090e0 comp rx=0 tx=0).stop 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 msgr2=0x7f3b540c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b540c4a50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800b420 tx=0x7f3b4800b8e0 comp rx=0 tx=0).stop 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 shutdown_connections 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b540c55f0 0x7f3b541e5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b3406bbd0 0x7f3b3406e090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b540c5210 0x7f3b54152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 --2- 172.21.15.6:0/4075147129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b5406ede0 0x7f3b540c4a50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 >> 172.21.15.6:0/4075147129 conn(0x7f3b540cb890 msgr2=0x7f3b54071480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 shutdown_connections 2024-03-22T13:04:11.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:11.137+0000 7f3b5a400640 1 -- 172.21.15.6:0/4075147129 wait complete. 2024-03-22T13:04:11.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:11.758 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:12.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:12.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.957+0000 7f1cc5c00640 1 Processor -- start 2024-03-22T13:04:12.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.957+0000 7f1cc5c00640 1 -- start start 2024-03-22T13:04:12.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.958+0000 7f1cc5c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1cc00cc640 0x7f1cc006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:12.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.958+0000 7f1cc5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1cc00cca20 0x7f1cc00ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:12.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.958+0000 7f1cc5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc015d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:12.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.958+0000 7f1cc5c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc006d330 con 0x7f1cc00cca20 2024-03-22T13:04:12.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.958+0000 7f1cc5c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc006dd00 con 0x7f1cc00cc640 2024-03-22T13:04:12.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.958+0000 7f1cc5c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc006c440 con 0x7f1cc015b7a0 2024-03-22T13:04:12.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.959+0000 7f1cbfe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1cc00cc640 0x7f1cc006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:12.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.959+0000 7f1cc5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:12.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.959+0000 7f1cbfe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1cc00cc640 0x7f1cc006d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40684/0 (socket says 172.21.15.6:40684) 2024-03-22T13:04:12.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.959+0000 7f1cbfe00640 1 -- 172.21.15.6:0/3411689363 learned_addr learned my addr 172.21.15.6:0/3411689363 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:12.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.962+0000 7f1cc5200640 1 -- 172.21.15.6:0/3411689363 >> v2:172.21.15.92:3300/0 conn(0x7f1cc00cc640 msgr2=0x7f1cc006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:12.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.962+0000 7f1cc5200640 1 --2- 172.21.15.6:0/3411689363 >> v2:172.21.15.92:3300/0 conn(0x7f1cc00cc640 0x7f1cc006d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.962+0000 7f1cc5200640 1 -- 172.21.15.6:0/3411689363 >> v2:172.21.15.6:3300/0 conn(0x7f1cc00cca20 msgr2=0x7f1cc00ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:12.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.962+0000 7f1cc5200640 1 --2- 172.21.15.6:0/3411689363 >> v2:172.21.15.6:3300/0 conn(0x7f1cc00cca20 0x7f1cc00ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.962+0000 7f1cc5200640 1 -- 172.21.15.6:0/3411689363 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc006c720 con 0x7f1cc015b7a0 2024-03-22T13:04:12.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.965+0000 7f1cc5200640 1 --2- 172.21.15.6:0/3411689363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc015d250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00cc320 tx=0x7f1cb802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=828b5421085585fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:12.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.970+0000 7f1cbea00640 1 -- 172.21.15.6:0/3411689363 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8039070 con 0x7f1cc015b7a0 2024-03-22T13:04:12.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.970+0000 7f1cbea00640 1 -- 172.21.15.6:0/3411689363 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb802cc60 con 0x7f1cc015b7a0 2024-03-22T13:04:12.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.970+0000 7f1cbea00640 1 -- 172.21.15.6:0/3411689363 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8034660 con 0x7f1cc015b7a0 2024-03-22T13:04:12.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 -- 172.21.15.6:0/3411689363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 msgr2=0x7f1cc015d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:12.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/3411689363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc015d250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00cc320 tx=0x7f1cb802beb0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 -- 172.21.15.6:0/3411689363 shutdown_connections 2024-03-22T13:04:12.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/3411689363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc015d250 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/3411689363 >> v2:172.21.15.6:3300/0 conn(0x7f1cc00cca20 0x7f1cc00ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/3411689363 >> v2:172.21.15.92:3300/0 conn(0x7f1cc00cc640 0x7f1cc006d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 -- 172.21.15.6:0/3411689363 >> 172.21.15.6:0/3411689363 conn(0x7f1cc00cf9c0 msgr2=0x7f1cc00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:12.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 -- 172.21.15.6:0/3411689363 shutdown_connections 2024-03-22T13:04:12.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.971+0000 7f1cc5c00640 1 -- 172.21.15.6:0/3411689363 wait complete. 2024-03-22T13:04:12.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.974+0000 7f1cc5c00640 1 Processor -- start 2024-03-22T13:04:12.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.974+0000 7f1cc5c00640 1 -- start start 2024-03-22T13:04:12.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cc5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cc00cc640 0x7f1cc01e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:12.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cc5c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc00cca20 0x7f1cc00b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cc5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc00b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cc5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc006cfc0 con 0x7f1cc00cc640 2024-03-22T13:04:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cc5c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc006cce0 con 0x7f1cc00cca20 2024-03-22T13:04:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cc5c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc015b000 con 0x7f1cc015b7a0 2024-03-22T13:04:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cc5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc00b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.976+0000 7f1cbf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc00cca20 0x7f1cc00b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.977+0000 7f1cbf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc00cca20 0x7f1cc00b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40688/0 (socket says 172.21.15.6:40688) 2024-03-22T13:04:12.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.977+0000 7f1cbf400640 1 -- 172.21.15.6:0/1087316079 learned_addr learned my addr 172.21.15.6:0/1087316079 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:12.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.979+0000 7f1cbfe00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cc00cc640 0x7f1cc01e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:12.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.981+0000 7f1cc5200640 1 -- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc00cca20 msgr2=0x7f1cc00b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:12.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.981+0000 7f1cc5200640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc00cca20 0x7f1cc00b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.981+0000 7f1cc5200640 1 -- 172.21.15.6:0/1087316079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cc00cc640 msgr2=0x7f1cc01e5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:12.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.981+0000 7f1cc5200640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cc00cc640 0x7f1cc01e5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:12.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.981+0000 7f1cc5200640 1 -- 172.21.15.6:0/1087316079 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc01f0690 con 0x7f1cc015b7a0 2024-03-22T13:04:12.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.983+0000 7f1cc5200640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc00b52a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1cc006c5c0 tx=0x7f1cb8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:12.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.988+0000 7f1cb7e00640 1 -- 172.21.15.6:0/1087316079 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8044020 con 0x7f1cc015b7a0 2024-03-22T13:04:12.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.988+0000 7f1cb7e00640 1 -- 172.21.15.6:0/1087316079 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb803f470 con 0x7f1cc015b7a0 2024-03-22T13:04:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.988+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc01f0970 con 0x7f1cc015b7a0 2024-03-22T13:04:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.988+0000 7f1cb7e00640 1 -- 172.21.15.6:0/1087316079 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8039040 con 0x7f1cc015b7a0 2024-03-22T13:04:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.989+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc01f0f30 con 0x7f1cc015b7a0 2024-03-22T13:04:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:12.990+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc0075650 con 0x7f1cc015b7a0 2024-03-22T13:04:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.015+0000 7f1cb7e00640 1 -- 172.21.15.6:0/1087316079 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1cb8047020 con 0x7f1cc015b7a0 2024-03-22T13:04:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.016+0000 7f1cb7e00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c9c06bc20 0x7f1c9c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.017+0000 7f1cbfe00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c9c06bc20 0x7f1c9c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.017+0000 7f1cb7e00640 1 -- 172.21.15.6:0/1087316079 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1cb8079d00 con 0x7f1cc015b7a0 2024-03-22T13:04:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.018+0000 7f1cbfe00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c9c06bc20 0x7f1c9c06e0e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009bd0 tx=0x7f1cb0009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:13.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.307+0000 7f1cb7e00640 1 -- 172.21.15.6:0/1087316079 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1cb804e030 con 0x7f1cc015b7a0 2024-03-22T13:04:13.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.558+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cc00af100 con 0x7f1cc015b7a0 2024-03-22T13:04:13.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.629+0000 7f1cb7e00640 1 -- 172.21.15.6:0/1087316079 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f1cb807bc60 con 0x7f1cc015b7a0 2024-03-22T13:04:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:13.630 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:13.630 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c9c06bc20 msgr2=0x7f1c9c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c9c06bc20 0x7f1c9c06e0e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009bd0 tx=0x7f1cb0009240 comp rx=0 tx=0).stop 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 msgr2=0x7f1cc00b52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc00b52a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1cc006c5c0 tx=0x7f1cb8004a40 comp rx=0 tx=0).stop 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 shutdown_connections 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cc015b7a0 0x7f1cc00b52a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1c9c06bc20 0x7f1c9c06e0e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc00cca20 0x7f1cc00b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 --2- 172.21.15.6:0/1087316079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cc00cc640 0x7f1cc01e5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 >> 172.21.15.6:0/1087316079 conn(0x7f1cc00cf9c0 msgr2=0x7f1cc00c6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 shutdown_connections 2024-03-22T13:04:13.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:13.633+0000 7f1cc5c00640 1 -- 172.21.15.6:0/1087316079 wait complete. 2024-03-22T13:04:14.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:14.254 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:15.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:15.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.454+0000 7fc7f5000640 1 Processor -- start 2024-03-22T13:04:15.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.455+0000 7fc7f5000640 1 -- start start 2024-03-22T13:04:15.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.456+0000 7fc7f5000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc7f006b080 0x7fc7f006b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:15.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.456+0000 7fc7f5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7f006fea0 0x7fc7f015d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:15.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.456+0000 7fc7f5000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f00c3d30 con 0x7fc7f015de30 2024-03-22T13:04:15.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.456+0000 7fc7f5000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f00c3a50 con 0x7fc7f006fea0 2024-03-22T13:04:15.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.456+0000 7fc7f5000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f015a450 con 0x7fc7f006b080 2024-03-22T13:04:15.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.457+0000 7fc7ed400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7f006fea0 0x7fc7f015d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:15.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.457+0000 7fc7ede00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc7f006b080 0x7fc7f006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:15.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.457+0000 7fc7ed400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7f006fea0 0x7fc7f015d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46206/0 (socket says 172.21.15.6:46206) 2024-03-22T13:04:15.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.457+0000 7fc7ed400640 1 -- 172.21.15.6:0/3506293166 learned_addr learned my addr 172.21.15.6:0/3506293166 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.460+0000 7fc7ed400640 1 -- 172.21.15.6:0/3506293166 >> v2:172.21.15.92:3300/0 conn(0x7fc7f006b080 msgr2=0x7fc7f006b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.460+0000 7fc7ed400640 1 --2- 172.21.15.6:0/3506293166 >> v2:172.21.15.92:3300/0 conn(0x7fc7f006b080 0x7fc7f006b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:15.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.460+0000 7fc7ed400640 1 -- 172.21.15.6:0/3506293166 >> v1:172.21.15.92:6789/0 conn(0x7fc7f015de30 legacy=0x7fc7f00c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:15.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.460+0000 7fc7ed400640 1 -- 172.21.15.6:0/3506293166 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f015a730 con 0x7fc7f006fea0 2024-03-22T13:04:15.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.462+0000 7fc7ed400640 1 --2- 172.21.15.6:0/3506293166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006fea0 0x7fc7f015d8f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00504c0 tx=0x7fc7dc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f239efa9adfe66 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:15.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.465+0000 7fc7eca00640 1 -- 172.21.15.6:0/3506293166 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc02f850 con 0x7fc7f006fea0 2024-03-22T13:04:15.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.467+0000 7fc7eca00640 1 -- 172.21.15.6:0/3506293166 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc02f9f0 con 0x7fc7f006fea0 2024-03-22T13:04:15.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.467+0000 7fc7eca00640 1 -- 172.21.15.6:0/3506293166 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc02fdc0 con 0x7fc7f006fea0 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 -- 172.21.15.6:0/3506293166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006fea0 msgr2=0x7fc7f015d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 --2- 172.21.15.6:0/3506293166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006fea0 0x7fc7f015d8f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00504c0 tx=0x7fc7dc02ba30 comp rx=0 tx=0).stop 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 -- 172.21.15.6:0/3506293166 shutdown_connections 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 --2- 172.21.15.6:0/3506293166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006fea0 0x7fc7f015d8f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 --2- 172.21.15.6:0/3506293166 >> v2:172.21.15.92:3300/0 conn(0x7fc7f006b080 0x7fc7f006b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 -- 172.21.15.6:0/3506293166 >> 172.21.15.6:0/3506293166 conn(0x7fc7f00ccb50 msgr2=0x7fc7f00ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 -- 172.21.15.6:0/3506293166 shutdown_connections 2024-03-22T13:04:15.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.468+0000 7fc7f5000640 1 -- 172.21.15.6:0/3506293166 wait complete. 2024-03-22T13:04:15.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.471+0000 7fc7f5000640 1 Processor -- start 2024-03-22T13:04:15.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.471+0000 7fc7f5000640 1 -- start start 2024-03-22T13:04:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.472+0000 7fc7f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006b080 0x7fc7f015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.472+0000 7fc7f5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7f006fea0 0x7fc7f00c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.473+0000 7fc7f5000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7f015de30 0x7fc7f01f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:15.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.473+0000 7fc7f5000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f015b210 con 0x7fc7f006b080 2024-03-22T13:04:15.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.473+0000 7fc7f5000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f015acf0 con 0x7fc7f006fea0 2024-03-22T13:04:15.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.473+0000 7fc7f5000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0154930 con 0x7fc7f015de30 2024-03-22T13:04:15.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.473+0000 7fc7ede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006b080 0x7fc7f015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:15.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.473+0000 7fc7ed400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7f006fea0 0x7fc7f00c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:15.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.473+0000 7fc7ee800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7f015de30 0x7fc7f01f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:15.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.474+0000 7fc7ed400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7f006fea0 0x7fc7f00c65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40716/0 (socket says 172.21.15.6:40716) 2024-03-22T13:04:15.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.474+0000 7fc7ed400640 1 -- 172.21.15.6:0/250044136 learned_addr learned my addr 172.21.15.6:0/250044136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.477+0000 7fc7ee800640 1 -- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7f006fea0 msgr2=0x7fc7f00c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.477+0000 7fc7ee800640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7f006fea0 0x7fc7f00c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.477+0000 7fc7ee800640 1 -- 172.21.15.6:0/250044136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006b080 msgr2=0x7fc7f015b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.477+0000 7fc7ee800640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006b080 0x7fc7f015b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:15.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.477+0000 7fc7ee800640 1 -- 172.21.15.6:0/250044136 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f014ca40 con 0x7fc7f015de30 2024-03-22T13:04:15.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.478+0000 7fc7ede00640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006b080 0x7fc7f015b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:04:15.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.480+0000 7fc7ee800640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7f015de30 0x7fc7f01f0b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0072700 tx=0x7fc7e800cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:15.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.484+0000 7fc7e6000640 1 -- 172.21.15.6:0/250044136 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8003b70 con 0x7fc7f015de30 2024-03-22T13:04:15.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.484+0000 7fc7e6000640 1 -- 172.21.15.6:0/250044136 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e8003d10 con 0x7fc7f015de30 2024-03-22T13:04:15.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.484+0000 7fc7e6000640 1 -- 172.21.15.6:0/250044136 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8005310 con 0x7fc7f015de30 2024-03-22T13:04:15.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.484+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f00c4b50 con 0x7fc7f015de30 2024-03-22T13:04:15.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.484+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f00c6ae0 con 0x7fc7f015de30 2024-03-22T13:04:15.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.486+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f00c9240 con 0x7fc7f015de30 2024-03-22T13:04:15.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.506+0000 7fc7e6000640 1 -- 172.21.15.6:0/250044136 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc7e80054b0 con 0x7fc7f015de30 2024-03-22T13:04:15.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.507+0000 7fc7e6000640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7d006bc20 0x7fc7d006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:15.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.507+0000 7fc7e6000640 1 -- 172.21.15.6:0/250044136 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc7e80057a0 con 0x7fc7f015de30 2024-03-22T13:04:15.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.507+0000 7fc7ede00640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7d006bc20 0x7fc7d006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:15.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.508+0000 7fc7ede00640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7d006bc20 0x7fc7d006e0e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009e90 tx=0x7fc7e0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:15.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:15.757+0000 7fc7e6000640 1 -- 172.21.15.6:0/250044136 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc7f00c9240 con 0x7fc7f015de30 2024-03-22T13:04:16.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.065+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7f00756c0 con 0x7fc7f015de30 2024-03-22T13:04:16.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.085+0000 7fc7e6000640 1 -- 172.21.15.6:0/250044136 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fc7e8059660 con 0x7fc7f015de30 2024-03-22T13:04:16.085 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:16.085 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:16.085 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.087+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7d006bc20 msgr2=0x7fc7d006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.087+0000 7fc7f5000640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7d006bc20 0x7fc7d006e0e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009e90 tx=0x7fc7e0009190 comp rx=0 tx=0).stop 2024-03-22T13:04:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.087+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7f015de30 msgr2=0x7fc7f01f0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.087+0000 7fc7f5000640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7f015de30 0x7fc7f01f0b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0072700 tx=0x7fc7e800cb20 comp rx=0 tx=0).stop 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.087+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 shutdown_connections 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.088+0000 7fc7f5000640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7f015de30 0x7fc7f01f0b70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.088+0000 7fc7f5000640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc7d006bc20 0x7fc7d006e0e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.088+0000 7fc7f5000640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7f006fea0 0x7fc7f00c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.088+0000 7fc7f5000640 1 --2- 172.21.15.6:0/250044136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7f006b080 0x7fc7f015b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.088+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 >> 172.21.15.6:0/250044136 conn(0x7fc7f00ccb50 msgr2=0x7fc7f0152d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.088+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 shutdown_connections 2024-03-22T13:04:16.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:16.088+0000 7fc7f5000640 1 -- 172.21.15.6:0/250044136 wait complete. 2024-03-22T13:04:16.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:16.710 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:17.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:17.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.901+0000 7fc89f200640 1 Processor -- start 2024-03-22T13:04:17.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.901+0000 7fc89f200640 1 -- start start 2024-03-22T13:04:17.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.902+0000 7fc89f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a00c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:17.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.902+0000 7fc89f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8a00cacf0 0x7fc8a015bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:17.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.902+0000 7fc89f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a00c90a0 con 0x7fc8a015c1e0 2024-03-22T13:04:17.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.902+0000 7fc89f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a015b530 con 0x7fc8a00cacf0 2024-03-22T13:04:17.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.902+0000 7fc89f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a01584d0 con 0x7fc8a0152100 2024-03-22T13:04:17.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.906+0000 7fc89de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a00c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:17.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.906+0000 7fc89de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a00c33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45754/0 (socket says 172.21.15.6:45754) 2024-03-22T13:04:17.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.906+0000 7fc89de00640 1 -- 172.21.15.6:0/1918088972 learned_addr learned my addr 172.21.15.6:0/1918088972 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:17.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.910+0000 7fc89de00640 1 -- 172.21.15.6:0/1918088972 >> v2:172.21.15.6:3300/0 conn(0x7fc8a00cacf0 msgr2=0x7fc8a015bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:17.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.910+0000 7fc89de00640 1 --2- 172.21.15.6:0/1918088972 >> v2:172.21.15.6:3300/0 conn(0x7fc8a00cacf0 0x7fc8a015bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:17.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.910+0000 7fc89de00640 1 -- 172.21.15.6:0/1918088972 >> v1:172.21.15.6:6789/0 conn(0x7fc8a015c1e0 legacy=0x7fc8a0071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:17.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.910+0000 7fc89de00640 1 -- 172.21.15.6:0/1918088972 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a01587b0 con 0x7fc8a0152100 2024-03-22T13:04:17.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.913+0000 7fc89de00640 1 --2- 172.21.15.6:0/1918088972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a00c33f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc8a015b6b0 tx=0x7fc89002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d46d0c8034eec908 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:17.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.915+0000 7fc89ca00640 1 -- 172.21.15.6:0/1918088972 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890035020 con 0x7fc8a0152100 2024-03-22T13:04:17.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.917+0000 7fc89ca00640 1 -- 172.21.15.6:0/1918088972 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc890031640 con 0x7fc8a0152100 2024-03-22T13:04:17.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.917+0000 7fc89ca00640 1 -- 172.21.15.6:0/1918088972 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89002d040 con 0x7fc8a0152100 2024-03-22T13:04:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 -- 172.21.15.6:0/1918088972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 msgr2=0x7fc8a00c33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 --2- 172.21.15.6:0/1918088972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a00c33f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc8a015b6b0 tx=0x7fc89002b8f0 comp rx=0 tx=0).stop 2024-03-22T13:04:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 -- 172.21.15.6:0/1918088972 shutdown_connections 2024-03-22T13:04:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 --2- 172.21.15.6:0/1918088972 >> v2:172.21.15.6:3300/0 conn(0x7fc8a00cacf0 0x7fc8a015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 --2- 172.21.15.6:0/1918088972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a00c33f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 -- 172.21.15.6:0/1918088972 >> 172.21.15.6:0/1918088972 conn(0x7fc8a006b4a0 msgr2=0x7fc8a006f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 -- 172.21.15.6:0/1918088972 shutdown_connections 2024-03-22T13:04:17.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.918+0000 7fc89f200640 1 -- 172.21.15.6:0/1918088972 wait complete. 2024-03-22T13:04:17.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.922+0000 7fc89f200640 1 Processor -- start 2024-03-22T13:04:17.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.922+0000 7fc89f200640 1 -- start start 2024-03-22T13:04:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.924+0000 7fc89f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8a00cacf0 0x7fc8a01efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.924+0000 7fc89f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.924+0000 7fc89f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8a015c1e0 0x7fc8a00c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.925+0000 7fc89f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a015b780 con 0x7fc8a015c1e0 2024-03-22T13:04:17.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.925+0000 7fc89f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a0158d70 con 0x7fc8a00cacf0 2024-03-22T13:04:17.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.925+0000 7fc89f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00c3930 con 0x7fc8a0152100 2024-03-22T13:04:17.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.925+0000 7fc89d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:17.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.926+0000 7fc89d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a015a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45768/0 (socket says 172.21.15.6:45768) 2024-03-22T13:04:17.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.926+0000 7fc89d400640 1 -- 172.21.15.6:0/2428260363 learned_addr learned my addr 172.21.15.6:0/2428260363 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:17.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.931+0000 7fc89d400640 1 -- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8a00cacf0 msgr2=0x7fc8a01efcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:17.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.931+0000 7fc89d400640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8a00cacf0 0x7fc8a01efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:17.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.931+0000 7fc89d400640 1 -- 172.21.15.6:0/2428260363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8a015c1e0 msgr2=0x7fc8a00c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:17.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.931+0000 7fc89d400640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8a015c1e0 0x7fc8a00c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:17.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.931+0000 7fc89d400640 1 -- 172.21.15.6:0/2428260363 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a006c5f0 con 0x7fc8a0152100 2024-03-22T13:04:17.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.935+0000 7fc89d400640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a015a5d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00b4d0 tx=0x7fc88c00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:17.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.940+0000 7fc896000640 1 -- 172.21.15.6:0/2428260363 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c014070 con 0x7fc8a0152100 2024-03-22T13:04:17.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.940+0000 7fc896000640 1 -- 172.21.15.6:0/2428260363 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c010870 con 0x7fc8a0152100 2024-03-22T13:04:17.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.940+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a006c8b0 con 0x7fc8a0152100 2024-03-22T13:04:17.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.940+0000 7fc896000640 1 -- 172.21.15.6:0/2428260363 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c010c60 con 0x7fc8a0152100 2024-03-22T13:04:17.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.940+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a006ce70 con 0x7fc8a0152100 2024-03-22T13:04:17.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.941+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a0075650 con 0x7fc8a0152100 2024-03-22T13:04:17.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.977+0000 7fc896000640 1 -- 172.21.15.6:0/2428260363 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc88c017070 con 0x7fc8a0152100 2024-03-22T13:04:17.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.980+0000 7fc896000640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc88006bc50 0x7fc88006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:17.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.981+0000 7fc89de00640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc88006bc50 0x7fc88006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:17.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:17.982+0000 7fc89de00640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc88006bc50 0x7fc88006e110 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc890036000 tx=0x7fc890030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:18.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.228+0000 7fc896000640 1 -- 172.21.15.6:0/2428260363 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc88c0570c0 con 0x7fc8a0152100 2024-03-22T13:04:18.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.247+0000 7fc896000640 1 -- 172.21.15.6:0/2428260363 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc88c059720 con 0x7fc8a0152100 2024-03-22T13:04:18.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.534+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8a006d5f0 con 0x7fc8a0152100 2024-03-22T13:04:18.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.556+0000 7fc896000640 1 -- 172.21.15.6:0/2428260363 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fc88c059020 con 0x7fc8a0152100 2024-03-22T13:04:18.556 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:18.557 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:18.557 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:18.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc88006bc50 msgr2=0x7fc88006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:18.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc88006bc50 0x7fc88006e110 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc890036000 tx=0x7fc890030000 comp rx=0 tx=0).stop 2024-03-22T13:04:18.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 msgr2=0x7fc8a015a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a015a5d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00b4d0 tx=0x7fc88c00b990 comp rx=0 tx=0).stop 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 shutdown_connections 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8a015c1e0 0x7fc8a00c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc8a0152100 0x7fc8a015a5d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc88006bc50 0x7fc88006e110 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 --2- 172.21.15.6:0/2428260363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8a00cacf0 0x7fc8a01efcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.559+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 >> 172.21.15.6:0/2428260363 conn(0x7fc8a006b4a0 msgr2=0x7fc8a00c9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.560+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 shutdown_connections 2024-03-22T13:04:18.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:18.560+0000 7fc89f200640 1 -- 172.21.15.6:0/2428260363 wait complete. 2024-03-22T13:04:19.182 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:19.182 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:20.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:20.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.386+0000 7f59ea600640 1 Processor -- start 2024-03-22T13:04:20.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.386+0000 7f59ea600640 1 -- start start 2024-03-22T13:04:20.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.387+0000 7f59ea600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59e4154980 0x7f59e40ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:20.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.387+0000 7f59ea600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e415b0c0 0x7f59e40ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:20.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.387+0000 7f59ea600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e40cc380 con 0x7f59e40cd300 2024-03-22T13:04:20.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.387+0000 7f59ea600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e415a520 con 0x7f59e4154980 2024-03-22T13:04:20.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.387+0000 7f59ea600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40cc660 con 0x7f59e415b0c0 2024-03-22T13:04:20.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.388+0000 7f59e3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59e4154980 0x7f59e40ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:20.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.388+0000 7f59daa00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f59e40cd300 0x7f59e415a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45202/0 (socket says 172.21.15.6:45202) 2024-03-22T13:04:20.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.388+0000 7f59daa00640 1 -- 172.21.15.6:0/3670521901 learned_addr learned my addr 172.21.15.6:0/3670521901 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:20.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.391+0000 7f59e3400640 1 -- 172.21.15.6:0/3670521901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e415b0c0 msgr2=0x7f59e40ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:20.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.391+0000 7f59e3400640 1 --2- 172.21.15.6:0/3670521901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e415b0c0 0x7f59e40ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:20.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.391+0000 7f59e3400640 1 -- 172.21.15.6:0/3670521901 >> v1:172.21.15.92:6789/0 conn(0x7f59e40cd300 legacy=0x7f59e415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:20.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.391+0000 7f59e3400640 1 -- 172.21.15.6:0/3670521901 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40cc940 con 0x7f59e4154980 2024-03-22T13:04:20.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.394+0000 7f59e3400640 1 --2- 172.21.15.6:0/3670521901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 0x7f59e40ce300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009d90 tx=0x7f59d402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52fc0824f543adc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.398+0000 7f59e2a00640 1 -- 172.21.15.6:0/3670521901 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4039070 con 0x7f59e4154980 2024-03-22T13:04:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.398+0000 7f59e2a00640 1 -- 172.21.15.6:0/3670521901 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d402cc50 con 0x7f59e4154980 2024-03-22T13:04:20.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.398+0000 7f59e2a00640 1 -- 172.21.15.6:0/3670521901 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4034440 con 0x7f59e4154980 2024-03-22T13:04:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 -- 172.21.15.6:0/3670521901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 msgr2=0x7f59e40ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 --2- 172.21.15.6:0/3670521901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 0x7f59e40ce300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009d90 tx=0x7f59d402bea0 comp rx=0 tx=0).stop 2024-03-22T13:04:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 -- 172.21.15.6:0/3670521901 shutdown_connections 2024-03-22T13:04:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 --2- 172.21.15.6:0/3670521901 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e415b0c0 0x7f59e40ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:20.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 --2- 172.21.15.6:0/3670521901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 0x7f59e40ce300 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:20.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 -- 172.21.15.6:0/3670521901 >> 172.21.15.6:0/3670521901 conn(0x7f59e40b16c0 msgr2=0x7f59e40b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:20.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 -- 172.21.15.6:0/3670521901 shutdown_connections 2024-03-22T13:04:20.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.399+0000 7f59ea600640 1 -- 172.21.15.6:0/3670521901 wait complete. 2024-03-22T13:04:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.401+0000 7f59ea600640 1 Processor -- start 2024-03-22T13:04:20.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.401+0000 7f59ea600640 1 -- start start 2024-03-22T13:04:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.402+0000 7f59ea600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e40cd300 0x7f59e414e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.402+0000 7f59ea600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 0x7f59e41f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.402+0000 7f59ea600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59e415b0c0 0x7f59e41e5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.402+0000 7f59ea600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40ceea0 con 0x7f59e4154980 2024-03-22T13:04:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.402+0000 7f59ea600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40ccf00 con 0x7f59e415b0c0 2024-03-22T13:04:20.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.402+0000 7f59ea600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40cf020 con 0x7f59e40cd300 2024-03-22T13:04:20.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.403+0000 7f59e3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59e415b0c0 0x7f59e41e5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:20.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.403+0000 7f59daa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 0x7f59e41f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:20.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.403+0000 7f59e3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59e415b0c0 0x7f59e41e5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40740/0 (socket says 172.21.15.6:40740) 2024-03-22T13:04:20.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.403+0000 7f59e3e00640 1 -- 172.21.15.6:0/3402018428 learned_addr learned my addr 172.21.15.6:0/3402018428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:20.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.409+0000 7f59e3e00640 1 -- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e40cd300 msgr2=0x7f59e414e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:20.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.409+0000 7f59e3e00640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e40cd300 0x7f59e414e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:20.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.409+0000 7f59e3e00640 1 -- 172.21.15.6:0/3402018428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 msgr2=0x7f59e41f0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:20.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.409+0000 7f59e3e00640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 0x7f59e41f0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:20.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.409+0000 7f59e3e00640 1 -- 172.21.15.6:0/3402018428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40b27d0 con 0x7f59e415b0c0 2024-03-22T13:04:20.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.412+0000 7f59e3e00640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59e415b0c0 0x7f59e41e5960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00b410 tx=0x7f59dc00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:20.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.417+0000 7f59dbe00640 1 -- 172.21.15.6:0/3402018428 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc015020 con 0x7f59e415b0c0 2024-03-22T13:04:20.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.417+0000 7f59dbe00640 1 -- 172.21.15.6:0/3402018428 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc011850 con 0x7f59e415b0c0 2024-03-22T13:04:20.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.417+0000 7f59dbe00640 1 -- 172.21.15.6:0/3402018428 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc011c40 con 0x7f59e415b0c0 2024-03-22T13:04:20.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.417+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e40b2a90 con 0x7f59e415b0c0 2024-03-22T13:04:20.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.417+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e40b3050 con 0x7f59e415b0c0 2024-03-22T13:04:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.418+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e4153000 con 0x7f59e415b0c0 2024-03-22T13:04:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.449+0000 7f59dbe00640 1 -- 172.21.15.6:0/3402018428 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f59dc018020 con 0x7f59e415b0c0 2024-03-22T13:04:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.451+0000 7f59dbe00640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f59c006bc50 0x7f59c006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:20.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.452+0000 7f59e3400640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f59c006bc50 0x7f59c006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:20.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.452+0000 7f59e3400640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f59c006bc50 0x7f59c006e110 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:20.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.454+0000 7f59dbe00640 1 -- 172.21.15.6:0/3402018428 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f59dc011de0 con 0x7f59e415b0c0 2024-03-22T13:04:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.730+0000 7f59dbe00640 1 -- 172.21.15.6:0/3402018428 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59e4153000 con 0x7f59e415b0c0 2024-03-22T13:04:20.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.984+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e40b5770 con 0x7f59e415b0c0 2024-03-22T13:04:20.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:20.999+0000 7f59dbe00640 1 -- 172.21.15.6:0/3402018428 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f59dc01f020 con 0x7f59e415b0c0 2024-03-22T13:04:20.999 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:20.999 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:21.000 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.002+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f59c006bc50 msgr2=0x7f59c006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.002+0000 7f59ea600640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f59c006bc50 0x7f59c006e110 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d402e000 comp rx=0 tx=0).stop 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.002+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59e415b0c0 msgr2=0x7f59e41e5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.002+0000 7f59ea600640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59e415b0c0 0x7f59e41e5960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00b410 tx=0x7f59dc00b8d0 comp rx=0 tx=0).stop 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 shutdown_connections 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f59c006bc50 0x7f59c006e110 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59e415b0c0 0x7f59e41e5960 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59e4154980 0x7f59e41f0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 --2- 172.21.15.6:0/3402018428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59e40cd300 0x7f59e414e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 >> 172.21.15.6:0/3402018428 conn(0x7f59e40b16c0 msgr2=0x7f59e40c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 shutdown_connections 2024-03-22T13:04:21.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:21.003+0000 7f59ea600640 1 -- 172.21.15.6:0/3402018428 wait complete. 2024-03-22T13:04:21.624 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:21.625 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:22.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.823+0000 7fd655200640 1 Processor -- start 2024-03-22T13:04:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.823+0000 7fd655200640 1 -- start start 2024-03-22T13:04:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.824+0000 7fd655200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd65006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.825+0000 7fd655200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65006c7a0 con 0x7fd65015b0e0 2024-03-22T13:04:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.825+0000 7fd655200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65006ca80 con 0x7fd6500ccad0 2024-03-22T13:04:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.825+0000 7fd655200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6500cd000 con 0x7fd6500ca520 2024-03-22T13:04:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.826+0000 7fd64de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd65006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.826+0000 7fd64e800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd6500ccad0 0x7fd650164db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45214/0 (socket says 172.21.15.6:45214) 2024-03-22T13:04:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.826+0000 7fd64e800640 1 -- 172.21.15.6:0/2993623850 learned_addr learned my addr 172.21.15.6:0/2993623850 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.829+0000 7fd64de00640 1 -- 172.21.15.6:0/2993623850 >> v1:172.21.15.92:6789/0 conn(0x7fd6500ccad0 legacy=0x7fd650164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.829+0000 7fd64de00640 1 -- 172.21.15.6:0/2993623850 >> v1:172.21.15.6:6789/0 conn(0x7fd65015b0e0 legacy=0x7fd6500cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.829+0000 7fd64de00640 1 -- 172.21.15.6:0/2993623850 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65006cd60 con 0x7fd6500ca520 2024-03-22T13:04:22.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.831+0000 7fd64de00640 1 --2- 172.21.15.6:0/2993623850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd65006d840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd6400097d0 tx=0x7fd64002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6254973635d604b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.834+0000 7fd64ca00640 1 -- 172.21.15.6:0/2993623850 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640035020 con 0x7fd6500ca520 2024-03-22T13:04:22.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.836+0000 7fd64ca00640 1 -- 172.21.15.6:0/2993623850 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd640031890 con 0x7fd6500ca520 2024-03-22T13:04:22.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.836+0000 7fd64ca00640 1 -- 172.21.15.6:0/2993623850 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640031c60 con 0x7fd6500ca520 2024-03-22T13:04:22.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.837+0000 7fd655200640 1 -- 172.21.15.6:0/2993623850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 msgr2=0x7fd65006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:22.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.837+0000 7fd655200640 1 --2- 172.21.15.6:0/2993623850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd65006d840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd6400097d0 tx=0x7fd64002b930 comp rx=0 tx=0).stop 2024-03-22T13:04:22.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.837+0000 7fd655200640 1 -- 172.21.15.6:0/2993623850 shutdown_connections 2024-03-22T13:04:22.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.837+0000 7fd655200640 1 --2- 172.21.15.6:0/2993623850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd65006d840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:22.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.837+0000 7fd655200640 1 -- 172.21.15.6:0/2993623850 >> 172.21.15.6:0/2993623850 conn(0x7fd6500b16b0 msgr2=0x7fd6500b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:22.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.837+0000 7fd655200640 1 -- 172.21.15.6:0/2993623850 shutdown_connections 2024-03-22T13:04:22.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.837+0000 7fd655200640 1 -- 172.21.15.6:0/2993623850 wait complete. 2024-03-22T13:04:22.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.840+0000 7fd655200640 1 Processor -- start 2024-03-22T13:04:22.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.840+0000 7fd655200640 1 -- start start 2024-03-22T13:04:22.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd655200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd650158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:22.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd655200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6500ccad0 0x7fd65015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:22.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd655200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd65015b0e0 0x7fd6500c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:22.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd655200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd650154c20 con 0x7fd6500ccad0 2024-03-22T13:04:22.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd655200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65006d320 con 0x7fd65015b0e0 2024-03-22T13:04:22.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd655200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65015b980 con 0x7fd6500ca520 2024-03-22T13:04:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd64de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd650158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.842+0000 7fd64e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd65015b0e0 0x7fd6500c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.843+0000 7fd64d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6500ccad0 0x7fd65015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.843+0000 7fd64de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd650158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45810/0 (socket says 172.21.15.6:45810) 2024-03-22T13:04:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.843+0000 7fd64de00640 1 -- 172.21.15.6:0/1332085545 learned_addr learned my addr 172.21.15.6:0/1332085545 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:22.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.846+0000 7fd64de00640 1 -- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd65015b0e0 msgr2=0x7fd6500c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:22.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.846+0000 7fd64de00640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd65015b0e0 0x7fd6500c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:22.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.846+0000 7fd64de00640 1 -- 172.21.15.6:0/1332085545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6500ccad0 msgr2=0x7fd65015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:22.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.846+0000 7fd64de00640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6500ccad0 0x7fd65015a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:22.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.846+0000 7fd64de00640 1 -- 172.21.15.6:0/1332085545 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6501577c0 con 0x7fd6500ca520 2024-03-22T13:04:22.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.849+0000 7fd64de00640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd650158380 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd64002bf40 tx=0x7fd6400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.853+0000 7fd646000640 1 -- 172.21.15.6:0/1332085545 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640035040 con 0x7fd6500ca520 2024-03-22T13:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.853+0000 7fd646000640 1 -- 172.21.15.6:0/1332085545 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd640046480 con 0x7fd6500ca520 2024-03-22T13:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.853+0000 7fd646000640 1 -- 172.21.15.6:0/1332085545 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640046870 con 0x7fd6500ca520 2024-03-22T13:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.853+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd650157a60 con 0x7fd6500ca520 2024-03-22T13:04:22.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.853+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6501f3ec0 con 0x7fd6500ca520 2024-03-22T13:04:22.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.855+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6500756c0 con 0x7fd6500ca520 2024-03-22T13:04:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.877+0000 7fd646000640 1 -- 172.21.15.6:0/1332085545 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd640038020 con 0x7fd6500ca520 2024-03-22T13:04:22.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.879+0000 7fd646000640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd63006bc20 0x7fd63006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.879+0000 7fd64d400640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd63006bc20 0x7fd63006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:22.880+0000 7fd64d400640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd63006bc20 0x7fd63006e0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd650157470 tx=0x7fd63c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:23.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.199+0000 7fd646000640 1 -- 172.21.15.6:0/1332085545 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd640079ad0 con 0x7fd6500ca520 2024-03-22T13:04:23.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.211+0000 7fd646000640 1 -- 172.21.15.6:0/1332085545 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd64002d040 con 0x7fd6500ca520 2024-03-22T13:04:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.489+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6500cb080 con 0x7fd6500ca520 2024-03-22T13:04:23.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.501+0000 7fd646000640 1 -- 172.21.15.6:0/1332085545 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fd64007ba30 con 0x7fd6500ca520 2024-03-22T13:04:23.501 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:23.501 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:23.502 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:23.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd63006bc20 msgr2=0x7fd63006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:23.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd63006bc20 0x7fd63006e0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd650157470 tx=0x7fd63c009040 comp rx=0 tx=0).stop 2024-03-22T13:04:23.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 msgr2=0x7fd650158380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd650158380 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd64002bf40 tx=0x7fd6400046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 shutdown_connections 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd63006bc20 0x7fd63006e0e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd65015b0e0 0x7fd6500c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6500ccad0 0x7fd65015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 --2- 172.21.15.6:0/1332085545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd6500ca520 0x7fd650158380 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 >> 172.21.15.6:0/1332085545 conn(0x7fd6500b16b0 msgr2=0x7fd6500cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.504+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 shutdown_connections 2024-03-22T13:04:23.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:23.505+0000 7fd655200640 1 -- 172.21.15.6:0/1332085545 wait complete. 2024-03-22T13:04:24.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:24.125 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:25.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:25.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.322+0000 7f02ae000640 1 Processor -- start 2024-03-22T13:04:25.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.322+0000 7f02ae000640 1 -- start start 2024-03-22T13:04:25.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.323+0000 7f02ae000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f02a8154240 0x7f02a815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:25.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.323+0000 7f02ae000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02a80c7ee0 0x7f02a80c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:25.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.323+0000 7f02ae000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c68c0 0x7f02a806b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:25.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.323+0000 7f02ae000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a815e130 con 0x7f02a80c7ee0 2024-03-22T13:04:25.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.323+0000 7f02ae000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a8154620 con 0x7f02a8154240 2024-03-22T13:04:25.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.323+0000 7f02ae000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a80c6e00 con 0x7f02a80c68c0 2024-03-22T13:04:25.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.324+0000 7f02a7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c68c0 0x7f02a806b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:25.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.324+0000 7f02a7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f02a8154240 0x7f02a815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:25.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.324+0000 7f02a6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02a80c7ee0 0x7f02a80c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:25.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.325+0000 7f02a7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f02a8154240 0x7f02a815ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36658/0 (socket says 172.21.15.6:36658) 2024-03-22T13:04:25.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.325+0000 7f02a7400640 1 -- 172.21.15.6:0/1751414714 learned_addr learned my addr 172.21.15.6:0/1751414714 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:25.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.327+0000 7f02a7e00640 1 -- 172.21.15.6:0/1751414714 >> v2:172.21.15.92:3300/0 conn(0x7f02a8154240 msgr2=0x7f02a815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:25.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.327+0000 7f02a7e00640 1 --2- 172.21.15.6:0/1751414714 >> v2:172.21.15.92:3300/0 conn(0x7f02a8154240 0x7f02a815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.327+0000 7f02a7e00640 1 -- 172.21.15.6:0/1751414714 >> v2:172.21.15.6:3300/0 conn(0x7f02a80c7ee0 msgr2=0x7f02a80c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:25.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.327+0000 7f02a7e00640 1 --2- 172.21.15.6:0/1751414714 >> v2:172.21.15.6:3300/0 conn(0x7f02a80c7ee0 0x7f02a80c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.327+0000 7f02a7e00640 1 -- 172.21.15.6:0/1751414714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a80c70e0 con 0x7f02a80c68c0 2024-03-22T13:04:25.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.330+0000 7f02a7e00640 1 --2- 172.21.15.6:0/1751414714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c68c0 0x7f02a806b080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f02a8071390 tx=0x7f02a002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c7d6c7d2ac75cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:25.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02a6000640 1 -- 172.21.15.6:0/1751414714 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0039070 con 0x7f02a80c68c0 2024-03-22T13:04:25.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02a6000640 1 -- 172.21.15.6:0/1751414714 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a002cc60 con 0x7f02a80c68c0 2024-03-22T13:04:25.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02a6000640 1 -- 172.21.15.6:0/1751414714 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0034660 con 0x7f02a80c68c0 2024-03-22T13:04:25.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02ae000640 1 -- 172.21.15.6:0/1751414714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c68c0 msgr2=0x7f02a806b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:25.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02ae000640 1 --2- 172.21.15.6:0/1751414714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c68c0 0x7f02a806b080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f02a8071390 tx=0x7f02a002beb0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02ae000640 1 -- 172.21.15.6:0/1751414714 shutdown_connections 2024-03-22T13:04:25.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02ae000640 1 --2- 172.21.15.6:0/1751414714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c68c0 0x7f02a806b080 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02ae000640 1 --2- 172.21.15.6:0/1751414714 >> v2:172.21.15.6:3300/0 conn(0x7f02a80c7ee0 0x7f02a80c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02ae000640 1 --2- 172.21.15.6:0/1751414714 >> v2:172.21.15.92:3300/0 conn(0x7f02a8154240 0x7f02a815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.337+0000 7f02ae000640 1 -- 172.21.15.6:0/1751414714 >> 172.21.15.6:0/1751414714 conn(0x7f02a80cb930 msgr2=0x7f02a80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:25.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.338+0000 7f02ae000640 1 -- 172.21.15.6:0/1751414714 shutdown_connections 2024-03-22T13:04:25.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.338+0000 7f02ae000640 1 -- 172.21.15.6:0/1751414714 wait complete. 2024-03-22T13:04:25.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.339+0000 7f02ae000640 1 Processor -- start 2024-03-22T13:04:25.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.339+0000 7f02ae000640 1 -- start start 2024-03-22T13:04:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.340+0000 7f02ae000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02a80c68c0 0x7f02a81e3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.340+0000 7f02ae000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c7ee0 0x7f02a8071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.340+0000 7f02ae000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02a8154240 0x7f02a814edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.340+0000 7f02ae000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a815ddc0 con 0x7f02a80c68c0 2024-03-22T13:04:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.340+0000 7f02ae000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a815dae0 con 0x7f02a8154240 2024-03-22T13:04:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.340+0000 7f02ae000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a815e5c0 con 0x7f02a80c7ee0 2024-03-22T13:04:25.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.341+0000 7f02a6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c7ee0 0x7f02a8071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:25.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.341+0000 7f02a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02a8154240 0x7f02a814edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:25.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.341+0000 7f02a7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02a80c68c0 0x7f02a81e3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:25.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.341+0000 7f02a6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c7ee0 0x7f02a8071840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56926/0 (socket says 172.21.15.6:56926) 2024-03-22T13:04:25.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.341+0000 7f02a6a00640 1 -- 172.21.15.6:0/1627537767 learned_addr learned my addr 172.21.15.6:0/1627537767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:25.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.345+0000 7f02a6a00640 1 -- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02a8154240 msgr2=0x7f02a814edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:25.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.345+0000 7f02a6a00640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02a8154240 0x7f02a814edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.345+0000 7f02a6a00640 1 -- 172.21.15.6:0/1627537767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02a80c68c0 msgr2=0x7f02a81e3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:25.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.345+0000 7f02a6a00640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02a80c68c0 0x7f02a81e3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.345+0000 7f02a6a00640 1 -- 172.21.15.6:0/1627537767 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a81f3950 con 0x7f02a80c7ee0 2024-03-22T13:04:25.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.347+0000 7f02a6a00640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c7ee0 0x7f02a8071840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f029400e860 tx=0x7f029400ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:25.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.352+0000 7f029f400640 1 -- 172.21.15.6:0/1627537767 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294017070 con 0x7f02a80c7ee0 2024-03-22T13:04:25.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.352+0000 7f029f400640 1 -- 172.21.15.6:0/1627537767 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029400c940 con 0x7f02a80c7ee0 2024-03-22T13:04:25.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.352+0000 7f029f400640 1 -- 172.21.15.6:0/1627537767 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029400cd10 con 0x7f02a80c7ee0 2024-03-22T13:04:25.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.352+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a81f3c10 con 0x7f02a80c7ee0 2024-03-22T13:04:25.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.352+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a81f41d0 con 0x7f02a80c7ee0 2024-03-22T13:04:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.353+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a815b1b0 con 0x7f02a80c7ee0 2024-03-22T13:04:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.378+0000 7f029f400640 1 -- 172.21.15.6:0/1627537767 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f029401a070 con 0x7f02a80c7ee0 2024-03-22T13:04:25.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.379+0000 7f029f400640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f028806bc20 0x7f028806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.379+0000 7f02a7400640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f028806bc20 0x7f028806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:25.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.380+0000 7f02a7400640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f028806bc20 0x7f028806e0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0298009b30 tx=0x7f0298009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:25.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.624+0000 7f029f400640 1 -- 172.21.15.6:0/1627537767 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0294057720 con 0x7f02a80c7ee0 2024-03-22T13:04:25.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.640+0000 7f029f400640 1 -- 172.21.15.6:0/1627537767 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0294059680 con 0x7f02a80c7ee0 2024-03-22T13:04:25.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.920+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02a80c7ab0 con 0x7f02a80c7ee0 2024-03-22T13:04:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.935+0000 7f029f400640 1 -- 172.21.15.6:0/1627537767 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f029405d720 con 0x7f02a80c7ee0 2024-03-22T13:04:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:25.936 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:25.936 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f028806bc20 msgr2=0x7f028806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f028806bc20 0x7f028806e0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0298009b30 tx=0x7f0298009240 comp rx=0 tx=0).stop 2024-03-22T13:04:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c7ee0 msgr2=0x7f02a8071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c7ee0 0x7f02a8071840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f029400e860 tx=0x7f029400ed20 comp rx=0 tx=0).stop 2024-03-22T13:04:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 shutdown_connections 2024-03-22T13:04:25.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f028806bc20 0x7f028806e0e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02a8154240 0x7f02a814edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02a80c7ee0 0x7f02a8071840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 --2- 172.21.15.6:0/1627537767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02a80c68c0 0x7f02a81e3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:25.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 >> 172.21.15.6:0/1627537767 conn(0x7f02a80cb930 msgr2=0x7f02a806f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:25.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 shutdown_connections 2024-03-22T13:04:25.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:25.939+0000 7f02ae000640 1 -- 172.21.15.6:0/1627537767 wait complete. 2024-03-22T13:04:26.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:26.562 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:27.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:27.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.766+0000 7f3ab7c00640 1 Processor -- start 2024-03-22T13:04:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.767+0000 7f3ab7c00640 1 -- start start 2024-03-22T13:04:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.769+0000 7f3ab7c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ab0150f50 0x7f3ab00c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.769+0000 7f3ab7c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab00c3bf0 con 0x7f3ab00c5c20 2024-03-22T13:04:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.769+0000 7f3ab7c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab0071e70 con 0x7f3ab006b080 2024-03-22T13:04:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.769+0000 7f3ab7c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab00c3760 con 0x7f3ab0150f50 2024-03-22T13:04:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.770+0000 7f3ab5c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3ab006b080 0x7f3ab0161850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35322/0 (socket says 172.21.15.6:35322) 2024-03-22T13:04:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.770+0000 7f3ab5c00640 1 -- 172.21.15.6:0/1194905844 learned_addr learned my addr 172.21.15.6:0/1194905844 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.770+0000 7f3ab5200640 1 --2- 172.21.15.6:0/1194905844 >> v2:172.21.15.6:3300/0 conn(0x7f3ab0150f50 0x7f3ab00c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.773+0000 7f3aaf400640 1 -- 172.21.15.6:0/1194905844 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308420132 0 0) 0x7f3ab0071e70 con 0x7f3ab006b080 2024-03-22T13:04:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.773+0000 7f3aaf400640 1 -- 172.21.15.6:0/1194905844 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a98003520 con 0x7f3ab006b080 2024-03-22T13:04:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.775+0000 7f3ab5200640 1 -- 172.21.15.6:0/1194905844 >> v1:172.21.15.92:6789/0 conn(0x7f3ab006b080 legacy=0x7f3ab0161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.775+0000 7f3ab5200640 1 -- 172.21.15.6:0/1194905844 >> v1:172.21.15.6:6789/0 conn(0x7f3ab00c5c20 legacy=0x7f3ab0156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.775+0000 7f3ab5200640 1 -- 172.21.15.6:0/1194905844 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab0072150 con 0x7f3ab0150f50 2024-03-22T13:04:27.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.782+0000 7f3ab5200640 1 --2- 172.21.15.6:0/1194905844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab0150f50 0x7f3ab00c79e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40097a0 tx=0x7f3aa402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e015d900745f71fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:27.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.791+0000 7f3aaf400640 1 -- 172.21.15.6:0/1194905844 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa402f810 con 0x7f3ab0150f50 2024-03-22T13:04:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.796+0000 7f3aaf400640 1 -- 172.21.15.6:0/1194905844 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa402f9b0 con 0x7f3ab0150f50 2024-03-22T13:04:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.796+0000 7f3aaf400640 1 -- 172.21.15.6:0/1194905844 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa402fd60 con 0x7f3ab0150f50 2024-03-22T13:04:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.797+0000 7f3ab7c00640 1 -- 172.21.15.6:0/1194905844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab0150f50 msgr2=0x7f3ab00c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:27.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.797+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/1194905844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab0150f50 0x7f3ab00c79e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40097a0 tx=0x7f3aa402b9f0 comp rx=0 tx=0).stop 2024-03-22T13:04:27.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.797+0000 7f3ab7c00640 1 -- 172.21.15.6:0/1194905844 shutdown_connections 2024-03-22T13:04:27.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.797+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/1194905844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab0150f50 0x7f3ab00c79e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:27.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.797+0000 7f3ab7c00640 1 -- 172.21.15.6:0/1194905844 >> 172.21.15.6:0/1194905844 conn(0x7f3ab00cb930 msgr2=0x7f3ab00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:27.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.797+0000 7f3ab7c00640 1 -- 172.21.15.6:0/1194905844 shutdown_connections 2024-03-22T13:04:27.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.797+0000 7f3ab7c00640 1 -- 172.21.15.6:0/1194905844 wait complete. 2024-03-22T13:04:27.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.800+0000 7f3ab7c00640 1 Processor -- start 2024-03-22T13:04:27.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.800+0000 7f3ab7c00640 1 -- start start 2024-03-22T13:04:27.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.801+0000 7f3ab7c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ab006b080 0x7f3ab014f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:27.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.801+0000 7f3ab7c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab00c5c20 0x7f3ab0144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:27.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.801+0000 7f3ab7c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ab0150f50 0x7f3ab0144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:27.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.801+0000 7f3ab7c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab00c33f0 con 0x7f3ab00c5c20 2024-03-22T13:04:27.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.801+0000 7f3ab7c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab00c3110 con 0x7f3ab0150f50 2024-03-22T13:04:27.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.801+0000 7f3ab7c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab0151330 con 0x7f3ab006b080 2024-03-22T13:04:27.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.802+0000 7f3ab5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ab006b080 0x7f3ab014f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:27.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.802+0000 7f3ab5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ab006b080 0x7f3ab014f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56932/0 (socket says 172.21.15.6:56932) 2024-03-22T13:04:27.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.802+0000 7f3ab5200640 1 -- 172.21.15.6:0/3061192685 learned_addr learned my addr 172.21.15.6:0/3061192685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:27.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.803+0000 7f3ab5c00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ab0150f50 0x7f3ab0144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:27.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.805+0000 7f3aafe00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab00c5c20 0x7f3ab0144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.807+0000 7f3ab5200640 1 -- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ab0150f50 msgr2=0x7f3ab0144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.807+0000 7f3ab5200640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ab0150f50 0x7f3ab0144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:27.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.808+0000 7f3ab5200640 1 -- 172.21.15.6:0/3061192685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab00c5c20 msgr2=0x7f3ab0144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:27.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.808+0000 7f3ab5200640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab00c5c20 0x7f3ab0144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:27.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.808+0000 7f3ab5200640 1 -- 172.21.15.6:0/3061192685 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab014deb0 con 0x7f3ab006b080 2024-03-22T13:04:27.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.811+0000 7f3ab5200640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ab006b080 0x7f3ab014f230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3aa402ba30 tx=0x7f3aa40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:27.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.864+0000 7f3aacc00640 1 -- 172.21.15.6:0/3061192685 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4004250 con 0x7f3ab006b080 2024-03-22T13:04:27.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.864+0000 7f3aacc00640 1 -- 172.21.15.6:0/3061192685 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa40043f0 con 0x7f3ab006b080 2024-03-22T13:04:27.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.864+0000 7f3aacc00640 1 -- 172.21.15.6:0/3061192685 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4003810 con 0x7f3ab006b080 2024-03-22T13:04:27.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.864+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab014e190 con 0x7f3ab006b080 2024-03-22T13:04:27.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.864+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab014e750 con 0x7f3ab006b080 2024-03-22T13:04:27.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.866+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab00756c0 con 0x7f3ab006b080 2024-03-22T13:04:27.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.897+0000 7f3aacc00640 1 -- 172.21.15.6:0/3061192685 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3aa4048060 con 0x7f3ab006b080 2024-03-22T13:04:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.899+0000 7f3aacc00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3a9006bc20 0x7f3a9006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:27.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.899+0000 7f3aafe00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3a9006bc20 0x7f3a9006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:27.900+0000 7f3aafe00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3a9006bc20 0x7f3a9006e0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0154cd0 tx=0x7f3a9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.153+0000 7f3aacc00640 1 -- 172.21.15.6:0/3061192685 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3aa4078500 con 0x7f3ab006b080 2024-03-22T13:04:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.172+0000 7f3aacc00640 1 -- 172.21.15.6:0/3061192685 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3aa4033030 con 0x7f3ab006b080 2024-03-22T13:04:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.426+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ab00cbe80 con 0x7f3ab006b080 2024-03-22T13:04:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.441+0000 7f3aacc00640 1 -- 172.21.15.6:0/3061192685 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f3aa407ab60 con 0x7f3ab006b080 2024-03-22T13:04:28.441 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:28.441 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3a9006bc20 msgr2=0x7f3a9006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3a9006bc20 0x7f3a9006e0e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0154cd0 tx=0x7f3a9c009040 comp rx=0 tx=0).stop 2024-03-22T13:04:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ab006b080 msgr2=0x7f3ab014f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ab006b080 0x7f3ab014f230 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3aa402ba30 tx=0x7f3aa40046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 shutdown_connections 2024-03-22T13:04:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3a9006bc20 0x7f3a9006e0e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ab0150f50 0x7f3ab0144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ab00c5c20 0x7f3ab0144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 --2- 172.21.15.6:0/3061192685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ab006b080 0x7f3ab014f230 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 >> 172.21.15.6:0/3061192685 conn(0x7f3ab00cb930 msgr2=0x7f3ab00744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 shutdown_connections 2024-03-22T13:04:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:28.444+0000 7f3ab7c00640 1 -- 172.21.15.6:0/3061192685 wait complete. 2024-03-22T13:04:29.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:29.065 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:30.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.276+0000 7fa59d400640 1 Processor -- start 2024-03-22T13:04:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.277+0000 7fa59d400640 1 -- start start 2024-03-22T13:04:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.278+0000 7fa59d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa598154580 0x7fa59806b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.278+0000 7fa59d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa59806bbc0 0x7fa59815a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.278+0000 7fa59d400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5980c90e0 con 0x7fa5981541a0 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.278+0000 7fa59d400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5980c79b0 con 0x7fa598154580 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.278+0000 7fa59d400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59815b030 con 0x7fa59806bbc0 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.278+0000 7fa597e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa5981541a0 0x7fa5980c72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35328/0 (socket says 172.21.15.6:35328) 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.279+0000 7fa597e00640 1 -- 172.21.15.6:0/2429596500 learned_addr learned my addr 172.21.15.6:0/2429596500 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.279+0000 7fa597400640 1 --2- 172.21.15.6:0/2429596500 >> v2:172.21.15.6:3300/0 conn(0x7fa598154580 0x7fa59806b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.279+0000 7fa59ca00640 1 --2- 172.21.15.6:0/2429596500 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa59806bbc0 0x7fa59815a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.281+0000 7fa596a00640 1 -- 172.21.15.6:0/2429596500 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116977346 0 0) 0x7fa5980c90e0 con 0x7fa5981541a0 2024-03-22T13:04:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.282+0000 7fa596a00640 1 -- 172.21.15.6:0/2429596500 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa57c003520 con 0x7fa5981541a0 2024-03-22T13:04:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.283+0000 7fa597400640 1 -- 172.21.15.6:0/2429596500 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa59806bbc0 msgr2=0x7fa59815a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.283+0000 7fa597400640 1 --2- 172.21.15.6:0/2429596500 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa59806bbc0 0x7fa59815a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.283+0000 7fa597400640 1 -- 172.21.15.6:0/2429596500 >> v1:172.21.15.92:6789/0 conn(0x7fa5981541a0 legacy=0x7fa5980c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.283+0000 7fa597400640 1 -- 172.21.15.6:0/2429596500 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5980c9350 con 0x7fa598154580 2024-03-22T13:04:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.286+0000 7fa597400640 1 --2- 172.21.15.6:0/2429596500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 0x7fa59806b680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa5840097a0 tx=0x7fa58402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843e5ffba527c464 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.291+0000 7fa596a00640 1 -- 172.21.15.6:0/2429596500 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584035020 con 0x7fa598154580 2024-03-22T13:04:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.291+0000 7fa596a00640 1 -- 172.21.15.6:0/2429596500 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa584031830 con 0x7fa598154580 2024-03-22T13:04:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.292+0000 7fa596a00640 1 -- 172.21.15.6:0/2429596500 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584031c00 con 0x7fa598154580 2024-03-22T13:04:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.292+0000 7fa59d400640 1 -- 172.21.15.6:0/2429596500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 msgr2=0x7fa59806b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.292+0000 7fa59d400640 1 --2- 172.21.15.6:0/2429596500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 0x7fa59806b680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa5840097a0 tx=0x7fa58402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.293+0000 7fa59d400640 1 -- 172.21.15.6:0/2429596500 shutdown_connections 2024-03-22T13:04:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.293+0000 7fa59d400640 1 --2- 172.21.15.6:0/2429596500 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa59806bbc0 0x7fa59815a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.293+0000 7fa59d400640 1 --2- 172.21.15.6:0/2429596500 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 0x7fa59806b680 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.293+0000 7fa59d400640 1 -- 172.21.15.6:0/2429596500 >> 172.21.15.6:0/2429596500 conn(0x7fa598070140 msgr2=0x7fa598070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.293+0000 7fa59d400640 1 -- 172.21.15.6:0/2429596500 shutdown_connections 2024-03-22T13:04:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.293+0000 7fa59d400640 1 -- 172.21.15.6:0/2429596500 wait complete. 2024-03-22T13:04:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.295+0000 7fa59d400640 1 Processor -- start 2024-03-22T13:04:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.295+0000 7fa59d400640 1 -- start start 2024-03-22T13:04:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.296+0000 7fa59d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa59806bbc0 0x7fa5980cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.296+0000 7fa59d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5981541a0 0x7fa59815c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.296+0000 7fa59d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 0x7fa59815c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.296+0000 7fa59d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5980c9bf0 con 0x7fa598154580 2024-03-22T13:04:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.296+0000 7fa59d400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5980c9910 con 0x7fa59806bbc0 2024-03-22T13:04:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.296+0000 7fa59d400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5980ca110 con 0x7fa5981541a0 2024-03-22T13:04:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.297+0000 7fa597e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa59806bbc0 0x7fa5980cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.297+0000 7fa597400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5981541a0 0x7fa59815c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.297+0000 7fa59ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 0x7fa59815c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.297+0000 7fa597e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa59806bbc0 0x7fa5980cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36686/0 (socket says 172.21.15.6:36686) 2024-03-22T13:04:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.297+0000 7fa597e00640 1 -- 172.21.15.6:0/3977881058 learned_addr learned my addr 172.21.15.6:0/3977881058 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.302+0000 7fa597e00640 1 -- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5981541a0 msgr2=0x7fa59815c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.302+0000 7fa597e00640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5981541a0 0x7fa59815c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.302+0000 7fa597e00640 1 -- 172.21.15.6:0/3977881058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 msgr2=0x7fa59815c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.302+0000 7fa597e00640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 0x7fa59815c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.302+0000 7fa597e00640 1 -- 172.21.15.6:0/3977881058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5981595c0 con 0x7fa59806bbc0 2024-03-22T13:04:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.305+0000 7fa597e00640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa59806bbc0 0x7fa5980cb760 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa58800c460 tx=0x7fa58800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:30.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.311+0000 7fa58fe00640 1 -- 172.21.15.6:0/3977881058 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588017070 con 0x7fa59806bbc0 2024-03-22T13:04:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.311+0000 7fa58fe00640 1 -- 172.21.15.6:0/3977881058 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa588005ce0 con 0x7fa59806bbc0 2024-03-22T13:04:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.311+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5981598a0 con 0x7fa59806bbc0 2024-03-22T13:04:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.311+0000 7fa58fe00640 1 -- 172.21.15.6:0/3977881058 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588007290 con 0x7fa59806bbc0 2024-03-22T13:04:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.311+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa598157f50 con 0x7fa59806bbc0 2024-03-22T13:04:30.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.313+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59815ba30 con 0x7fa59806bbc0 2024-03-22T13:04:30.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.601+0000 7fa58fe00640 1 -- 172.21.15.6:0/3977881058 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa58801a070 con 0x7fa59806bbc0 2024-03-22T13:04:30.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.603+0000 7fa58fe00640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa57806bc20 0x7fa57806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:30.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.604+0000 7fa597400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa57806bc20 0x7fa57806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.605+0000 7fa597400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa57806bc20 0x7fa57806e0e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa584036000 tx=0x7fa58403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:30.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.606+0000 7fa58fe00640 1 -- 172.21.15.6:0/3977881058 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa588056590 con 0x7fa59806bbc0 2024-03-22T13:04:30.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.623+0000 7fa58fe00640 1 -- 172.21.15.6:0/3977881058 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa588021080 con 0x7fa59806bbc0 2024-03-22T13:04:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.893+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa598075650 con 0x7fa59806bbc0 2024-03-22T13:04:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.914+0000 7fa58fe00640 1 -- 172.21.15.6:0/3977881058 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fa5880584f0 con 0x7fa59806bbc0 2024-03-22T13:04:30.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:30.915 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:30.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa57806bc20 msgr2=0x7fa57806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:30.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa57806bc20 0x7fa57806e0e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa584036000 tx=0x7fa58403c040 comp rx=0 tx=0).stop 2024-03-22T13:04:30.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa59806bbc0 msgr2=0x7fa5980cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:30.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa59806bbc0 0x7fa5980cb760 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa58800c460 tx=0x7fa58800c920 comp rx=0 tx=0).stop 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 shutdown_connections 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa598154580 0x7fa59815c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5981541a0 0x7fa59815c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa57806bc20 0x7fa57806e0e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 --2- 172.21.15.6:0/3977881058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa59806bbc0 0x7fa5980cb760 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 >> 172.21.15.6:0/3977881058 conn(0x7fa598070140 msgr2=0x7fa5980cab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 shutdown_connections 2024-03-22T13:04:30.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:30.917+0000 7fa59d400640 1 -- 172.21.15.6:0/3977881058 wait complete. 2024-03-22T13:04:31.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:04:31.380 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:04:31.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:31.538 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:32.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:32.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.732+0000 7fac94000640 1 Processor -- start 2024-03-22T13:04:32.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.733+0000 7fac94000640 1 -- start start 2024-03-22T13:04:32.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.734+0000 7fac94000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0ca590 0x7fac8c0ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:32.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.734+0000 7fac94000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac8c0cdfd0 0x7fac8c0ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:32.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.734+0000 7fac94000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8c0ce9a0 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.734+0000 7fac94000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8c0cdb40 con 0x7fac8c0cdfd0 2024-03-22T13:04:32.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.734+0000 7fac94000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c15a790 con 0x7fac8c0ca590 2024-03-22T13:04:32.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.735+0000 7fac92000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac8c0cdfd0 0x7fac8c0ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:32.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.735+0000 7fac91600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0ca590 0x7fac8c0ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:32.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.735+0000 7fac90c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac8c0cc5f0 0x7fac8c0cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40140/0 (socket says 172.21.15.6:40140) 2024-03-22T13:04:32.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.735+0000 7fac90c00640 1 -- 172.21.15.6:0/1376524821 learned_addr learned my addr 172.21.15.6:0/1376524821 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:32.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.738+0000 7fac92000640 1 -- 172.21.15.6:0/1376524821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0ca590 msgr2=0x7fac8c0ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:32.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.738+0000 7fac8be00640 1 -- 172.21.15.6:0/1376524821 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881933714 0 0) 0x7fac8c0ce9a0 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.738+0000 7fac92000640 1 --2- 172.21.15.6:0/1376524821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0ca590 0x7fac8c0ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:32.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.738+0000 7fac92000640 1 -- 172.21.15.6:0/1376524821 >> v1:172.21.15.6:6789/0 conn(0x7fac8c0cc5f0 legacy=0x7fac8c0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:32.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.738+0000 7fac92000640 1 -- 172.21.15.6:0/1376524821 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c15aa70 con 0x7fac8c0cdfd0 2024-03-22T13:04:32.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.741+0000 7fac92000640 1 --2- 172.21.15.6:0/1376524821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0cdfd0 0x7fac8c0ce460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fac84009d90 tx=0x7fac8402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ed2c626a48a6df server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:32.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.746+0000 7fac8be00640 1 -- 172.21.15.6:0/1376524821 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84039070 con 0x7fac8c0cdfd0 2024-03-22T13:04:32.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.746+0000 7fac8be00640 1 -- 172.21.15.6:0/1376524821 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8402cce0 con 0x7fac8c0cdfd0 2024-03-22T13:04:32.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.746+0000 7fac8be00640 1 -- 172.21.15.6:0/1376524821 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84034660 con 0x7fac8c0cdfd0 2024-03-22T13:04:32.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 -- 172.21.15.6:0/1376524821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0cdfd0 msgr2=0x7fac8c0ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:32.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 --2- 172.21.15.6:0/1376524821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0cdfd0 0x7fac8c0ce460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fac84009d90 tx=0x7fac8402bf30 comp rx=0 tx=0).stop 2024-03-22T13:04:32.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 -- 172.21.15.6:0/1376524821 shutdown_connections 2024-03-22T13:04:32.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 --2- 172.21.15.6:0/1376524821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0cdfd0 0x7fac8c0ce460 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:32.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 --2- 172.21.15.6:0/1376524821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0ca590 0x7fac8c0ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:32.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 -- 172.21.15.6:0/1376524821 >> 172.21.15.6:0/1376524821 conn(0x7fac8c0b1720 msgr2=0x7fac8c0b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:32.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 -- 172.21.15.6:0/1376524821 shutdown_connections 2024-03-22T13:04:32.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.747+0000 7fac94000640 1 -- 172.21.15.6:0/1376524821 wait complete. 2024-03-22T13:04:32.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.750+0000 7fac94000640 1 Processor -- start 2024-03-22T13:04:32.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.750+0000 7fac94000640 1 -- start start 2024-03-22T13:04:32.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.751+0000 7fac94000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0ca590 0x7fac8c159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:32.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.751+0000 7fac94000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac8c0cc5f0 0x7fac8c1e47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:32.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.751+0000 7fac94000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0cdfd0 0x7fac8c1e4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:32.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.751+0000 7fac94000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c15b310 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.751+0000 7fac94000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c15b030 con 0x7fac8c0ca590 2024-03-22T13:04:32.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.751+0000 7fac94000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c15a480 con 0x7fac8c0cdfd0 2024-03-22T13:04:32.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.752+0000 7fac91600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0ca590 0x7fac8c159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:32.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.752+0000 7fac91600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0ca590 0x7fac8c159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36694/0 (socket says 172.21.15.6:36694) 2024-03-22T13:04:32.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.752+0000 7fac91600640 1 -- 172.21.15.6:0/2320823255 learned_addr learned my addr 172.21.15.6:0/2320823255 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:32.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.752+0000 7fac92000640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0cdfd0 0x7fac8c1e4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:32.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.752+0000 7fac90c00640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac8c0cc5f0 0x7fac8c1e47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:32.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.757+0000 7fac90c00640 1 -- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0cdfd0 msgr2=0x7fac8c1e4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:32.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.757+0000 7fac90c00640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0cdfd0 0x7fac8c1e4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:32.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.757+0000 7fac90c00640 1 -- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0ca590 msgr2=0x7fac8c159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:32.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.757+0000 7fac90c00640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0ca590 0x7fac8c159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:32.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.757+0000 7fac90c00640 1 -- 172.21.15.6:0/2320823255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c1f4780 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.760+0000 7fac90c00640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac8c0cc5f0 0x7fac8c1e47e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fac7800c410 tx=0x7fac7800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.767+0000 7fac89600640 1 -- 172.21.15.6:0/2320823255 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac780074e0 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.767+0000 7fac89600640 1 -- 172.21.15.6:0/2320823255 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac780047e0 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.767+0000 7fac89600640 1 -- 172.21.15.6:0/2320823255 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78004bd0 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.767+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8c1f04b0 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.767+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8c1f0a30 con 0x7fac8c0cc5f0 2024-03-22T13:04:32.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:32.768+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8c0756c0 con 0x7fac8c0cc5f0 2024-03-22T13:04:33.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.058+0000 7fac89600640 1 -- 172.21.15.6:0/2320823255 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac78011070 con 0x7fac8c0cc5f0 2024-03-22T13:04:33.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.061+0000 7fac89600640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac6c06bc20 0x7fac6c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:33.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.061+0000 7fac91600640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac6c06bc20 0x7fac6c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:33.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.062+0000 7fac91600640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac6c06bc20 0x7fac6c06e0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fac80009c00 tx=0x7fac80009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:33.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.089+0000 7fac89600640 1 -- 172.21.15.6:0/2320823255 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fac78057120 con 0x7fac8c0cc5f0 2024-03-22T13:04:33.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.089+0000 7fac89600640 1 -- 172.21.15.6:0/2320823255 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac78059080 con 0x7fac8c0cc5f0 2024-03-22T13:04:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.361+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac8c0c9bc0 con 0x7fac8c0cc5f0 2024-03-22T13:04:33.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.384+0000 7fac89600640 1 -- 172.21.15.6:0/2320823255 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fac7805d120 con 0x7fac8c0cc5f0 2024-03-22T13:04:33.384 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:33.385 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:33.385 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac6c06bc20 msgr2=0x7fac6c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac6c06bc20 0x7fac6c06e0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fac80009c00 tx=0x7fac80009190 comp rx=0 tx=0).stop 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac8c0cc5f0 msgr2=0x7fac8c1e47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac8c0cc5f0 0x7fac8c1e47e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fac7800c410 tx=0x7fac7800c8d0 comp rx=0 tx=0).stop 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 shutdown_connections 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac8c0cdfd0 0x7fac8c1e4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac8c0cc5f0 0x7fac8c1e47e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fac6c06bc20 0x7fac6c06e0e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.387+0000 7fac94000640 1 --2- 172.21.15.6:0/2320823255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac8c0ca590 0x7fac8c159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.388+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 >> 172.21.15.6:0/2320823255 conn(0x7fac8c0b1720 msgr2=0x7fac8c0c7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.388+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 shutdown_connections 2024-03-22T13:04:33.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:33.388+0000 7fac94000640 1 -- 172.21.15.6:0/2320823255 wait complete. 2024-03-22T13:04:34.009 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:34.009 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:35.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:35.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.202+0000 7f2d54a00640 1 Processor -- start 2024-03-22T13:04:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.202+0000 7f2d54a00640 1 -- start start 2024-03-22T13:04:35.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.204+0000 7f2d54a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2d500ca520 0x7f2d5006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:35.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.204+0000 7f2d54a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5006c7a0 con 0x7f2d500ccad0 2024-03-22T13:04:35.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.204+0000 7f2d54a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5006ca80 con 0x7f2d5015b0e0 2024-03-22T13:04:35.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.204+0000 7f2d54a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d500cd000 con 0x7f2d500ca520 2024-03-22T13:04:35.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.205+0000 7f2d4d400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2d5015b0e0 0x7f2d500cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:55842/0 (socket says 172.21.15.6:55842) 2024-03-22T13:04:35.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.205+0000 7f2d4d400640 1 -- 172.21.15.6:0/2959367455 learned_addr learned my addr 172.21.15.6:0/2959367455 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:35.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.205+0000 7f2d4de00640 1 --2- 172.21.15.6:0/2959367455 >> v2:172.21.15.6:3300/0 conn(0x7f2d500ca520 0x7f2d5006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:35.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.208+0000 7f2d4ca00640 1 -- 172.21.15.6:0/2959367455 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511249995 0 0) 0x7f2d5006ca80 con 0x7f2d5015b0e0 2024-03-22T13:04:35.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.208+0000 7f2d4ca00640 1 -- 172.21.15.6:0/2959367455 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d34003520 con 0x7f2d5015b0e0 2024-03-22T13:04:35.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.209+0000 7f2d4de00640 1 -- 172.21.15.6:0/2959367455 >> v1:172.21.15.92:6789/0 conn(0x7f2d5015b0e0 legacy=0x7f2d500cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:35.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.209+0000 7f2d4de00640 1 -- 172.21.15.6:0/2959367455 >> v1:172.21.15.6:6789/0 conn(0x7f2d500ccad0 legacy=0x7f2d50164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:35.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.209+0000 7f2d4de00640 1 -- 172.21.15.6:0/2959367455 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5006cd60 con 0x7f2d500ca520 2024-03-22T13:04:35.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.215+0000 7f2d4de00640 1 --2- 172.21.15.6:0/2959367455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d500ca520 0x7f2d5006d840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2d400097a0 tx=0x7f2d4002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f816f575f64226b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:35.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.221+0000 7f2d4ca00640 1 -- 172.21.15.6:0/2959367455 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d40034070 con 0x7f2d500ca520 2024-03-22T13:04:35.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.221+0000 7f2d4ca00640 1 -- 172.21.15.6:0/2959367455 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d40030610 con 0x7f2d500ca520 2024-03-22T13:04:35.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.221+0000 7f2d4ca00640 1 -- 172.21.15.6:0/2959367455 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d400309e0 con 0x7f2d500ca520 2024-03-22T13:04:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.222+0000 7f2d54a00640 1 -- 172.21.15.6:0/2959367455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d500ca520 msgr2=0x7f2d5006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.222+0000 7f2d54a00640 1 --2- 172.21.15.6:0/2959367455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d500ca520 0x7f2d5006d840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2d400097a0 tx=0x7f2d4002b960 comp rx=0 tx=0).stop 2024-03-22T13:04:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.222+0000 7f2d54a00640 1 -- 172.21.15.6:0/2959367455 shutdown_connections 2024-03-22T13:04:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.222+0000 7f2d54a00640 1 --2- 172.21.15.6:0/2959367455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d500ca520 0x7f2d5006d840 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.222+0000 7f2d54a00640 1 -- 172.21.15.6:0/2959367455 >> 172.21.15.6:0/2959367455 conn(0x7f2d500b16b0 msgr2=0x7f2d500b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.222+0000 7f2d54a00640 1 -- 172.21.15.6:0/2959367455 shutdown_connections 2024-03-22T13:04:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.222+0000 7f2d54a00640 1 -- 172.21.15.6:0/2959367455 wait complete. 2024-03-22T13:04:35.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.224+0000 7f2d54a00640 1 Processor -- start 2024-03-22T13:04:35.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.224+0000 7f2d54a00640 1 -- start start 2024-03-22T13:04:35.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.225+0000 7f2d54a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d500ca520 0x7f2d5015db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:35.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.225+0000 7f2d54a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d500ccad0 0x7f2d501574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:35.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.225+0000 7f2d54a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5015b0e0 0x7f2d501579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:35.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.225+0000 7f2d54a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d50154c20 con 0x7f2d5015b0e0 2024-03-22T13:04:35.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.225+0000 7f2d54a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5006d320 con 0x7f2d500ccad0 2024-03-22T13:04:35.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.225+0000 7f2d54a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5015b980 con 0x7f2d500ca520 2024-03-22T13:04:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.226+0000 7f2d4d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d500ccad0 0x7f2d501574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.226+0000 7f2d4e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5015b0e0 0x7f2d501579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.226+0000 7f2d4de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d500ca520 0x7f2d5015db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.226+0000 7f2d4d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d500ccad0 0x7f2d501574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37382/0 (socket says 172.21.15.6:37382) 2024-03-22T13:04:35.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.226+0000 7f2d4d400640 1 -- 172.21.15.6:0/84896988 learned_addr learned my addr 172.21.15.6:0/84896988 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.230+0000 7f2d4d400640 1 -- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d500ca520 msgr2=0x7f2d5015db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.230+0000 7f2d4d400640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d500ca520 0x7f2d5015db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.230+0000 7f2d4d400640 1 -- 172.21.15.6:0/84896988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5015b0e0 msgr2=0x7f2d501579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.230+0000 7f2d4d400640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5015b0e0 0x7f2d501579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.230+0000 7f2d4d400640 1 -- 172.21.15.6:0/84896988 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d501e5a30 con 0x7f2d500ccad0 2024-03-22T13:04:35.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.234+0000 7f2d4d400640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d500ccad0 0x7f2d501574a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00c3f0 tx=0x7f2d3c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.238+0000 7f2d46000640 1 -- 172.21.15.6:0/84896988 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c017070 con 0x7f2d500ccad0 2024-03-22T13:04:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.238+0000 7f2d46000640 1 -- 172.21.15.6:0/84896988 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3c005ce0 con 0x7f2d500ccad0 2024-03-22T13:04:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.238+0000 7f2d46000640 1 -- 172.21.15.6:0/84896988 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0072f0 con 0x7f2d500ccad0 2024-03-22T13:04:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.238+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d501e5cd0 con 0x7f2d500ccad0 2024-03-22T13:04:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.239+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d501e6290 con 0x7f2d500ccad0 2024-03-22T13:04:35.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.240+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d50158170 con 0x7f2d500ccad0 2024-03-22T13:04:35.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.504+0000 7f2d46000640 1 -- 172.21.15.6:0/84896988 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2d3c01a070 con 0x7f2d500ccad0 2024-03-22T13:04:35.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.506+0000 7f2d46000640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2d3006bc50 0x7f2d3006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:35.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.507+0000 7f2d46000640 1 -- 172.21.15.6:0/84896988 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f2d3c08a530 con 0x7f2d500ccad0 2024-03-22T13:04:35.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.507+0000 7f2d4de00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2d3006bc50 0x7f2d3006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:35.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.508+0000 7f2d4de00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2d3006bc50 0x7f2d3006e110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d40036000 tx=0x7f2d4003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:35.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.516+0000 7f2d46000640 1 -- 172.21.15.6:0/84896988 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2d50158170 con 0x7f2d500ccad0 2024-03-22T13:04:35.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.766+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d500cb080 con 0x7f2d500ccad0 2024-03-22T13:04:35.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.780+0000 7f2d46000640 1 -- 172.21.15.6:0/84896988 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f2d3c021020 con 0x7f2d500ccad0 2024-03-22T13:04:35.780 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:35.781 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:35.781 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2d3006bc50 msgr2=0x7f2d3006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2d3006bc50 0x7f2d3006e110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d40036000 tx=0x7f2d4003b040 comp rx=0 tx=0).stop 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d500ccad0 msgr2=0x7f2d501574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d500ccad0 0x7f2d501574a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00c3f0 tx=0x7f2d3c00c8b0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 shutdown_connections 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5015b0e0 0x7f2d501579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2d3006bc50 0x7f2d3006e110 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d500ccad0 0x7f2d501574a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 --2- 172.21.15.6:0/84896988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d500ca520 0x7f2d5015db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 >> 172.21.15.6:0/84896988 conn(0x7f2d500b16b0 msgr2=0x7f2d500b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 shutdown_connections 2024-03-22T13:04:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:35.783+0000 7f2d54a00640 1 -- 172.21.15.6:0/84896988 wait complete. 2024-03-22T13:04:36.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:36.406 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:37.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:37.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.600+0000 7f9465800640 1 Processor -- start 2024-03-22T13:04:37.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.600+0000 7f9465800640 1 -- start start 2024-03-22T13:04:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.601+0000 7f9465800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f94600ca510 0x7f94600ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.602+0000 7f9465800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946015a6b0 con 0x7f946006c9c0 2024-03-22T13:04:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.602+0000 7f9465800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946015b4d0 con 0x7f94600ccd80 2024-03-22T13:04:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.602+0000 7f9465800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9460154d60 con 0x7f94600ca510 2024-03-22T13:04:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.603+0000 7f945ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f94600ccd80 0x7f94600cf080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:55852/0 (socket says 172.21.15.6:55852) 2024-03-22T13:04:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.603+0000 7f945ea00640 1 -- 172.21.15.6:0/47819297 learned_addr learned my addr 172.21.15.6:0/47819297 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:37.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.603+0000 7f945e000640 1 --2- 172.21.15.6:0/47819297 >> v2:172.21.15.92:3300/0 conn(0x7f94600ca510 0x7f94600ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.606+0000 7f945d600640 1 -- 172.21.15.6:0/47819297 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810848531 0 0) 0x7f946015a6b0 con 0x7f946006c9c0 2024-03-22T13:04:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.606+0000 7f945d600640 1 -- 172.21.15.6:0/47819297 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9444003520 con 0x7f946006c9c0 2024-03-22T13:04:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.608+0000 7f945e000640 1 -- 172.21.15.6:0/47819297 >> v1:172.21.15.92:6789/0 conn(0x7f94600ccd80 legacy=0x7f94600cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.608+0000 7f945e000640 1 -- 172.21.15.6:0/47819297 >> v1:172.21.15.6:6789/0 conn(0x7f946006c9c0 legacy=0x7f946006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.608+0000 7f945e000640 1 -- 172.21.15.6:0/47819297 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946015a950 con 0x7f94600ca510 2024-03-22T13:04:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.612+0000 7f945e000640 1 --2- 172.21.15.6:0/47819297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600ca950 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f944c0097a0 tx=0x7f944c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f0123a8cb95945f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.618+0000 7f945d600640 1 -- 172.21.15.6:0/47819297 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c034070 con 0x7f94600ca510 2024-03-22T13:04:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.618+0000 7f945d600640 1 -- 172.21.15.6:0/47819297 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c030610 con 0x7f94600ca510 2024-03-22T13:04:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.618+0000 7f945d600640 1 -- 172.21.15.6:0/47819297 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c0309e0 con 0x7f94600ca510 2024-03-22T13:04:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.619+0000 7f9465800640 1 -- 172.21.15.6:0/47819297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 msgr2=0x7f94600ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.619+0000 7f9465800640 1 --2- 172.21.15.6:0/47819297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600ca950 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f944c0097a0 tx=0x7f944c02b960 comp rx=0 tx=0).stop 2024-03-22T13:04:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.619+0000 7f9465800640 1 -- 172.21.15.6:0/47819297 shutdown_connections 2024-03-22T13:04:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.619+0000 7f9465800640 1 --2- 172.21.15.6:0/47819297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600ca950 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.619+0000 7f9465800640 1 -- 172.21.15.6:0/47819297 >> 172.21.15.6:0/47819297 conn(0x7f94600b16c0 msgr2=0x7f94600b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.619+0000 7f9465800640 1 -- 172.21.15.6:0/47819297 shutdown_connections 2024-03-22T13:04:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.619+0000 7f9465800640 1 -- 172.21.15.6:0/47819297 wait complete. 2024-03-22T13:04:37.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.621+0000 7f9465800640 1 Processor -- start 2024-03-22T13:04:37.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.622+0000 7f9465800640 1 -- start start 2024-03-22T13:04:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.623+0000 7f9465800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f946006c9c0 0x7f9460157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.623+0000 7f9465800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.623+0000 7f9465800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94600ccd80 0x7f94600c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.623+0000 7f9465800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946015b1f0 con 0x7f946006c9c0 2024-03-22T13:04:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.623+0000 7f9465800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946015af10 con 0x7f94600ca510 2024-03-22T13:04:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.623+0000 7f9465800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946006db10 con 0x7f94600ccd80 2024-03-22T13:04:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.624+0000 7f945e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.624+0000 7f945ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f946006c9c0 0x7f9460157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.624+0000 7f945e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37390/0 (socket says 172.21.15.6:37390) 2024-03-22T13:04:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.624+0000 7f945e000640 1 -- 172.21.15.6:0/3829377924 learned_addr learned my addr 172.21.15.6:0/3829377924 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.624+0000 7f945f400640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94600ccd80 0x7f94600c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.628+0000 7f945e000640 1 -- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94600ccd80 msgr2=0x7f94600c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.628+0000 7f945e000640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94600ccd80 0x7f94600c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.628+0000 7f945e000640 1 -- 172.21.15.6:0/3829377924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f946006c9c0 msgr2=0x7f9460157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.628+0000 7f945e000640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f946006c9c0 0x7f9460157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.628+0000 7f945e000640 1 -- 172.21.15.6:0/3829377924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94600c60b0 con 0x7f94600ca510 2024-03-22T13:04:37.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.630+0000 7f945e000640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600c3f00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f944c0098c0 tx=0x7f944c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:37.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.635+0000 7f9456a00640 1 -- 172.21.15.6:0/3829377924 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c034040 con 0x7f94600ca510 2024-03-22T13:04:37.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.635+0000 7f9456a00640 1 -- 172.21.15.6:0/3829377924 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c030410 con 0x7f94600ca510 2024-03-22T13:04:37.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.635+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94600c6390 con 0x7f94600ca510 2024-03-22T13:04:37.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.635+0000 7f9456a00640 1 -- 172.21.15.6:0/3829377924 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c039c50 con 0x7f94600ca510 2024-03-22T13:04:37.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.635+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9460153c00 con 0x7f94600ca510 2024-03-22T13:04:37.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.636+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94600756c0 con 0x7f94600ca510 2024-03-22T13:04:37.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.661+0000 7f9456a00640 1 -- 172.21.15.6:0/3829377924 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f944c037070 con 0x7f94600ca510 2024-03-22T13:04:37.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.663+0000 7f9456a00640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f944006bc20 0x7f944006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:37.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.663+0000 7f945ea00640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f944006bc20 0x7f944006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:37.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.664+0000 7f945ea00640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f944006bc20 0x7f944006e0e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9450005880 tx=0x7f9450009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:37.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.664+0000 7f9456a00640 1 -- 172.21.15.6:0/3829377924 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f944c0305c0 con 0x7f94600ca510 2024-03-22T13:04:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:37.946+0000 7f9456a00640 1 -- 172.21.15.6:0/3829377924 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f944c07b970 con 0x7f94600ca510 2024-03-22T13:04:38.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.198+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94600c6560 con 0x7f94600ca510 2024-03-22T13:04:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.222+0000 7f9456a00640 1 -- 172.21.15.6:0/3829377924 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f944c07fa10 con 0x7f94600ca510 2024-03-22T13:04:38.222 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:38.222 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f944006bc20 msgr2=0x7f944006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f944006bc20 0x7f944006e0e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9450005880 tx=0x7f9450009040 comp rx=0 tx=0).stop 2024-03-22T13:04:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 msgr2=0x7f94600c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600c3f00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f944c0098c0 tx=0x7f944c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 shutdown_connections 2024-03-22T13:04:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94600ccd80 0x7f94600c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f944006bc20 0x7f944006e0e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94600ca510 0x7f94600c3f00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 --2- 172.21.15.6:0/3829377924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f946006c9c0 0x7f9460157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 >> 172.21.15.6:0/3829377924 conn(0x7f94600b16c0 msgr2=0x7f94600ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 shutdown_connections 2024-03-22T13:04:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:38.224+0000 7f9465800640 1 -- 172.21.15.6:0/3829377924 wait complete. 2024-03-22T13:04:38.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:38.846 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:39.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.044+0000 7f5f2c800640 1 Processor -- start 2024-03-22T13:04:40.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.044+0000 7f5f2c800640 1 -- start start 2024-03-22T13:04:40.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.045+0000 7f5f2c800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f240ccad0 0x7f5f2406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:40.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.046+0000 7f5f2c800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2406c7a0 con 0x7f5f240ca520 2024-03-22T13:04:40.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.046+0000 7f5f2c800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2406ca80 con 0x7f5f2415b0e0 2024-03-22T13:04:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.046+0000 7f5f2c800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f240cd000 con 0x7f5f240ccad0 2024-03-22T13:04:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.046+0000 7f5f2a800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5f2415b0e0 0x7f5f24164db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:55858/0 (socket says 172.21.15.6:55858) 2024-03-22T13:04:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.046+0000 7f5f2a800640 1 -- 172.21.15.6:0/2162939320 learned_addr learned my addr 172.21.15.6:0/2162939320 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.046+0000 7f5f29400640 1 --2- 172.21.15.6:0/2162939320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f240ccad0 0x7f5f2406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.050+0000 7f5f29400640 1 -- 172.21.15.6:0/2162939320 >> v1:172.21.15.92:6789/0 conn(0x7f5f2415b0e0 legacy=0x7f5f24164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.050+0000 7f5f29400640 1 -- 172.21.15.6:0/2162939320 >> v1:172.21.15.6:6789/0 conn(0x7f5f240ca520 legacy=0x7f5f240cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.050+0000 7f5f28a00640 1 -- 172.21.15.6:0/2162939320 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941461541 0 0) 0x7f5f2406ca80 con 0x7f5f2415b0e0 2024-03-22T13:04:40.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.050+0000 7f5f29400640 1 -- 172.21.15.6:0/2162939320 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2406cd60 con 0x7f5f240ccad0 2024-03-22T13:04:40.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.053+0000 7f5f29400640 1 --2- 172.21.15.6:0/2162939320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f240ccad0 0x7f5f2406d840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5f140097d0 tx=0x7f5f1402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7a63face8a75cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.057+0000 7f5f28a00640 1 -- 172.21.15.6:0/2162939320 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14035020 con 0x7f5f240ccad0 2024-03-22T13:04:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.057+0000 7f5f28a00640 1 -- 172.21.15.6:0/2162939320 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f14031890 con 0x7f5f240ccad0 2024-03-22T13:04:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.057+0000 7f5f28a00640 1 -- 172.21.15.6:0/2162939320 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14031c60 con 0x7f5f240ccad0 2024-03-22T13:04:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.058+0000 7f5f2c800640 1 -- 172.21.15.6:0/2162939320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f240ccad0 msgr2=0x7f5f2406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.058+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2162939320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f240ccad0 0x7f5f2406d840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5f140097d0 tx=0x7f5f1402b930 comp rx=0 tx=0).stop 2024-03-22T13:04:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.058+0000 7f5f2c800640 1 -- 172.21.15.6:0/2162939320 shutdown_connections 2024-03-22T13:04:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.058+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2162939320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f240ccad0 0x7f5f2406d840 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.058+0000 7f5f2c800640 1 -- 172.21.15.6:0/2162939320 >> 172.21.15.6:0/2162939320 conn(0x7f5f240b16b0 msgr2=0x7f5f240b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.058+0000 7f5f2c800640 1 -- 172.21.15.6:0/2162939320 shutdown_connections 2024-03-22T13:04:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.058+0000 7f5f2c800640 1 -- 172.21.15.6:0/2162939320 wait complete. 2024-03-22T13:04:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.061+0000 7f5f2c800640 1 Processor -- start 2024-03-22T13:04:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.061+0000 7f5f2c800640 1 -- start start 2024-03-22T13:04:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.063+0000 7f5f2c800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f240ca520 0x7f5f24153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.063+0000 7f5f2c800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f240ccad0 0x7f5f241558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.063+0000 7f5f2c800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f2415b0e0 0x7f5f241573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.063+0000 7f5f2c800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f24154c20 con 0x7f5f240ca520 2024-03-22T13:04:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.063+0000 7f5f2c800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2406d320 con 0x7f5f240ccad0 2024-03-22T13:04:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.063+0000 7f5f2c800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2415b980 con 0x7f5f2415b0e0 2024-03-22T13:04:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.064+0000 7f5f2a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f2415b0e0 0x7f5f241573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.064+0000 7f5f2a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f2415b0e0 0x7f5f241573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:37710/0 (socket says 172.21.15.6:37710) 2024-03-22T13:04:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.064+0000 7f5f2a800640 1 -- 172.21.15.6:0/2573612190 learned_addr learned my addr 172.21.15.6:0/2573612190 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.064+0000 7f5f29e00640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f240ca520 0x7f5f24153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.064+0000 7f5f29400640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f240ccad0 0x7f5f241558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.068+0000 7f5f29400640 1 -- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f2415b0e0 msgr2=0x7f5f241573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.068+0000 7f5f29400640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f2415b0e0 0x7f5f241573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.068+0000 7f5f29400640 1 -- 172.21.15.6:0/2573612190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f240ca520 msgr2=0x7f5f24153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.068+0000 7f5f29400640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f240ca520 0x7f5f24153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.068+0000 7f5f29400640 1 -- 172.21.15.6:0/2573612190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24154150 con 0x7f5f240ccad0 2024-03-22T13:04:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.071+0000 7f5f29400640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f240ccad0 0x7f5f241558b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f14033000 tx=0x7f5f140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.073+0000 7f5f1e000640 1 -- 172.21.15.6:0/2573612190 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f14035040 con 0x7f5f240ccad0 2024-03-22T13:04:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.073+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f24158700 con 0x7f5f240ccad0 2024-03-22T13:04:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.073+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f241550b0 con 0x7f5f240ccad0 2024-03-22T13:04:40.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.075+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f24158bc0 con 0x7f5f240ccad0 2024-03-22T13:04:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.075+0000 7f5f1e000640 1 -- 172.21.15.6:0/2573612190 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f14031840 con 0x7f5f240ccad0 2024-03-22T13:04:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.075+0000 7f5f1e000640 1 -- 172.21.15.6:0/2573612190 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f140466c0 con 0x7f5f240ccad0 2024-03-22T13:04:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.339+0000 7f5f1e000640 1 -- 172.21.15.6:0/2573612190 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5f14038020 con 0x7f5f240ccad0 2024-03-22T13:04:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.341+0000 7f5f1e000640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f0806bbd0 0x7f5f0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.342+0000 7f5f1e000640 1 -- 172.21.15.6:0/2573612190 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5f140ad9c0 con 0x7f5f240ccad0 2024-03-22T13:04:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.342+0000 7f5f29e00640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f0806bbd0 0x7f5f0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.343+0000 7f5f29e00640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f0806bbd0 0x7f5f0806e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f18001780 tx=0x7f5f18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.353+0000 7f5f1e000640 1 -- 172.21.15.6:0/2573612190 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5f24158bc0 con 0x7f5f240ccad0 2024-03-22T13:04:40.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.608+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f240756c0 con 0x7f5f240ccad0 2024-03-22T13:04:40.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.622+0000 7f5f1e000640 1 -- 172.21.15.6:0/2573612190 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f5f1402d040 con 0x7f5f240ccad0 2024-03-22T13:04:40.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:40.622 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:40.622 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:40.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f0806bbd0 msgr2=0x7f5f0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:40.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f0806bbd0 0x7f5f0806e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f18001780 tx=0x7f5f18009040 comp rx=0 tx=0).stop 2024-03-22T13:04:40.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f240ccad0 msgr2=0x7f5f241558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:40.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f240ccad0 0x7f5f241558b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f14033000 tx=0x7f5f140046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 shutdown_connections 2024-03-22T13:04:40.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f2415b0e0 0x7f5f241573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f0806bbd0 0x7f5f0806e090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f240ccad0 0x7f5f241558b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 --2- 172.21.15.6:0/2573612190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f240ca520 0x7f5f24153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:40.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 >> 172.21.15.6:0/2573612190 conn(0x7f5f240b16b0 msgr2=0x7f5f240cf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:40.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 shutdown_connections 2024-03-22T13:04:40.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:40.626+0000 7f5f2c800640 1 -- 172.21.15.6:0/2573612190 wait complete. 2024-03-22T13:04:41.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:41.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:41.881 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:00.216 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:41.915 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:00.305 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:41.954 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:00.288 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:41.992 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:00.382 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:42.036 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:09:00.426 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:42.109 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:09:00.499 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:42.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:42.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.461+0000 7f1777e00640 1 Processor -- start 2024-03-22T13:04:42.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.462+0000 7f1777e00640 1 -- start start 2024-03-22T13:04:42.462 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:00.851 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:42.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.463+0000 7f1777e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780cc380 0x7f177806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:42.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.463+0000 7f1777e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f17780ce8a0 0x7f177815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:42.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.463+0000 7f1777e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17780cc760 0x7f17780cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:42.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.463+0000 7f1777e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780cee00 con 0x7f17780cc760 2024-03-22T13:04:42.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.463+0000 7f1777e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780cf0c0 con 0x7f17780ce8a0 2024-03-22T13:04:42.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.463+0000 7f1777e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780cf3a0 con 0x7f17780cc380 2024-03-22T13:04:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.486+0000 7f1776000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f17780ce8a0 0x7f177815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.487+0000 7f1776000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f17780ce8a0 0x7f177815ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37402/0 (socket says 172.21.15.6:37402) 2024-03-22T13:04:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.487+0000 7f1776000640 1 -- 172.21.15.6:0/1813898817 learned_addr learned my addr 172.21.15.6:0/1813898817 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:42.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.497+0000 7f1776000640 1 -- 172.21.15.6:0/1813898817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780cc380 msgr2=0x7f177806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:42.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.497+0000 7f1776000640 1 --2- 172.21.15.6:0/1813898817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780cc380 0x7f177806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:42.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.497+0000 7f1776000640 1 -- 172.21.15.6:0/1813898817 >> v2:172.21.15.6:3300/0 conn(0x7f17780cc760 msgr2=0x7f17780cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:42.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.497+0000 7f1776000640 1 --2- 172.21.15.6:0/1813898817 >> v2:172.21.15.6:3300/0 conn(0x7f17780cc760 0x7f17780cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:42.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.497+0000 7f1776000640 1 -- 172.21.15.6:0/1813898817 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780cf680 con 0x7f17780ce8a0 2024-03-22T13:04:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.510+0000 7f1776000640 1 --2- 172.21.15.6:0/1813898817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780ce8a0 0x7f177815ae30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f17780d0a70 tx=0x7f176402b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=338b8b784eb5b281 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:42.539 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:09:00.873 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.547+0000 7f1775600640 1 -- 172.21.15.6:0/1813898817 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1764035020 con 0x7f17780ce8a0 2024-03-22T13:04:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.548+0000 7f1775600640 1 -- 172.21.15.6:0/1813898817 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176402d040 con 0x7f17780ce8a0 2024-03-22T13:04:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.548+0000 7f1775600640 1 -- 172.21.15.6:0/1813898817 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1764031a80 con 0x7f17780ce8a0 2024-03-22T13:04:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 -- 172.21.15.6:0/1813898817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780ce8a0 msgr2=0x7f177815ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 --2- 172.21.15.6:0/1813898817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780ce8a0 0x7f177815ae30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f17780d0a70 tx=0x7f176402b900 comp rx=0 tx=0).stop 2024-03-22T13:04:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 -- 172.21.15.6:0/1813898817 shutdown_connections 2024-03-22T13:04:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 --2- 172.21.15.6:0/1813898817 >> v2:172.21.15.6:3300/0 conn(0x7f17780cc760 0x7f17780cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 --2- 172.21.15.6:0/1813898817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780ce8a0 0x7f177815ae30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 --2- 172.21.15.6:0/1813898817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780cc380 0x7f177806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 -- 172.21.15.6:0/1813898817 >> 172.21.15.6:0/1813898817 conn(0x7f17780b1480 msgr2=0x7f17780b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 -- 172.21.15.6:0/1813898817 shutdown_connections 2024-03-22T13:04:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.549+0000 7f1777e00640 1 -- 172.21.15.6:0/1813898817 wait complete. 2024-03-22T13:04:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.552+0000 7f1777e00640 1 Processor -- start 2024-03-22T13:04:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.552+0000 7f1777e00640 1 -- start start 2024-03-22T13:04:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.553+0000 7f1777e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780cc380 0x7f17781e44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.553+0000 7f1777e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17780cc760 0x7f17781e56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.553+0000 7f1777e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780ce8a0 0x7f17781f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.553+0000 7f1777e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17781549d0 con 0x7f17780cc760 2024-03-22T13:04:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.553+0000 7f1777e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177815a910 con 0x7f17780cc380 2024-03-22T13:04:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.553+0000 7f1777e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780ccb90 con 0x7f17780ce8a0 2024-03-22T13:04:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.559+0000 7f1776000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17780cc760 0x7f17781e56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:42.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.561+0000 7f1776a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780cc380 0x7f17781e44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:42.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.564+0000 7f1776000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17780cc760 0x7f17781e56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38298/0 (socket says 172.21.15.6:38298) 2024-03-22T13:04:42.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.564+0000 7f1776000640 1 -- 172.21.15.6:0/3253099899 learned_addr learned my addr 172.21.15.6:0/3253099899 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:42.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.573+0000 7f1776000640 1 -- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780ce8a0 msgr2=0x7f17781f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:42.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.573+0000 7f1776000640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780ce8a0 0x7f17781f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:42.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.573+0000 7f1776000640 1 -- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780cc380 msgr2=0x7f17781e44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:42.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.573+0000 7f1776000640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780cc380 0x7f17781e44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:42.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.573+0000 7f1776000640 1 -- 172.21.15.6:0/3253099899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177815dd00 con 0x7f17780cc760 2024-03-22T13:04:42.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.580+0000 7f1776000640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17780cc760 0x7f17781e56b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f176402b940 tx=0x7f17640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:42.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.622+0000 7f176ea00640 1 -- 172.21.15.6:0/3253099899 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176403ac60 con 0x7f17780cc760 2024-03-22T13:04:42.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.622+0000 7f176ea00640 1 -- 172.21.15.6:0/3253099899 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176403ae00 con 0x7f17780cc760 2024-03-22T13:04:42.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.622+0000 7f176ea00640 1 -- 172.21.15.6:0/3253099899 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1764044630 con 0x7f17780cc760 2024-03-22T13:04:42.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.622+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177815dfc0 con 0x7f17780cc760 2024-03-22T13:04:42.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.622+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17781e4eb0 con 0x7f17780cc760 2024-03-22T13:04:42.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.624+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1778075650 con 0x7f17780cc760 2024-03-22T13:04:42.636 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:09:01.027 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:04:42.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.918+0000 7f176ea00640 1 -- 172.21.15.6:0/3253099899 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1764038020 con 0x7f17780cc760 2024-03-22T13:04:42.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.919+0000 7f176ea00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f175806bc50 0x7f175806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:42.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.919+0000 7f1776a00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f175806bc50 0x7f175806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.920+0000 7f1776a00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f175806bc50 0x7f175806e110 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f17680098e0 tx=0x7f1768009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:42.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.957+0000 7f176ea00640 1 -- 172.21.15.6:0/3253099899 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1764079840 con 0x7f17780cc760 2024-03-22T13:04:42.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:42.958+0000 7f176ea00640 1 -- 172.21.15.6:0/3253099899 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f176403f070 con 0x7f17780cc760 2024-03-22T13:04:43.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.233+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17780c9ca0 con 0x7f17780cc760 2024-03-22T13:04:43.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.317+0000 7f176ea00640 1 -- 172.21.15.6:0/3253099899 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f176407b7a0 con 0x7f17780cc760 2024-03-22T13:04:43.318 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:43.318 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:43.318 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f175806bc50 msgr2=0x7f175806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f175806bc50 0x7f175806e110 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f17680098e0 tx=0x7f1768009060 comp rx=0 tx=0).stop 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17780cc760 msgr2=0x7f17781e56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17780cc760 0x7f17781e56b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f176402b940 tx=0x7f17640046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 shutdown_connections 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17780ce8a0 0x7f17781f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17780cc760 0x7f17781e56b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f175806bc50 0x7f175806e110 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 --2- 172.21.15.6:0/3253099899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17780cc380 0x7f17781e44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:43.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 >> 172.21.15.6:0/3253099899 conn(0x7f17780b1480 msgr2=0x7f1778158cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:43.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 shutdown_connections 2024-03-22T13:04:43.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:43.321+0000 7f1777e00640 1 -- 172.21.15.6:0/3253099899 wait complete. 2024-03-22T13:04:43.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:43.941 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:44.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.129+0000 7fdb88a00640 1 Processor -- start 2024-03-22T13:04:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.129+0000 7fdb88a00640 1 -- start start 2024-03-22T13:04:45.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.130+0000 7fdb88a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840cc380 0x7fdb8406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:45.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.130+0000 7fdb88a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb840ce8a0 0x7fdb8415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:45.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.130+0000 7fdb88a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb840cee00 con 0x7fdb840cc760 2024-03-22T13:04:45.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.130+0000 7fdb88a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb840cf0c0 con 0x7fdb840ce8a0 2024-03-22T13:04:45.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.130+0000 7fdb88a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840cf3a0 con 0x7fdb840cc380 2024-03-22T13:04:45.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.131+0000 7fdb83400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840cc380 0x7fdb8406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:45.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.131+0000 7fdb82a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb840ce8a0 0x7fdb8415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:45.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.131+0000 7fdb83400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840cc380 0x7fdb8406c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33424/0 (socket says 172.21.15.6:33424) 2024-03-22T13:04:45.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.131+0000 7fdb83400640 1 -- 172.21.15.6:0/3099657203 learned_addr learned my addr 172.21.15.6:0/3099657203 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.135+0000 7fdb83400640 1 -- 172.21.15.6:0/3099657203 >> v2:172.21.15.6:3300/0 conn(0x7fdb840ce8a0 msgr2=0x7fdb8415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:45.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.135+0000 7fdb83400640 1 --2- 172.21.15.6:0/3099657203 >> v2:172.21.15.6:3300/0 conn(0x7fdb840ce8a0 0x7fdb8415ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.135+0000 7fdb83400640 1 -- 172.21.15.6:0/3099657203 >> v1:172.21.15.6:6789/0 conn(0x7fdb840cc760 legacy=0x7fdb840cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:45.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.135+0000 7fdb83400640 1 -- 172.21.15.6:0/3099657203 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb840cf680 con 0x7fdb840cc380 2024-03-22T13:04:45.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.138+0000 7fdb83400640 1 --2- 172.21.15.6:0/3099657203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840cc380 0x7fdb8406c230 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009870 tx=0x7fdb7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cfd6479ebaaced6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.142+0000 7fdb82000640 1 -- 172.21.15.6:0/3099657203 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74034070 con 0x7fdb840cc380 2024-03-22T13:04:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.142+0000 7fdb82000640 1 -- 172.21.15.6:0/3099657203 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb74030840 con 0x7fdb840cc380 2024-03-22T13:04:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.142+0000 7fdb82000640 1 -- 172.21.15.6:0/3099657203 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74030c10 con 0x7fdb840cc380 2024-03-22T13:04:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 -- 172.21.15.6:0/3099657203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840cc380 msgr2=0x7fdb8406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 --2- 172.21.15.6:0/3099657203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840cc380 0x7fdb8406c230 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009870 tx=0x7fdb7402b980 comp rx=0 tx=0).stop 2024-03-22T13:04:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 -- 172.21.15.6:0/3099657203 shutdown_connections 2024-03-22T13:04:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 --2- 172.21.15.6:0/3099657203 >> v2:172.21.15.6:3300/0 conn(0x7fdb840ce8a0 0x7fdb8415ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 --2- 172.21.15.6:0/3099657203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840cc380 0x7fdb8406c230 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 -- 172.21.15.6:0/3099657203 >> 172.21.15.6:0/3099657203 conn(0x7fdb840b1480 msgr2=0x7fdb840b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 -- 172.21.15.6:0/3099657203 shutdown_connections 2024-03-22T13:04:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.143+0000 7fdb88a00640 1 -- 172.21.15.6:0/3099657203 wait complete. 2024-03-22T13:04:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.145+0000 7fdb88a00640 1 Processor -- start 2024-03-22T13:04:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.145+0000 7fdb88a00640 1 -- start start 2024-03-22T13:04:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.146+0000 7fdb88a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb840cc380 0x7fdb841e60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.146+0000 7fdb88a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb840cc760 0x7fdb84142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.146+0000 7fdb88a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840ce8a0 0x7fdb841e4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.146+0000 7fdb88a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb841549d0 con 0x7fdb840cc380 2024-03-22T13:04:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.146+0000 7fdb88a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8415a910 con 0x7fdb840cc760 2024-03-22T13:04:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.146+0000 7fdb88a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840ccb90 con 0x7fdb840ce8a0 2024-03-22T13:04:45.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.147+0000 7fdb83400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb840cc380 0x7fdb841e60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:45.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.147+0000 7fdb82a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb840cc760 0x7fdb84142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:45.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.147+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840ce8a0 0x7fdb841e4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:45.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.147+0000 7fdb82a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb840cc760 0x7fdb84142890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50216/0 (socket says 172.21.15.6:50216) 2024-03-22T13:04:45.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.147+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840ce8a0 0x7fdb841e4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33440/0 (socket says 172.21.15.6:33440) 2024-03-22T13:04:45.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.147+0000 7fdb82a00640 1 -- 172.21.15.6:0/2695761937 learned_addr learned my addr 172.21.15.6:0/2695761937 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.151+0000 7fdb83400640 1 -- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840ce8a0 msgr2=0x7fdb841e4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.151+0000 7fdb83400640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840ce8a0 0x7fdb841e4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.151+0000 7fdb83400640 1 -- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb840cc760 msgr2=0x7fdb84142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.151+0000 7fdb83400640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb840cc760 0x7fdb84142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.151+0000 7fdb83400640 1 -- 172.21.15.6:0/2695761937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb841e36f0 con 0x7fdb840cc380 2024-03-22T13:04:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.154+0000 7fdb83400640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb840cc380 0x7fdb841e60f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdb840cf520 tx=0x7fdb740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:45.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.158+0000 7fdb7b400640 1 -- 172.21.15.6:0/2695761937 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74034040 con 0x7fdb840cc380 2024-03-22T13:04:45.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.158+0000 7fdb7b400640 1 -- 172.21.15.6:0/2695761937 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb74039d90 con 0x7fdb840cc380 2024-03-22T13:04:45.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.158+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb841e3970 con 0x7fdb840cc380 2024-03-22T13:04:45.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.158+0000 7fdb7b400640 1 -- 172.21.15.6:0/2695761937 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74043610 con 0x7fdb840cc380 2024-03-22T13:04:45.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.159+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb84148100 con 0x7fdb840cc380 2024-03-22T13:04:45.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.160+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb84075650 con 0x7fdb840cc380 2024-03-22T13:04:45.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.193+0000 7fdb7b400640 1 -- 172.21.15.6:0/2695761937 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdb7404a060 con 0x7fdb840cc380 2024-03-22T13:04:45.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.194+0000 7fdb7b400640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb6406bc50 0x7fdb6406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:45.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.195+0000 7fdb82a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb6406bc50 0x7fdb6406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:45.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.196+0000 7fdb82a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb6406bc50 0x7fdb6406e110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb84068340 tx=0x7fdb70009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:45.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.378+0000 7fdb7b400640 1 -- 172.21.15.6:0/2695761937 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdb74079000 con 0x7fdb840cc380 2024-03-22T13:04:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.464+0000 7fdb7b400640 1 -- 172.21.15.6:0/2695761937 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdb841e36f0 con 0x7fdb840cc380 2024-03-22T13:04:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.716+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb840af180 con 0x7fdb840cc380 2024-03-22T13:04:45.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.731+0000 7fdb7b400640 1 -- 172.21.15.6:0/2695761937 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fdb7403e070 con 0x7fdb840cc380 2024-03-22T13:04:45.804 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:45.804 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:45.805 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:45.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.733+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb6406bc50 msgr2=0x7fdb6406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:45.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.733+0000 7fdb88a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb6406bc50 0x7fdb6406e110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb84068340 tx=0x7fdb70009190 comp rx=0 tx=0).stop 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.733+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb840cc380 msgr2=0x7fdb841e60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.733+0000 7fdb88a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb840cc380 0x7fdb841e60f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdb840cf520 tx=0x7fdb740046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 shutdown_connections 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb840ce8a0 0x7fdb841e4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb6406bc50 0x7fdb6406e110 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb840cc760 0x7fdb84142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 --2- 172.21.15.6:0/2695761937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb840cc380 0x7fdb841e60f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 >> 172.21.15.6:0/2695761937 conn(0x7fdb840b1480 msgr2=0x7fdb84147350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 shutdown_connections 2024-03-22T13:04:45.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:45.734+0000 7fdb88a00640 1 -- 172.21.15.6:0/2695761937 wait complete. 2024-03-22T13:04:46.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:46.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:47.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:47.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.615+0000 7f6ccf600640 1 Processor -- start 2024-03-22T13:04:47.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.615+0000 7f6ccf600640 1 -- start start 2024-03-22T13:04:47.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.617+0000 7f6ccf600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc80ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:47.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.617+0000 7f6ccf600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6cc815ab40 0x7f6cc815afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:47.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.617+0000 7f6ccf600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc815b510 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.617+0000 7f6ccf600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc80cd0f0 con 0x7f6cc815ab40 2024-03-22T13:04:47.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.617+0000 7f6ccf600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc80ce230 con 0x7f6cc80cc790 2024-03-22T13:04:47.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.617+0000 7f6cccc00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6cc80cc3b0 0x7f6cc80cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:45284/0 (socket says 172.21.15.6:45284) 2024-03-22T13:04:47.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.617+0000 7f6cccc00640 1 -- 172.21.15.6:0/1349969733 learned_addr learned my addr 172.21.15.6:0/1349969733 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:04:47.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.618+0000 7f6cc7e00640 1 --2- 172.21.15.6:0/1349969733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc80ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:47.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.618+0000 7f6ccd600640 1 --2- 172.21.15.6:0/1349969733 >> v2:172.21.15.6:3300/0 conn(0x7f6cc815ab40 0x7f6cc815afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:47.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.620+0000 7f6cc7400640 1 -- 172.21.15.6:0/1349969733 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288220602 0 0) 0x7f6cc815b510 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.620+0000 7f6cc7400640 1 -- 172.21.15.6:0/1349969733 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb0003520 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.621+0000 7f6ccd600640 1 -- 172.21.15.6:0/1349969733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 msgr2=0x7f6cc80ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:47.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.621+0000 7f6ccd600640 1 --2- 172.21.15.6:0/1349969733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc80ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:47.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.621+0000 7f6ccd600640 1 -- 172.21.15.6:0/1349969733 >> v1:172.21.15.92:6789/0 conn(0x7f6cc80cc3b0 legacy=0x7f6cc80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:47.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.621+0000 7f6ccd600640 1 -- 172.21.15.6:0/1349969733 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc80ce510 con 0x7f6cc815ab40 2024-03-22T13:04:47.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.623+0000 7f6ccd600640 1 --2- 172.21.15.6:0/1349969733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 0x7f6cc815afd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009d90 tx=0x7f6cc002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72daed30baaa6139 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:47.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.628+0000 7f6cc7400640 1 -- 172.21.15.6:0/1349969733 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0039070 con 0x7f6cc815ab40 2024-03-22T13:04:47.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.628+0000 7f6cc7400640 1 -- 172.21.15.6:0/1349969733 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc002cbc0 con 0x7f6cc815ab40 2024-03-22T13:04:47.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.628+0000 7f6cc7400640 1 -- 172.21.15.6:0/1349969733 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0034440 con 0x7f6cc815ab40 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 -- 172.21.15.6:0/1349969733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 msgr2=0x7f6cc815afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 --2- 172.21.15.6:0/1349969733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 0x7f6cc815afd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009d90 tx=0x7f6cc002be10 comp rx=0 tx=0).stop 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 -- 172.21.15.6:0/1349969733 shutdown_connections 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 --2- 172.21.15.6:0/1349969733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 0x7f6cc815afd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 --2- 172.21.15.6:0/1349969733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc80ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 -- 172.21.15.6:0/1349969733 >> 172.21.15.6:0/1349969733 conn(0x7f6cc80b16d0 msgr2=0x7f6cc80b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 -- 172.21.15.6:0/1349969733 shutdown_connections 2024-03-22T13:04:47.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.629+0000 7f6ccf600640 1 -- 172.21.15.6:0/1349969733 wait complete. 2024-03-22T13:04:47.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.632+0000 7f6ccf600640 1 Processor -- start 2024-03-22T13:04:47.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.632+0000 7f6ccf600640 1 -- start start 2024-03-22T13:04:47.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6ccf600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cc80cc3b0 0x7f6cc80b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:47.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6ccf600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:47.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6ccf600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 0x7f6cc80b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:47.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6ccf600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc80cedb0 con 0x7f6cc815ab40 2024-03-22T13:04:47.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6ccf600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc80cead0 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6ccf600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc806dc10 con 0x7f6cc80cc790 2024-03-22T13:04:47.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6cccc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cc80cc3b0 0x7f6cc80b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:47.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6ccd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 0x7f6cc80b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:47.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.634+0000 7f6cc7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:47.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.635+0000 7f6cccc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cc80cc3b0 0x7f6cc80b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50224/0 (socket says 172.21.15.6:50224) 2024-03-22T13:04:47.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.635+0000 7f6cccc00640 1 -- 172.21.15.6:0/3796258737 learned_addr learned my addr 172.21.15.6:0/3796258737 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:47.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.638+0000 7f6cccc00640 1 -- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 msgr2=0x7f6cc815dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:47.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.638+0000 7f6cccc00640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc815dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:47.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.638+0000 7f6cccc00640 1 -- 172.21.15.6:0/3796258737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 msgr2=0x7f6cc80b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:47.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.638+0000 7f6cccc00640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 0x7f6cc80b2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:47.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.638+0000 7f6cccc00640 1 -- 172.21.15.6:0/3796258737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc80b4f00 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.641+0000 7f6cccc00640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cc80cc3b0 0x7f6cc80b44a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00c490 tx=0x7f6cbc00c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:47.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.644+0000 7f6cc4c00640 1 -- 172.21.15.6:0/3796258737 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc017070 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.645+0000 7f6cc4c00640 1 -- 172.21.15.6:0/3796258737 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc005ce0 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.645+0000 7f6cc4c00640 1 -- 172.21.15.6:0/3796258737 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0072f0 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.645+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc80b51e0 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.645+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc80b57a0 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.646+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc80756c0 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.908+0000 7f6cc4c00640 1 -- 172.21.15.6:0/3796258737 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6cbc01a070 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.910+0000 7f6cc4c00640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6ca806bc50 0x7f6ca806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.910+0000 7f6cc4c00640 1 -- 172.21.15.6:0/3796258737 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6cbc08a5f0 con 0x7f6cc80cc3b0 2024-03-22T13:04:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.910+0000 7f6cc7e00640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6ca806bc50 0x7f6ca806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:47.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.912+0000 7f6cc7e00640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6ca806bc50 0x7f6ca806e110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80af9e0 tx=0x7f6cb4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:47.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:47.922+0000 7f6cc4c00640 1 -- 172.21.15.6:0/3796258737 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6cbc021020 con 0x7f6cc80cc3b0 2024-03-22T13:04:48.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.177+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cc80b5b40 con 0x7f6cc80cc3b0 2024-03-22T13:04:48.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.191+0000 7f6cc4c00640 1 -- 172.21.15.6:0/3796258737 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f6cbc058770 con 0x7f6cc80cc3b0 2024-03-22T13:04:48.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:48.191 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:48.191 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6ca806bc50 msgr2=0x7f6ca806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6ca806bc50 0x7f6ca806e110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80af9e0 tx=0x7f6cb4009190 comp rx=0 tx=0).stop 2024-03-22T13:04:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cc80cc3b0 msgr2=0x7f6cc80b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cc80cc3b0 0x7f6cc80b44a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00c490 tx=0x7f6cbc00c950 comp rx=0 tx=0).stop 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 shutdown_connections 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6cc815ab40 0x7f6cc80b2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6cc80cc790 0x7f6cc815dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6ca806bc50 0x7f6ca806e110 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 --2- 172.21.15.6:0/3796258737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6cc80cc3b0 0x7f6cc80b44a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 >> 172.21.15.6:0/3796258737 conn(0x7f6cc80b16d0 msgr2=0x7f6cc8154ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 shutdown_connections 2024-03-22T13:04:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:48.193+0000 7f6ccf600640 1 -- 172.21.15.6:0/3796258737 wait complete. 2024-03-22T13:04:48.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:48.814 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:49.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:50.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.015+0000 7f2862a00640 1 Processor -- start 2024-03-22T13:04:50.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.015+0000 7f2862a00640 1 -- start start 2024-03-22T13:04:50.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.016+0000 7f2862a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:50.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.016+0000 7f2862a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f285c0cf770 0x7f285c0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:50.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.016+0000 7f2862a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f285c0cccd0 0x7f285c15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:50.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.016+0000 7f2862a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c0ce230 con 0x7f285c0cccd0 2024-03-22T13:04:50.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.016+0000 7f2862a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c0ce510 con 0x7f285c0cf770 2024-03-22T13:04:50.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.016+0000 7f2862a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0ce7f0 con 0x7f285c0cc3b0 2024-03-22T13:04:50.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.017+0000 7f285b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f285c0cf770 0x7f285c0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:50.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.017+0000 7f285be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:50.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.017+0000 7f2860a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f285c0cccd0 0x7f285c15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:50.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.017+0000 7f285b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f285c0cf770 0x7f285c0cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50238/0 (socket says 172.21.15.6:50238) 2024-03-22T13:04:50.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.018+0000 7f285b400640 1 -- 172.21.15.6:0/2407918691 learned_addr learned my addr 172.21.15.6:0/2407918691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.020+0000 7f285b400640 1 -- 172.21.15.6:0/2407918691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 msgr2=0x7f285c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.020+0000 7f285b400640 1 --2- 172.21.15.6:0/2407918691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.020+0000 7f285b400640 1 -- 172.21.15.6:0/2407918691 >> v2:172.21.15.6:3300/0 conn(0x7f285c0cccd0 msgr2=0x7f285c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.020+0000 7f285b400640 1 --2- 172.21.15.6:0/2407918691 >> v2:172.21.15.6:3300/0 conn(0x7f285c0cccd0 0x7f285c15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.020+0000 7f285b400640 1 -- 172.21.15.6:0/2407918691 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c0cead0 con 0x7f285c0cf770 2024-03-22T13:04:50.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.023+0000 7f285b400640 1 --2- 172.21.15.6:0/2407918691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c0cc790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f285c08d840 tx=0x7f284402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58a04b3fa502f06 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.028+0000 7f285aa00640 1 -- 172.21.15.6:0/2407918691 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284402e070 con 0x7f285c0cf770 2024-03-22T13:04:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.028+0000 7f285aa00640 1 -- 172.21.15.6:0/2407918691 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844037070 con 0x7f285c0cf770 2024-03-22T13:04:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.028+0000 7f285aa00640 1 -- 172.21.15.6:0/2407918691 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844031a00 con 0x7f285c0cf770 2024-03-22T13:04:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 -- 172.21.15.6:0/2407918691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 msgr2=0x7f285c0cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 --2- 172.21.15.6:0/2407918691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c0cc790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f285c08d840 tx=0x7f284402b880 comp rx=0 tx=0).stop 2024-03-22T13:04:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 -- 172.21.15.6:0/2407918691 shutdown_connections 2024-03-22T13:04:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 --2- 172.21.15.6:0/2407918691 >> v2:172.21.15.6:3300/0 conn(0x7f285c0cccd0 0x7f285c15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 --2- 172.21.15.6:0/2407918691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c0cc790 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 --2- 172.21.15.6:0/2407918691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 -- 172.21.15.6:0/2407918691 >> 172.21.15.6:0/2407918691 conn(0x7f285c0b16d0 msgr2=0x7f285c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 -- 172.21.15.6:0/2407918691 shutdown_connections 2024-03-22T13:04:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.029+0000 7f2862a00640 1 -- 172.21.15.6:0/2407918691 wait complete. 2024-03-22T13:04:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.031+0000 7f2862a00640 1 Processor -- start 2024-03-22T13:04:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.031+0000 7f2862a00640 1 -- start start 2024-03-22T13:04:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.032+0000 7f2862a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.032+0000 7f2862a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285c0cccd0 0x7f285c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.032+0000 7f2862a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.032+0000 7f2862a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c15a5d0 con 0x7f285c0cccd0 2024-03-22T13:04:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.032+0000 7f2862a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c06dc10 con 0x7f285c0cf770 2024-03-22T13:04:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.032+0000 7f2862a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c154bb0 con 0x7f285c0cc3b0 2024-03-22T13:04:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.033+0000 7f2860a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.033+0000 7f285be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.033+0000 7f285b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285c0cccd0 0x7f285c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.033+0000 7f2860a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:50250/0 (socket says 172.21.15.6:50250) 2024-03-22T13:04:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.033+0000 7f2860a00640 1 -- 172.21.15.6:0/3694946006 learned_addr learned my addr 172.21.15.6:0/3694946006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:50.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.038+0000 7f2860a00640 1 -- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 msgr2=0x7f285c0c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:50.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.038+0000 7f2860a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.038+0000 7f2860a00640 1 -- 172.21.15.6:0/3694946006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285c0cccd0 msgr2=0x7f285c0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:50.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.038+0000 7f2860a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285c0cccd0 0x7f285c0c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.038+0000 7f2860a00640 1 -- 172.21.15.6:0/3694946006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c0c92d0 con 0x7f285c0cf770 2024-03-22T13:04:50.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.041+0000 7f2860a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c153440 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f285400e960 tx=0x7f285400ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:50.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.046+0000 7f2853e00640 1 -- 172.21.15.6:0/3694946006 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854017070 con 0x7f285c0cf770 2024-03-22T13:04:50.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.046+0000 7f2853e00640 1 -- 172.21.15.6:0/3694946006 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285400c980 con 0x7f285c0cf770 2024-03-22T13:04:50.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.046+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c0c86b0 con 0x7f285c0cf770 2024-03-22T13:04:50.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.046+0000 7f2853e00640 1 -- 172.21.15.6:0/3694946006 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400cd70 con 0x7f285c0cf770 2024-03-22T13:04:50.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.046+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c151bf0 con 0x7f285c0cf770 2024-03-22T13:04:50.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.047+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c0b1050 con 0x7f285c0cf770 2024-03-22T13:04:50.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.078+0000 7f2853e00640 1 -- 172.21.15.6:0/3694946006 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f285401a070 con 0x7f285c0cf770 2024-03-22T13:04:50.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.080+0000 7f2853e00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f283806bc20 0x7f283806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:50.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.080+0000 7f285be00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f283806bc20 0x7f283806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:50.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.081+0000 7f285be00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f283806bc20 0x7f283806e0e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f284c011360 tx=0x7f284c00d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:50.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.083+0000 7f2853e00640 1 -- 172.21.15.6:0/3694946006 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f2854057730 con 0x7f285c0cf770 2024-03-22T13:04:50.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.341+0000 7f2853e00640 1 -- 172.21.15.6:0/3694946006 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f285c0b1050 con 0x7f285c0cf770 2024-03-22T13:04:50.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.601+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f285c0756c0 con 0x7f285c0cf770 2024-03-22T13:04:50.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.616+0000 7f2853e00640 1 -- 172.21.15.6:0/3694946006 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f2854059690 con 0x7f285c0cf770 2024-03-22T13:04:50.617 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:50.617 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:50.617 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f283806bc20 msgr2=0x7f283806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f283806bc20 0x7f283806e0e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f284c011360 tx=0x7f284c00d040 comp rx=0 tx=0).stop 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 msgr2=0x7f285c153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c153440 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f285400e960 tx=0x7f285400ee20 comp rx=0 tx=0).stop 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 shutdown_connections 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f283806bc20 0x7f283806e0e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f285c0cf770 0x7f285c153440 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285c0cccd0 0x7f285c0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 --2- 172.21.15.6:0/3694946006 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f285c0cc3b0 0x7f285c0c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 >> 172.21.15.6:0/3694946006 conn(0x7f285c0b16d0 msgr2=0x7f285c0b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:50.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 shutdown_connections 2024-03-22T13:04:50.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:50.619+0000 7f2862a00640 1 -- 172.21.15.6:0/3694946006 wait complete. 2024-03-22T13:04:51.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:51.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:52.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.438+0000 7fcf92c00640 1 Processor -- start 2024-03-22T13:04:52.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.439+0000 7fcf92c00640 1 -- start start 2024-03-22T13:04:52.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.440+0000 7fcf92c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c1589e0 0x7fcf8c15f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:52.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.440+0000 7fcf92c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcf8c072c10 0x7fcf8c1623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:52.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.440+0000 7fcf92c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcf8c1628f0 0x7fcf8c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:52.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.440+0000 7fcf92c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8c0c6dd0 con 0x7fcf8c1628f0 2024-03-22T13:04:52.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.440+0000 7fcf92c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8c161f20 con 0x7fcf8c072c10 2024-03-22T13:04:52.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.440+0000 7fcf92c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c15f670 con 0x7fcf8c1589e0 2024-03-22T13:04:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.441+0000 7fcf91800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c1589e0 0x7fcf8c15f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.441+0000 7fcf90e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcf8c072c10 0x7fcf8c1623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.441+0000 7fcf91800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c1589e0 0x7fcf8c15f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33492/0 (socket says 172.21.15.6:33492) 2024-03-22T13:04:52.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.441+0000 7fcf91800640 1 -- 172.21.15.6:0/3478136518 learned_addr learned my addr 172.21.15.6:0/3478136518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:52.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.444+0000 7fcf91800640 1 -- 172.21.15.6:0/3478136518 >> v2:172.21.15.92:3300/0 conn(0x7fcf8c072c10 msgr2=0x7fcf8c1623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:52.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.444+0000 7fcf91800640 1 --2- 172.21.15.6:0/3478136518 >> v2:172.21.15.92:3300/0 conn(0x7fcf8c072c10 0x7fcf8c1623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.444+0000 7fcf91800640 1 -- 172.21.15.6:0/3478136518 >> v2:172.21.15.6:3300/0 conn(0x7fcf8c1628f0 msgr2=0x7fcf8c06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:52.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.444+0000 7fcf91800640 1 --2- 172.21.15.6:0/3478136518 >> v2:172.21.15.6:3300/0 conn(0x7fcf8c1628f0 0x7fcf8c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.444+0000 7fcf91800640 1 -- 172.21.15.6:0/3478136518 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c0715b0 con 0x7fcf8c1589e0 2024-03-22T13:04:52.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.447+0000 7fcf91800640 1 --2- 172.21.15.6:0/3478136518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c1589e0 0x7fcf8c15f130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009870 tx=0x7fcf8002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fecf4e23bd07c70 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.452+0000 7fcf8be00640 1 -- 172.21.15.6:0/3478136518 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80035020 con 0x7fcf8c1589e0 2024-03-22T13:04:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.452+0000 7fcf8be00640 1 -- 172.21.15.6:0/3478136518 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf80031640 con 0x7fcf8c1589e0 2024-03-22T13:04:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.452+0000 7fcf8be00640 1 -- 172.21.15.6:0/3478136518 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8002d040 con 0x7fcf8c1589e0 2024-03-22T13:04:52.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 -- 172.21.15.6:0/3478136518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c1589e0 msgr2=0x7fcf8c15f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:52.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 --2- 172.21.15.6:0/3478136518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c1589e0 0x7fcf8c15f130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009870 tx=0x7fcf8002b8f0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 -- 172.21.15.6:0/3478136518 shutdown_connections 2024-03-22T13:04:52.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 --2- 172.21.15.6:0/3478136518 >> v2:172.21.15.6:3300/0 conn(0x7fcf8c1628f0 0x7fcf8c06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 --2- 172.21.15.6:0/3478136518 >> v2:172.21.15.92:3300/0 conn(0x7fcf8c072c10 0x7fcf8c1623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 --2- 172.21.15.6:0/3478136518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c1589e0 0x7fcf8c15f130 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 -- 172.21.15.6:0/3478136518 >> 172.21.15.6:0/3478136518 conn(0x7fcf8c0cb890 msgr2=0x7fcf8c0cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:52.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 -- 172.21.15.6:0/3478136518 shutdown_connections 2024-03-22T13:04:52.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.453+0000 7fcf92c00640 1 -- 172.21.15.6:0/3478136518 wait complete. 2024-03-22T13:04:52.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.456+0000 7fcf92c00640 1 Processor -- start 2024-03-22T13:04:52.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.456+0000 7fcf92c00640 1 -- start start 2024-03-22T13:04:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.457+0000 7fcf92c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c072c10 0x7fcf8c1f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.458+0000 7fcf92c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf8c1589e0 0x7fcf8c0c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.458+0000 7fcf92c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf8c1628f0 0x7fcf8c1487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.458+0000 7fcf92c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c072450 con 0x7fcf8c1589e0 2024-03-22T13:04:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.458+0000 7fcf92c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c072170 con 0x7fcf8c1628f0 2024-03-22T13:04:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.458+0000 7fcf92c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c158dc0 con 0x7fcf8c072c10 2024-03-22T13:04:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.458+0000 7fcf91800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c072c10 0x7fcf8c1f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.458+0000 7fcf92200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf8c1628f0 0x7fcf8c1487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:52.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.459+0000 7fcf91800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c072c10 0x7fcf8c1f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33496/0 (socket says 172.21.15.6:33496) 2024-03-22T13:04:52.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.459+0000 7fcf91800640 1 -- 172.21.15.6:0/2808631433 learned_addr learned my addr 172.21.15.6:0/2808631433 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:52.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.463+0000 7fcf91800640 1 -- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf8c1628f0 msgr2=0x7fcf8c1487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:52.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.463+0000 7fcf91800640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf8c1628f0 0x7fcf8c1487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.463+0000 7fcf91800640 1 -- 172.21.15.6:0/2808631433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf8c1589e0 msgr2=0x7fcf8c0c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:52.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.463+0000 7fcf91800640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf8c1589e0 0x7fcf8c0c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:52.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.463+0000 7fcf91800640 1 -- 172.21.15.6:0/2808631433 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c149d70 con 0x7fcf8c072c10 2024-03-22T13:04:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.466+0000 7fcf91800640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c072c10 0x7fcf8c1f4f60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcf800097c0 tx=0x7fcf800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.471+0000 7fcf89600640 1 -- 172.21.15.6:0/2808631433 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80040070 con 0x7fcf8c072c10 2024-03-22T13:04:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.471+0000 7fcf89600640 1 -- 172.21.15.6:0/2808631433 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf800399f0 con 0x7fcf8c072c10 2024-03-22T13:04:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.471+0000 7fcf89600640 1 -- 172.21.15.6:0/2808631433 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80035040 con 0x7fcf8c072c10 2024-03-22T13:04:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.471+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8c14a010 con 0x7fcf8c072c10 2024-03-22T13:04:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.471+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8c1f4760 con 0x7fcf8c072c10 2024-03-22T13:04:52.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.472+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8c075650 con 0x7fcf8c072c10 2024-03-22T13:04:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.504+0000 7fcf89600640 1 -- 172.21.15.6:0/2808631433 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcf80038020 con 0x7fcf8c072c10 2024-03-22T13:04:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.505+0000 7fcf89600640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf7006bbd0 0x7fcf7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.506+0000 7fcf90e00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf7006bbd0 0x7fcf7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.507+0000 7fcf90e00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf7006bbd0 0x7fcf7006e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c006a90 tx=0x7fcf7c006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.507+0000 7fcf89600640 1 -- 172.21.15.6:0/2808631433 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fcf800790a0 con 0x7fcf8c072c10 2024-03-22T13:04:52.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:52.871+0000 7fcf89600640 1 -- 172.21.15.6:0/2808631433 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcf8007b000 con 0x7fcf8c072c10 2024-03-22T13:04:53.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.120+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf8c152a60 con 0x7fcf8c072c10 2024-03-22T13:04:53.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.134+0000 7fcf89600640 1 -- 172.21.15.6:0/2808631433 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fcf8007f0a0 con 0x7fcf8c072c10 2024-03-22T13:04:53.135 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:53.135 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:53.136 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf7006bbd0 msgr2=0x7fcf7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf7006bbd0 0x7fcf7006e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c006a90 tx=0x7fcf7c006a20 comp rx=0 tx=0).stop 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c072c10 msgr2=0x7fcf8c1f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c072c10 0x7fcf8c1f4f60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcf800097c0 tx=0x7fcf800046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 shutdown_connections 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcf7006bbd0 0x7fcf7006e090 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf8c1628f0 0x7fcf8c1487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf8c1589e0 0x7fcf8c0c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:53.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 --2- 172.21.15.6:0/2808631433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8c072c10 0x7fcf8c1f4f60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:53.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 >> 172.21.15.6:0/2808631433 conn(0x7fcf8c0cb890 msgr2=0x7fcf8c157490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:53.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 shutdown_connections 2024-03-22T13:04:53.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:53.137+0000 7fcf92c00640 1 -- 172.21.15.6:0/2808631433 wait complete. 2024-03-22T13:04:53.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:53.758 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:54.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:54.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.970+0000 7f3f3f400640 1 Processor -- start 2024-03-22T13:04:54.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.970+0000 7f3f3f400640 1 -- start start 2024-03-22T13:04:54.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.971+0000 7f3f3f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3f38150f50 0x7f3f380c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:54.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.971+0000 7f3f3f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f3806b080 0x7f3f3806b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:54.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.972+0000 7f3f3f400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f380c3bf0 con 0x7f3f380c5c20 2024-03-22T13:04:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.972+0000 7f3f3f400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f380c3760 con 0x7f3f3806b080 2024-03-22T13:04:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.972+0000 7f3f3f400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f38071e70 con 0x7f3f38150f50 2024-03-22T13:04:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.972+0000 7f3f3ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3f38150f50 0x7f3f380c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.972+0000 7f3f3ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3f38150f50 0x7f3f380c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36768/0 (socket says 172.21.15.6:36768) 2024-03-22T13:04:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.972+0000 7f3f3ca00640 1 -- 172.21.15.6:0/374842950 learned_addr learned my addr 172.21.15.6:0/374842950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:54.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.974+0000 7f3f3d400640 1 --2- 172.21.15.6:0/374842950 >> v2:172.21.15.6:3300/0 conn(0x7f3f3806b080 0x7f3f3806b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:54.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.975+0000 7f3f3ca00640 1 -- 172.21.15.6:0/374842950 >> v2:172.21.15.6:3300/0 conn(0x7f3f3806b080 msgr2=0x7f3f3806b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:54.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.975+0000 7f3f3ca00640 1 --2- 172.21.15.6:0/374842950 >> v2:172.21.15.6:3300/0 conn(0x7f3f3806b080 0x7f3f3806b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:54.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.975+0000 7f3f3ca00640 1 -- 172.21.15.6:0/374842950 >> v1:172.21.15.6:6789/0 conn(0x7f3f380c5c20 legacy=0x7f3f38156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:54.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.975+0000 7f3f3ca00640 1 -- 172.21.15.6:0/374842950 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38072150 con 0x7f3f38150f50 2024-03-22T13:04:54.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.977+0000 7f3f3ca00640 1 --2- 172.21.15.6:0/374842950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f38150f50 0x7f3f380c79e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3f3806bfd0 tx=0x7f3f2c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ca4bb77538d188c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.983+0000 7f3f37400640 1 -- 172.21.15.6:0/374842950 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c034070 con 0x7f3f38150f50 2024-03-22T13:04:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.983+0000 7f3f37400640 1 -- 172.21.15.6:0/374842950 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c030860 con 0x7f3f38150f50 2024-03-22T13:04:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.983+0000 7f3f37400640 1 -- 172.21.15.6:0/374842950 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c030c30 con 0x7f3f38150f50 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 -- 172.21.15.6:0/374842950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f38150f50 msgr2=0x7f3f380c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 --2- 172.21.15.6:0/374842950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f38150f50 0x7f3f380c79e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3f3806bfd0 tx=0x7f3f2c02b9a0 comp rx=0 tx=0).stop 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 -- 172.21.15.6:0/374842950 shutdown_connections 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 --2- 172.21.15.6:0/374842950 >> v2:172.21.15.6:3300/0 conn(0x7f3f3806b080 0x7f3f3806b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 --2- 172.21.15.6:0/374842950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f38150f50 0x7f3f380c79e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 -- 172.21.15.6:0/374842950 >> 172.21.15.6:0/374842950 conn(0x7f3f380cb930 msgr2=0x7f3f380cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 -- 172.21.15.6:0/374842950 shutdown_connections 2024-03-22T13:04:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.984+0000 7f3f3f400640 1 -- 172.21.15.6:0/374842950 wait complete. 2024-03-22T13:04:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.987+0000 7f3f3f400640 1 Processor -- start 2024-03-22T13:04:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.987+0000 7f3f3f400640 1 -- start start 2024-03-22T13:04:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.988+0000 7f3f3f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3806b080 0x7f3f38162860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.988+0000 7f3f3f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f380c5c20 0x7f3f380cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.988+0000 7f3f3f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f38150f50 0x7f3f381f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.988+0000 7f3f3f400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f380c33f0 con 0x7f3f38150f50 2024-03-22T13:04:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.988+0000 7f3f3f400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f380c3110 con 0x7f3f3806b080 2024-03-22T13:04:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.988+0000 7f3f3f400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f38151330 con 0x7f3f380c5c20 2024-03-22T13:04:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.989+0000 7f3f3ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3806b080 0x7f3f38162860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.989+0000 7f3f3ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3806b080 0x7f3f38162860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36770/0 (socket says 172.21.15.6:36770) 2024-03-22T13:04:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.989+0000 7f3f3ca00640 1 -- 172.21.15.6:0/2639816015 learned_addr learned my addr 172.21.15.6:0/2639816015 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.989+0000 7f3f37e00640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f380c5c20 0x7f3f380cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.989+0000 7f3f3d400640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f38150f50 0x7f3f381f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.992+0000 7f3f3ca00640 1 -- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f380c5c20 msgr2=0x7f3f380cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.992+0000 7f3f3ca00640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f380c5c20 0x7f3f380cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.992+0000 7f3f3ca00640 1 -- 172.21.15.6:0/2639816015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f38150f50 msgr2=0x7f3f381f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.992+0000 7f3f3ca00640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f38150f50 0x7f3f381f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.992+0000 7f3f3ca00640 1 -- 172.21.15.6:0/2639816015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f381f45a0 con 0x7f3f3806b080 2024-03-22T13:04:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.994+0000 7f3f3ca00640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3806b080 0x7f3f38162860 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c02b9e0 tx=0x7f3f2c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.999+0000 7f3f34c00640 1 -- 172.21.15.6:0/2639816015 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c034040 con 0x7f3f3806b080 2024-03-22T13:04:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.999+0000 7f3f34c00640 1 -- 172.21.15.6:0/2639816015 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c0433f0 con 0x7f3f3806b080 2024-03-22T13:04:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.999+0000 7f3f34c00640 1 -- 172.21.15.6:0/2639816015 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0437e0 con 0x7f3f3806b080 2024-03-22T13:04:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.999+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f381f4880 con 0x7f3f3806b080 2024-03-22T13:04:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:54.999+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f38164610 con 0x7f3f3806b080 2024-03-22T13:04:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.001+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f380756c0 con 0x7f3f3806b080 2024-03-22T13:04:55.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.029+0000 7f3f34c00640 1 -- 172.21.15.6:0/2639816015 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f2c037070 con 0x7f3f3806b080 2024-03-22T13:04:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.031+0000 7f3f34c00640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f1806bc50 0x7f3f1806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.031+0000 7f3f37e00640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f1806bc50 0x7f3f1806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:55.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.032+0000 7f3f37e00640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f1806bc50 0x7f3f1806e110 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3f3814e440 tx=0x7f3f24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:55.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.270+0000 7f3f34c00640 1 -- 172.21.15.6:0/2639816015 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3f2c079a60 con 0x7f3f3806b080 2024-03-22T13:04:55.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.290+0000 7f3f34c00640 1 -- 172.21.15.6:0/2639816015 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f2c03e070 con 0x7f3f3806b080 2024-03-22T13:04:55.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.545+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f38164850 con 0x7f3f3806b080 2024-03-22T13:04:55.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.609+0000 7f3f34c00640 1 -- 172.21.15.6:0/2639816015 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f3f2c07b9c0 con 0x7f3f3806b080 2024-03-22T13:04:55.610 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:55.611 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:55.611 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.613+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f1806bc50 msgr2=0x7f3f1806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.613+0000 7f3f3f400640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f1806bc50 0x7f3f1806e110 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3f3814e440 tx=0x7f3f24009040 comp rx=0 tx=0).stop 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.613+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3806b080 msgr2=0x7f3f38162860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.613+0000 7f3f3f400640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3806b080 0x7f3f38162860 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c02b9e0 tx=0x7f3f2c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 shutdown_connections 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f38150f50 0x7f3f381f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f380c5c20 0x7f3f380cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3f1806bc50 0x7f3f1806e110 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 --2- 172.21.15.6:0/2639816015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3806b080 0x7f3f38162860 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 >> 172.21.15.6:0/2639816015 conn(0x7f3f380cb930 msgr2=0x7f3f380744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 shutdown_connections 2024-03-22T13:04:55.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:55.614+0000 7f3f3f400640 1 -- 172.21.15.6:0/2639816015 wait complete. 2024-03-22T13:04:56.243 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:56.243 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:57.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.445+0000 7f4559000640 1 Processor -- start 2024-03-22T13:04:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.446+0000 7f4559000640 1 -- start start 2024-03-22T13:04:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.447+0000 7f4559000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f455406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.447+0000 7f4559000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f455415b0e0 0x7f455415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.448+0000 7f4559000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455406c7a0 con 0x7f45540ca520 2024-03-22T13:04:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.448+0000 7f4559000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45540cd000 con 0x7f455415b0e0 2024-03-22T13:04:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.448+0000 7f4559000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455406ca80 con 0x7f45540ccad0 2024-03-22T13:04:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.448+0000 7f4551400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f455406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.449+0000 7f4552800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f455415b0e0 0x7f455415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.449+0000 7f4551400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f455406d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50866/0 (socket says 172.21.15.6:50866) 2024-03-22T13:04:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.449+0000 7f4551400640 1 -- 172.21.15.6:0/697479012 learned_addr learned my addr 172.21.15.6:0/697479012 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:57.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.452+0000 7f4551400640 1 -- 172.21.15.6:0/697479012 >> v2:172.21.15.92:3300/0 conn(0x7f455415b0e0 msgr2=0x7f455415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:57.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.452+0000 7f4551400640 1 --2- 172.21.15.6:0/697479012 >> v2:172.21.15.92:3300/0 conn(0x7f455415b0e0 0x7f455415b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:57.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.452+0000 7f4551400640 1 -- 172.21.15.6:0/697479012 >> v1:172.21.15.92:6789/0 conn(0x7f45540ca520 legacy=0x7f45540cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:57.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.452+0000 7f4551400640 1 -- 172.21.15.6:0/697479012 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455406cd60 con 0x7f45540ccad0 2024-03-22T13:04:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.455+0000 7f4551400640 1 --2- 172.21.15.6:0/697479012 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f455406d840 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4540009870 tx=0x7f454002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=206d0ab62c00df2b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.459+0000 7f4550a00640 1 -- 172.21.15.6:0/697479012 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540034070 con 0x7f45540ccad0 2024-03-22T13:04:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.459+0000 7f4550a00640 1 -- 172.21.15.6:0/697479012 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4540030840 con 0x7f45540ccad0 2024-03-22T13:04:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.459+0000 7f4550a00640 1 -- 172.21.15.6:0/697479012 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540030c10 con 0x7f45540ccad0 2024-03-22T13:04:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.460+0000 7f4559000640 1 -- 172.21.15.6:0/697479012 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 msgr2=0x7f455406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.460+0000 7f4559000640 1 --2- 172.21.15.6:0/697479012 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f455406d840 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4540009870 tx=0x7f454002b980 comp rx=0 tx=0).stop 2024-03-22T13:04:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.461+0000 7f4559000640 1 -- 172.21.15.6:0/697479012 shutdown_connections 2024-03-22T13:04:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.461+0000 7f4559000640 1 --2- 172.21.15.6:0/697479012 >> v2:172.21.15.92:3300/0 conn(0x7f455415b0e0 0x7f455415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.461+0000 7f4559000640 1 --2- 172.21.15.6:0/697479012 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f455406d840 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.461+0000 7f4559000640 1 -- 172.21.15.6:0/697479012 >> 172.21.15.6:0/697479012 conn(0x7f45540b16b0 msgr2=0x7f45540b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.461+0000 7f4559000640 1 -- 172.21.15.6:0/697479012 shutdown_connections 2024-03-22T13:04:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.461+0000 7f4559000640 1 -- 172.21.15.6:0/697479012 wait complete. 2024-03-22T13:04:57.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.463+0000 7f4559000640 1 Processor -- start 2024-03-22T13:04:57.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.463+0000 7f4559000640 1 -- start start 2024-03-22T13:04:57.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.465+0000 7f4559000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45540ca520 0x7f45540b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:57.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.465+0000 7f4559000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f45541f8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:57.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.465+0000 7f4559000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f455415b0e0 0x7f45541f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:57.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.465+0000 7f4559000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554154c20 con 0x7f455415b0e0 2024-03-22T13:04:57.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.465+0000 7f4559000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455406d320 con 0x7f45540ca520 2024-03-22T13:04:57.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.465+0000 7f4559000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45540ca900 con 0x7f45540ccad0 2024-03-22T13:04:57.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.466+0000 7f4551400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f45541f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:57.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.466+0000 7f4551e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45540ca520 0x7f45540b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:57.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.466+0000 7f4552800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f455415b0e0 0x7f45541f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:57.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.466+0000 7f4551400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f45541f8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50876/0 (socket says 172.21.15.6:50876) 2024-03-22T13:04:57.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.466+0000 7f4551400640 1 -- 172.21.15.6:0/161754495 learned_addr learned my addr 172.21.15.6:0/161754495 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:57.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.470+0000 7f4551400640 1 -- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45540ca520 msgr2=0x7f45540b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:57.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.470+0000 7f4551400640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45540ca520 0x7f45540b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:57.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.470+0000 7f4551400640 1 -- 172.21.15.6:0/161754495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f455415b0e0 msgr2=0x7f45541f89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:57.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.470+0000 7f4551400640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f455415b0e0 0x7f45541f89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:57.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.470+0000 7f4551400640 1 -- 172.21.15.6:0/161754495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45541e4530 con 0x7f45540ccad0 2024-03-22T13:04:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.473+0000 7f4551400640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f45541f8490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4540009990 tx=0x7f4540004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:57.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.475+0000 7f454a000640 1 -- 172.21.15.6:0/161754495 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540048070 con 0x7f45540ccad0 2024-03-22T13:04:57.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.476+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45540c3480 con 0x7f45540ccad0 2024-03-22T13:04:57.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.476+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45540c3a40 con 0x7f45540ccad0 2024-03-22T13:04:57.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.478+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45540756c0 con 0x7f45540ccad0 2024-03-22T13:04:57.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.478+0000 7f454a000640 1 -- 172.21.15.6:0/161754495 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4540039d90 con 0x7f45540ccad0 2024-03-22T13:04:57.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.478+0000 7f454a000640 1 -- 172.21.15.6:0/161754495 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540052060 con 0x7f45540ccad0 2024-03-22T13:04:57.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.502+0000 7f454a000640 1 -- 172.21.15.6:0/161754495 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f454004b070 con 0x7f45540ccad0 2024-03-22T13:04:57.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.504+0000 7f454a000640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f453406bc00 0x7f453406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.504+0000 7f4551e00640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f453406bc00 0x7f453406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.505+0000 7f4551e00640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f453406bc00 0x7f453406e0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4544000f60 tx=0x7f4544009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:57.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.505+0000 7f454a000640 1 -- 172.21.15.6:0/161754495 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f45400797b0 con 0x7f45540ccad0 2024-03-22T13:04:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:57.781+0000 7f454a000640 1 -- 172.21.15.6:0/161754495 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f454007b710 con 0x7f45540ccad0 2024-03-22T13:04:58.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.074+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45540cb080 con 0x7f45540ccad0 2024-03-22T13:04:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.088+0000 7f454a000640 1 -- 172.21.15.6:0/161754495 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f454007f7b0 con 0x7f45540ccad0 2024-03-22T13:04:58.090 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:04:58.090 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:04:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:04:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f453406bc00 msgr2=0x7f453406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f453406bc00 0x7f453406e0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4544000f60 tx=0x7f4544009040 comp rx=0 tx=0).stop 2024-03-22T13:04:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 msgr2=0x7f45541f8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f45541f8490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4540009990 tx=0x7f4540004730 comp rx=0 tx=0).stop 2024-03-22T13:04:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 shutdown_connections 2024-03-22T13:04:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f455415b0e0 0x7f45541f89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45540ccad0 0x7f45541f8490 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f453406bc00 0x7f453406e0c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 --2- 172.21.15.6:0/161754495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45540ca520 0x7f45540b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 >> 172.21.15.6:0/161754495 conn(0x7f45540b16b0 msgr2=0x7f45540ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 shutdown_connections 2024-03-22T13:04:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:58.091+0000 7f4559000640 1 -- 172.21.15.6:0/161754495 wait complete. 2024-03-22T13:04:58.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:04:58.713 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:04:59.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:04:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.912+0000 7f2818600640 1 Processor -- start 2024-03-22T13:04:59.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.913+0000 7f2818600640 1 -- start start 2024-03-22T13:04:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.914+0000 7f2818600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28100ca520 0x7f281006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.914+0000 7f2818600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f281015b0e0 0x7f281015b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.914+0000 7f2818600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281006c7a0 con 0x7f28100ca900 2024-03-22T13:04:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.914+0000 7f2818600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28100cd000 con 0x7f28100ca520 2024-03-22T13:04:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.914+0000 7f2818600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281006ca80 con 0x7f281015b0e0 2024-03-22T13:04:59.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.915+0000 7f2815c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28100ca520 0x7f281006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.915+0000 7f2815200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f281015b0e0 0x7f281015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.915+0000 7f2815200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f281015b0e0 0x7f281015b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50890/0 (socket says 172.21.15.6:50890) 2024-03-22T13:04:59.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.915+0000 7f2815200640 1 -- 172.21.15.6:0/343209832 learned_addr learned my addr 172.21.15.6:0/343209832 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:59.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.918+0000 7f2815200640 1 -- 172.21.15.6:0/343209832 >> v2:172.21.15.6:3300/0 conn(0x7f28100ca520 msgr2=0x7f281006d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:04:59.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.918+0000 7f2815200640 1 --2- 172.21.15.6:0/343209832 >> v2:172.21.15.6:3300/0 conn(0x7f28100ca520 0x7f281006d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:59.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.918+0000 7f2815200640 1 -- 172.21.15.6:0/343209832 >> v1:172.21.15.6:6789/0 conn(0x7f28100ca900 legacy=0x7f28100cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:59.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.918+0000 7f2815200640 1 -- 172.21.15.6:0/343209832 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281006cd60 con 0x7f281015b0e0 2024-03-22T13:04:59.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.920+0000 7f2815200640 1 --2- 172.21.15.6:0/343209832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f281015b0e0 0x7f281015b980 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009870 tx=0x7f27fc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a034a4a740c16d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.924+0000 7f280fe00640 1 -- 172.21.15.6:0/343209832 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc034070 con 0x7f281015b0e0 2024-03-22T13:04:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.924+0000 7f280fe00640 1 -- 172.21.15.6:0/343209832 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc030840 con 0x7f281015b0e0 2024-03-22T13:04:59.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.924+0000 7f280fe00640 1 -- 172.21.15.6:0/343209832 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc030c10 con 0x7f281015b0e0 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 -- 172.21.15.6:0/343209832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f281015b0e0 msgr2=0x7f281015b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 --2- 172.21.15.6:0/343209832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f281015b0e0 0x7f281015b980 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009870 tx=0x7f27fc02b980 comp rx=0 tx=0).stop 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 -- 172.21.15.6:0/343209832 shutdown_connections 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 --2- 172.21.15.6:0/343209832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f281015b0e0 0x7f281015b980 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 --2- 172.21.15.6:0/343209832 >> v2:172.21.15.6:3300/0 conn(0x7f28100ca520 0x7f281006d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 -- 172.21.15.6:0/343209832 >> 172.21.15.6:0/343209832 conn(0x7f28100b16b0 msgr2=0x7f28100b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 -- 172.21.15.6:0/343209832 shutdown_connections 2024-03-22T13:04:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.925+0000 7f2818600640 1 -- 172.21.15.6:0/343209832 wait complete. 2024-03-22T13:04:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.927+0000 7f2818600640 1 Processor -- start 2024-03-22T13:04:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.927+0000 7f2818600640 1 -- start start 2024-03-22T13:04:59.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.928+0000 7f2818600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100ca520 0x7f2810159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:59.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.928+0000 7f2818600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28100ca900 0x7f28100b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:59.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.928+0000 7f2818600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f281015b0e0 0x7f2810153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:04:59.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.928+0000 7f2818600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100cad60 con 0x7f28100ca520 2024-03-22T13:04:59.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.928+0000 7f2818600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281006d320 con 0x7f281015b0e0 2024-03-22T13:04:59.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.928+0000 7f2818600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2810154c20 con 0x7f28100ca900 2024-03-22T13:04:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.929+0000 7f2815200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28100ca900 0x7f28100b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.929+0000 7f2815c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100ca520 0x7f2810159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.929+0000 7f2816600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f281015b0e0 0x7f2810153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:04:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.929+0000 7f2815200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28100ca900 0x7f28100b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50898/0 (socket says 172.21.15.6:50898) 2024-03-22T13:04:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.929+0000 7f2815200640 1 -- 172.21.15.6:0/2004156092 learned_addr learned my addr 172.21.15.6:0/2004156092 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:04:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.933+0000 7f2815200640 1 -- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f281015b0e0 msgr2=0x7f2810153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.933+0000 7f2815200640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f281015b0e0 0x7f2810153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.933+0000 7f2815200640 1 -- 172.21.15.6:0/2004156092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100ca520 msgr2=0x7f2810159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:04:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.933+0000 7f2815200640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100ca520 0x7f2810159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:04:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.933+0000 7f2815200640 1 -- 172.21.15.6:0/2004156092 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810158c50 con 0x7f28100ca900 2024-03-22T13:04:59.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.936+0000 7f2815200640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28100ca900 0x7f28100b2750 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27fc000c00 tx=0x7f27fc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:04:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.938+0000 7f280d600640 1 -- 172.21.15.6:0/2004156092 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc034040 con 0x7f28100ca900 2024-03-22T13:04:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.938+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28101f0980 con 0x7f28100ca900 2024-03-22T13:04:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.938+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28101f0f40 con 0x7f28100ca900 2024-03-22T13:04:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.940+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28100756c0 con 0x7f28100ca900 2024-03-22T13:04:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.941+0000 7f280d600640 1 -- 172.21.15.6:0/2004156092 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc039d10 con 0x7f28100ca900 2024-03-22T13:04:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:04:59.941+0000 7f280d600640 1 -- 172.21.15.6:0/2004156092 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc030840 con 0x7f28100ca900 2024-03-22T13:05:00.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.228+0000 7f280d600640 1 -- 172.21.15.6:0/2004156092 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f27fc037070 con 0x7f28100ca900 2024-03-22T13:05:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.230+0000 7f280d600640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f27f006bd70 0x7f27f006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.230+0000 7f2815c00640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f27f006bd70 0x7f27f006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.231+0000 7f280d600640 1 -- 172.21.15.6:0/2004156092 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f27fc079bf0 con 0x7f28100ca900 2024-03-22T13:05:00.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.231+0000 7f2815c00640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f27f006bd70 0x7f27f006e230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2804009aa0 tx=0x7f2804009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.249+0000 7f280d600640 1 -- 172.21.15.6:0/2004156092 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f27fc07bb50 con 0x7f28100ca900 2024-03-22T13:05:00.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.517+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28100b1490 con 0x7f28100ca900 2024-03-22T13:05:00.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.531+0000 7f280d600640 1 -- 172.21.15.6:0/2004156092 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f27fc07fbf0 con 0x7f28100ca900 2024-03-22T13:05:00.532 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:00.532 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:00.532 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f27f006bd70 msgr2=0x7f27f006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f27f006bd70 0x7f27f006e230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2804009aa0 tx=0x7f2804009190 comp rx=0 tx=0).stop 2024-03-22T13:05:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28100ca900 msgr2=0x7f28100b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28100ca900 0x7f28100b2750 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27fc000c00 tx=0x7f27fc0046c0 comp rx=0 tx=0).stop 2024-03-22T13:05:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 shutdown_connections 2024-03-22T13:05:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f27f006bd70 0x7f27f006e230 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:00.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f281015b0e0 0x7f2810153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28100ca900 0x7f28100b2750 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 --2- 172.21.15.6:0/2004156092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100ca520 0x7f2810159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 >> 172.21.15.6:0/2004156092 conn(0x7f28100b16b0 msgr2=0x7f28100ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 shutdown_connections 2024-03-22T13:05:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:00.533+0000 7f2818600640 1 -- 172.21.15.6:0/2004156092 wait complete. 2024-03-22T13:05:01.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:01.164 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:01.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:05:01.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:05:02.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:02.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.363+0000 7f389ca00640 1 Processor -- start 2024-03-22T13:05:02.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.363+0000 7f389ca00640 1 -- start start 2024-03-22T13:05:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.364+0000 7f389ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38980c2e50 0x7f38980c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.364+0000 7f389ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3898158010 con 0x7f389815b850 2024-03-22T13:05:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.364+0000 7f389ca00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38981582d0 con 0x7f38980c9630 2024-03-22T13:05:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.364+0000 7f389ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3898158ee0 con 0x7f38980c2e50 2024-03-22T13:05:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.365+0000 7f3895e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f38980c9630 0x7f38980c8a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:36224/0 (socket says 172.21.15.6:36224) 2024-03-22T13:05:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.365+0000 7f3895e00640 1 -- 172.21.15.6:0/2015207993 learned_addr learned my addr 172.21.15.6:0/2015207993 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.397+0000 7f3895400640 1 --2- 172.21.15.6:0/2015207993 >> v2:172.21.15.6:3300/0 conn(0x7f38980c2e50 0x7f38980c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.402+0000 7f3895400640 1 -- 172.21.15.6:0/2015207993 >> v1:172.21.15.92:6789/0 conn(0x7f38980c9630 legacy=0x7f38980c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.402+0000 7f3895400640 1 -- 172.21.15.6:0/2015207993 >> v1:172.21.15.6:6789/0 conn(0x7f389815b850 legacy=0x7f389815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.402+0000 7f3895400640 1 -- 172.21.15.6:0/2015207993 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38981585b0 con 0x7f38980c2e50 2024-03-22T13:05:02.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.409+0000 7f3895400640 1 --2- 172.21.15.6:0/2015207993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c2e50 0x7f38980c9a10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3898159060 tx=0x7f388402b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1a8515afc73984 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:02.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.415+0000 7f3894a00640 1 -- 172.21.15.6:0/2015207993 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884034070 con 0x7f38980c2e50 2024-03-22T13:05:02.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.415+0000 7f3894a00640 1 -- 172.21.15.6:0/2015207993 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3884030890 con 0x7f38980c2e50 2024-03-22T13:05:02.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.415+0000 7f3894a00640 1 -- 172.21.15.6:0/2015207993 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884030c60 con 0x7f38980c2e50 2024-03-22T13:05:02.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.416+0000 7f389ca00640 1 -- 172.21.15.6:0/2015207993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c2e50 msgr2=0x7f38980c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:02.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.416+0000 7f389ca00640 1 --2- 172.21.15.6:0/2015207993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c2e50 0x7f38980c9a10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3898159060 tx=0x7f388402b9d0 comp rx=0 tx=0).stop 2024-03-22T13:05:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.416+0000 7f389ca00640 1 -- 172.21.15.6:0/2015207993 shutdown_connections 2024-03-22T13:05:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.416+0000 7f389ca00640 1 --2- 172.21.15.6:0/2015207993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c2e50 0x7f38980c9a10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.416+0000 7f389ca00640 1 -- 172.21.15.6:0/2015207993 >> 172.21.15.6:0/2015207993 conn(0x7f38980714f0 msgr2=0x7f3898071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.416+0000 7f389ca00640 1 -- 172.21.15.6:0/2015207993 shutdown_connections 2024-03-22T13:05:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.416+0000 7f389ca00640 1 -- 172.21.15.6:0/2015207993 wait complete. 2024-03-22T13:05:02.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.419+0000 7f389ca00640 1 Processor -- start 2024-03-22T13:05:02.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.419+0000 7f389ca00640 1 -- start start 2024-03-22T13:05:02.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.420+0000 7f389ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38980c2e50 0x7f38980c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:02.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.420+0000 7f389ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c9630 0x7f389814f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:02.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.420+0000 7f389ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f389815b850 0x7f389814fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:02.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.420+0000 7f389ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980ca330 con 0x7f38980c9630 2024-03-22T13:05:02.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.420+0000 7f389ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3898158b70 con 0x7f389815b850 2024-03-22T13:05:02.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.420+0000 7f389ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38981525f0 con 0x7f38980c2e50 2024-03-22T13:05:02.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.421+0000 7f3895400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c9630 0x7f389814f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:02.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.422+0000 7f3895400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c9630 0x7f389814f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40892/0 (socket says 172.21.15.6:40892) 2024-03-22T13:05:02.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.422+0000 7f3895400640 1 -- 172.21.15.6:0/3235595225 learned_addr learned my addr 172.21.15.6:0/3235595225 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:02.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.425+0000 7f3895400640 1 -- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38980c2e50 msgr2=0x7f38980c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:02.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.425+0000 7f3895400640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38980c2e50 0x7f38980c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:02.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.425+0000 7f3895400640 1 -- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f389815b850 msgr2=0x7f389814fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.425+0000 7f3895400640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f389815b850 0x7f389814fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.425+0000 7f3895400640 1 -- 172.21.15.6:0/3235595225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980c55d0 con 0x7f38980c9630 2024-03-22T13:05:02.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.428+0000 7f3895400640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c9630 0x7f389814f570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f388402bfe0 tx=0x7f38840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.432+0000 7f388e000640 1 -- 172.21.15.6:0/3235595225 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884046020 con 0x7f38980c9630 2024-03-22T13:05:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.432+0000 7f388e000640 1 -- 172.21.15.6:0/3235595225 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3884039d10 con 0x7f38980c9630 2024-03-22T13:05:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.432+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38980c7e80 con 0x7f38980c9630 2024-03-22T13:05:02.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.432+0000 7f388e000640 1 -- 172.21.15.6:0/3235595225 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884034040 con 0x7f38980c9630 2024-03-22T13:05:02.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.432+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3898152d00 con 0x7f38980c9630 2024-03-22T13:05:02.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.434+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38980b2e60 con 0x7f38980c9630 2024-03-22T13:05:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.464+0000 7f388e000640 1 -- 172.21.15.6:0/3235595225 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3884037070 con 0x7f38980c9630 2024-03-22T13:05:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.465+0000 7f388e000640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f387806bc20 0x7f387806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.466+0000 7f3895e00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f387806bc20 0x7f387806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.467+0000 7f3895e00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f387806bc20 0x7f387806e0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f38880015f0 tx=0x7f3888007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:02.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.813+0000 7f388e000640 1 -- 172.21.15.6:0/3235595225 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f388407a0b0 con 0x7f38980c9630 2024-03-22T13:05:02.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:02.814+0000 7f388e000640 1 -- 172.21.15.6:0/3235595225 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38980b2e60 con 0x7f38980c9630 2024-03-22T13:05:03.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.078+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38980cbce0 con 0x7f38980c9630 2024-03-22T13:05:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.093+0000 7f388e000640 1 -- 172.21.15.6:0/3235595225 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f38980cbce0 con 0x7f38980c9630 2024-03-22T13:05:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:03.093 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:03.093 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:03.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f387806bc20 msgr2=0x7f387806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f387806bc20 0x7f387806e0e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f38880015f0 tx=0x7f3888007a40 comp rx=0 tx=0).stop 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c9630 msgr2=0x7f389814f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c9630 0x7f389814f570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f388402bfe0 tx=0x7f38840046c0 comp rx=0 tx=0).stop 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 shutdown_connections 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f387806bc20 0x7f387806e0e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f389815b850 0x7f389814fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38980c9630 0x7f389814f570 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 --2- 172.21.15.6:0/3235595225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38980c2e50 0x7f38980c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 >> 172.21.15.6:0/3235595225 conn(0x7f38980714f0 msgr2=0x7f38980b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.095+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 shutdown_connections 2024-03-22T13:05:03.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:03.096+0000 7f389ca00640 1 -- 172.21.15.6:0/3235595225 wait complete. 2024-03-22T13:05:03.715 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:03.715 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:04.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.903+0000 7f1a60a00640 1 Processor -- start 2024-03-22T13:05:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.903+0000 7f1a60a00640 1 -- start start 2024-03-22T13:05:04.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.905+0000 7f1a60a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a5c1547d0 0x7f1a5c0c4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.905+0000 7f1a60a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a5c0cdc30 0x7f1a5c0ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.905+0000 7f1a60a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0ca970 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.905+0000 7f1a60a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c8930 con 0x7f1a5c0cdc30 2024-03-22T13:05:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.905+0000 7f1a60a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c8b80 con 0x7f1a5c1547d0 2024-03-22T13:05:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.906+0000 7f1a59e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a5c1547d0 0x7f1a5c0c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.906+0000 7f1a5a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1a5c0cdc30 0x7f1a5c0ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.906+0000 7f1a59e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a5c1547d0 0x7f1a5c0c4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47846/0 (socket says 172.21.15.6:47846) 2024-03-22T13:05:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.906+0000 7f1a59e00640 1 -- 172.21.15.6:0/258946882 learned_addr learned my addr 172.21.15.6:0/258946882 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.909+0000 7f1a59e00640 1 -- 172.21.15.6:0/258946882 >> v2:172.21.15.6:3300/0 conn(0x7f1a5c0cdc30 msgr2=0x7f1a5c0ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.909+0000 7f1a59e00640 1 --2- 172.21.15.6:0/258946882 >> v2:172.21.15.6:3300/0 conn(0x7f1a5c0cdc30 0x7f1a5c0ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.909+0000 7f1a59e00640 1 -- 172.21.15.6:0/258946882 >> v1:172.21.15.6:6789/0 conn(0x7f1a5c0c98d0 legacy=0x7f1a5c0ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.909+0000 7f1a59e00640 1 -- 172.21.15.6:0/258946882 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c15a5d0 con 0x7f1a5c1547d0 2024-03-22T13:05:04.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.912+0000 7f1a59e00640 1 --2- 172.21.15.6:0/258946882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c1547d0 0x7f1a5c0c4600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c071f60 tx=0x7f1a4c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c90107a8a13acee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:04.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.915+0000 7f1a58a00640 1 -- 172.21.15.6:0/258946882 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c034070 con 0x7f1a5c1547d0 2024-03-22T13:05:04.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.917+0000 7f1a58a00640 1 -- 172.21.15.6:0/258946882 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c030860 con 0x7f1a5c1547d0 2024-03-22T13:05:04.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.917+0000 7f1a58a00640 1 -- 172.21.15.6:0/258946882 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c030c30 con 0x7f1a5c1547d0 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 -- 172.21.15.6:0/258946882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c1547d0 msgr2=0x7f1a5c0c4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 --2- 172.21.15.6:0/258946882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c1547d0 0x7f1a5c0c4600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c071f60 tx=0x7f1a4c02b9a0 comp rx=0 tx=0).stop 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 -- 172.21.15.6:0/258946882 shutdown_connections 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 --2- 172.21.15.6:0/258946882 >> v2:172.21.15.6:3300/0 conn(0x7f1a5c0cdc30 0x7f1a5c0ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 --2- 172.21.15.6:0/258946882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c1547d0 0x7f1a5c0c4600 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 -- 172.21.15.6:0/258946882 >> 172.21.15.6:0/258946882 conn(0x7f1a5c071420 msgr2=0x7f1a5c071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 -- 172.21.15.6:0/258946882 shutdown_connections 2024-03-22T13:05:04.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.918+0000 7f1a60a00640 1 -- 172.21.15.6:0/258946882 wait complete. 2024-03-22T13:05:04.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.920+0000 7f1a60a00640 1 Processor -- start 2024-03-22T13:05:04.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.920+0000 7f1a60a00640 1 -- start start 2024-03-22T13:05:04.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.921+0000 7f1a60a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a5c0c98d0 0x7f1a5c15da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:04.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.921+0000 7f1a60a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c0cdc30 0x7f1a5c15df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:04.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.921+0000 7f1a60a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a5c1547d0 0x7f1a5c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:04.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.921+0000 7f1a60a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c15ae70 con 0x7f1a5c1547d0 2024-03-22T13:05:04.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.921+0000 7f1a60a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c15ab90 con 0x7f1a5c0cdc30 2024-03-22T13:05:04.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.921+0000 7f1a60a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c154bb0 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.922+0000 7f1a5a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a5c1547d0 0x7f1a5c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:04.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.922+0000 7f1a59e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a5c0c98d0 0x7f1a5c15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:04.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.922+0000 7f1a59e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a5c0c98d0 0x7f1a5c15da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51554/0 (socket says 172.21.15.6:51554) 2024-03-22T13:05:04.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.922+0000 7f1a59e00640 1 -- 172.21.15.6:0/868788890 learned_addr learned my addr 172.21.15.6:0/868788890 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:04.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.922+0000 7f1a59400640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c0cdc30 0x7f1a5c15df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:04.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.926+0000 7f1a59e00640 1 -- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c0cdc30 msgr2=0x7f1a5c15df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:04.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.926+0000 7f1a59e00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c0cdc30 0x7f1a5c15df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:04.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.926+0000 7f1a59e00640 1 -- 172.21.15.6:0/868788890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a5c1547d0 msgr2=0x7f1a5c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:04.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.926+0000 7f1a59e00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a5c1547d0 0x7f1a5c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:04.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.926+0000 7f1a59e00640 1 -- 172.21.15.6:0/868788890 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c15b680 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.928+0000 7f1a59e00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a5c0c98d0 0x7f1a5c15da00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c000c00 tx=0x7f1a4c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:04.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.933+0000 7f1a52000640 1 -- 172.21.15.6:0/868788890 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c048070 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.933+0000 7f1a52000640 1 -- 172.21.15.6:0/868788890 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c030660 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.933+0000 7f1a52000640 1 -- 172.21.15.6:0/868788890 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c034040 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.933+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0c7d20 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.933+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c0c5f50 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.935+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c159fe0 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.967+0000 7f1a52000640 1 -- 172.21.15.6:0/868788890 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a4c04b070 con 0x7f1a5c0c98d0 2024-03-22T13:05:04.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.969+0000 7f1a52000640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a3c06bc50 0x7f1a3c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:04.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.969+0000 7f1a59400640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a3c06bc50 0x7f1a3c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.970+0000 7f1a59400640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a3c06bc50 0x7f1a3c06e110 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c157e40 tx=0x7f1a48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:04.970+0000 7f1a52000640 1 -- 172.21.15.6:0/868788890 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1a4c030810 con 0x7f1a5c0c98d0 2024-03-22T13:05:05.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.221+0000 7f1a52000640 1 -- 172.21.15.6:0/868788890 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a5c159fe0 con 0x7f1a5c0c98d0 2024-03-22T13:05:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.472+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5c0756c0 con 0x7f1a5c0c98d0 2024-03-22T13:05:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.566+0000 7f1a52000640 1 -- 172.21.15.6:0/868788890 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f1a4c052020 con 0x7f1a5c0c98d0 2024-03-22T13:05:05.566 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:05.566 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.569+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a3c06bc50 msgr2=0x7f1a3c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.569+0000 7f1a60a00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a3c06bc50 0x7f1a3c06e110 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c157e40 tx=0x7f1a48009040 comp rx=0 tx=0).stop 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.569+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a5c0c98d0 msgr2=0x7f1a5c15da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.569+0000 7f1a60a00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a5c0c98d0 0x7f1a5c15da00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c000c00 tx=0x7f1a4c004730 comp rx=0 tx=0).stop 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 shutdown_connections 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a5c1547d0 0x7f1a5c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1a3c06bc50 0x7f1a3c06e110 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a5c0cdc30 0x7f1a5c15df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 --2- 172.21.15.6:0/868788890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a5c0c98d0 0x7f1a5c15da00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 >> 172.21.15.6:0/868788890 conn(0x7f1a5c071420 msgr2=0x7f1a5c0744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 shutdown_connections 2024-03-22T13:05:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:05.570+0000 7f1a60a00640 1 -- 172.21.15.6:0/868788890 wait complete. 2024-03-22T13:05:06.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:06.197 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:07.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.383+0000 7fb6b4200640 1 Processor -- start 2024-03-22T13:05:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.383+0000 7fb6b4200640 1 -- start start 2024-03-22T13:05:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.384+0000 7fb6b4200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c5040 0x7fb6ac0c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.384+0000 7fb6b4200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac1590b0 con 0x7fb6ac06f8c0 2024-03-22T13:05:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.384+0000 7fb6b4200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac0721f0 con 0x7fb6ac152400 2024-03-22T13:05:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.384+0000 7fb6b4200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac0724d0 con 0x7fb6ac0c5040 2024-03-22T13:05:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.385+0000 7fb6b2200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c5040 0x7fb6ac0c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.385+0000 7fb6b0e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb6ac06f8c0 0x7fb6ac0716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58492/0 (socket says 172.21.15.6:58492) 2024-03-22T13:05:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.385+0000 7fb6b0e00640 1 -- 172.21.15.6:0/408327071 learned_addr learned my addr 172.21.15.6:0/408327071 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.388+0000 7fb6b2200640 1 -- 172.21.15.6:0/408327071 >> v1:172.21.15.92:6789/0 conn(0x7fb6ac152400 legacy=0x7fb6ac15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.388+0000 7fb6b2200640 1 -- 172.21.15.6:0/408327071 >> v1:172.21.15.6:6789/0 conn(0x7fb6ac06f8c0 legacy=0x7fb6ac0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.388+0000 7fb6b2200640 1 -- 172.21.15.6:0/408327071 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac1582f0 con 0x7fb6ac0c5040 2024-03-22T13:05:07.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.390+0000 7fb6b2200640 1 --2- 172.21.15.6:0/408327071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c5040 0x7fb6ac0c5450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40097d0 tx=0x7fb6a402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5468d94a0db4545 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.393+0000 7fb6abe00640 1 -- 172.21.15.6:0/408327071 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4034070 con 0x7fb6ac0c5040 2024-03-22T13:05:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.395+0000 7fb6abe00640 1 -- 172.21.15.6:0/408327071 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a4030880 con 0x7fb6ac0c5040 2024-03-22T13:05:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.395+0000 7fb6abe00640 1 -- 172.21.15.6:0/408327071 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4030c50 con 0x7fb6ac0c5040 2024-03-22T13:05:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.396+0000 7fb6b4200640 1 -- 172.21.15.6:0/408327071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c5040 msgr2=0x7fb6ac0c5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.396+0000 7fb6b4200640 1 --2- 172.21.15.6:0/408327071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c5040 0x7fb6ac0c5450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40097d0 tx=0x7fb6a402b9c0 comp rx=0 tx=0).stop 2024-03-22T13:05:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.396+0000 7fb6b4200640 1 -- 172.21.15.6:0/408327071 shutdown_connections 2024-03-22T13:05:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.396+0000 7fb6b4200640 1 --2- 172.21.15.6:0/408327071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c5040 0x7fb6ac0c5450 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.396+0000 7fb6b4200640 1 -- 172.21.15.6:0/408327071 >> 172.21.15.6:0/408327071 conn(0x7fb6ac0cb930 msgr2=0x7fb6ac0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:07.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.397+0000 7fb6b4200640 1 -- 172.21.15.6:0/408327071 shutdown_connections 2024-03-22T13:05:07.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.397+0000 7fb6b4200640 1 -- 172.21.15.6:0/408327071 wait complete. 2024-03-22T13:05:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.399+0000 7fb6b4200640 1 Processor -- start 2024-03-22T13:05:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.399+0000 7fb6b4200640 1 -- start start 2024-03-22T13:05:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.401+0000 7fb6b4200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac06f8c0 0x7fb6ac14c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.401+0000 7fb6b4200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c5040 0x7fb6ac0caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.401+0000 7fb6b4200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac152400 0x7fb6ac13d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.401+0000 7fb6b4200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac158b90 con 0x7fb6ac06f8c0 2024-03-22T13:05:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.401+0000 7fb6b4200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1588b0 con 0x7fb6ac0c5040 2024-03-22T13:05:07.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.401+0000 7fb6b4200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1527e0 con 0x7fb6ac152400 2024-03-22T13:05:07.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.402+0000 7fb6b2200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac152400 0x7fb6ac13d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:07.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.402+0000 7fb6b2200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac152400 0x7fb6ac13d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51564/0 (socket says 172.21.15.6:51564) 2024-03-22T13:05:07.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.402+0000 7fb6b2200640 1 -- 172.21.15.6:0/254544146 learned_addr learned my addr 172.21.15.6:0/254544146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:07.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.402+0000 7fb6b1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac06f8c0 0x7fb6ac14c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:07.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.402+0000 7fb6b0e00640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c5040 0x7fb6ac0caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.406+0000 7fb6b2200640 1 -- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c5040 msgr2=0x7fb6ac0caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.406+0000 7fb6b2200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c5040 0x7fb6ac0caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.406+0000 7fb6b2200640 1 -- 172.21.15.6:0/254544146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac06f8c0 msgr2=0x7fb6ac14c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.406+0000 7fb6b2200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac06f8c0 0x7fb6ac14c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.406+0000 7fb6b2200640 1 -- 172.21.15.6:0/254544146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac1edeb0 con 0x7fb6ac152400 2024-03-22T13:05:07.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.409+0000 7fb6b2200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac152400 0x7fb6ac13d080 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40098f0 tx=0x7fb6a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:07.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.413+0000 7fb6a9600640 1 -- 172.21.15.6:0/254544146 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4034040 con 0x7fb6ac152400 2024-03-22T13:05:07.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.414+0000 7fb6a9600640 1 -- 172.21.15.6:0/254544146 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a40306a0 con 0x7fb6ac152400 2024-03-22T13:05:07.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.414+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac152ff0 con 0x7fb6ac152400 2024-03-22T13:05:07.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.414+0000 7fb6a9600640 1 -- 172.21.15.6:0/254544146 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a40423f0 con 0x7fb6ac152400 2024-03-22T13:05:07.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.414+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac1ee170 con 0x7fb6ac152400 2024-03-22T13:05:07.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.415+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac0756c0 con 0x7fb6ac152400 2024-03-22T13:05:07.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.440+0000 7fb6a9600640 1 -- 172.21.15.6:0/254544146 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb6a4037070 con 0x7fb6ac152400 2024-03-22T13:05:07.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.443+0000 7fb6a9600640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:07.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.444+0000 7fb6b1800640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:07.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.445+0000 7fb6b1800640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00012e0 tx=0x7fb6a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:07.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.686+0000 7fb6a9600640 1 -- 172.21.15.6:0/254544146 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb6a4079e10 con 0x7fb6ac152400 2024-03-22T13:05:07.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.705+0000 7fb6a9600640 1 -- 172.21.15.6:0/254544146 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb6a4030850 con 0x7fb6ac152400 2024-03-22T13:05:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.976+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6ac147e70 con 0x7fb6ac152400 2024-03-22T13:05:07.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:07.999+0000 7fb6a9600640 1 -- 172.21.15.6:0/254544146 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fb6a407bd70 con 0x7fb6ac152400 2024-03-22T13:05:07.999 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:07.999 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:07.999 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.001+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 msgr2=0x7fb68c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.001+0000 7fb6b4200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00012e0 tx=0x7fb6a0009040 comp rx=0 tx=0).stop 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac152400 msgr2=0x7fb6ac13d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac152400 0x7fb6ac13d080 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40098f0 tx=0x7fb6a40046c0 comp rx=0 tx=0).stop 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 shutdown_connections 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac152400 0x7fb6ac13d080 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:08.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c5040 0x7fb6ac0caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 --2- 172.21.15.6:0/254544146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac06f8c0 0x7fb6ac14c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 >> 172.21.15.6:0/254544146 conn(0x7fb6ac0cb930 msgr2=0x7fb6ac0715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 shutdown_connections 2024-03-22T13:05:08.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:08.002+0000 7fb6b4200640 1 -- 172.21.15.6:0/254544146 wait complete. 2024-03-22T13:05:08.624 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:08.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:09.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:09.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.828+0000 7f35f1600640 1 Processor -- start 2024-03-22T13:05:09.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.829+0000 7f35f1600640 1 -- start start 2024-03-22T13:05:09.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.830+0000 7f35f1600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f35ec15af60 0x7f35ec0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:09.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.830+0000 7f35f1600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35ec0cc860 0x7f35ec06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:09.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.830+0000 7f35f1600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:09.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.830+0000 7f35f1600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec0cc3d0 con 0x7f35ec0cc860 2024-03-22T13:05:09.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.830+0000 7f35f1600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec0ce380 con 0x7f35ec15af60 2024-03-22T13:05:09.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.830+0000 7f35f1600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec0ce660 con 0x7f35ec0ccca0 2024-03-22T13:05:09.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.838+0000 7f35f0c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.838+0000 7f35f0c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec0cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51576/0 (socket says 172.21.15.6:51576) 2024-03-22T13:05:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.838+0000 7f35f0c00640 1 -- 172.21.15.6:0/2955117970 learned_addr learned my addr 172.21.15.6:0/2955117970 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:09.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.842+0000 7f35f0c00640 1 -- 172.21.15.6:0/2955117970 >> v2:172.21.15.92:3300/0 conn(0x7f35ec15af60 msgr2=0x7f35ec0cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:09.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.842+0000 7f35f0c00640 1 --2- 172.21.15.6:0/2955117970 >> v2:172.21.15.92:3300/0 conn(0x7f35ec15af60 0x7f35ec0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:09.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.842+0000 7f35f0c00640 1 -- 172.21.15.6:0/2955117970 >> v2:172.21.15.6:3300/0 conn(0x7f35ec0cc860 msgr2=0x7f35ec06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:09.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.842+0000 7f35f0c00640 1 --2- 172.21.15.6:0/2955117970 >> v2:172.21.15.6:3300/0 conn(0x7f35ec0cc860 0x7f35ec06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:09.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.842+0000 7f35f0c00640 1 -- 172.21.15.6:0/2955117970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0ce940 con 0x7f35ec0ccca0 2024-03-22T13:05:09.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.845+0000 7f35f0c00640 1 --2- 172.21.15.6:0/2955117970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec0cec20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f35ec08cbc0 tx=0x7f35e402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0858aebbcb46303 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:09.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.850+0000 7f35eaa00640 1 -- 172.21.15.6:0/2955117970 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e402cb00 con 0x7f35ec0ccca0 2024-03-22T13:05:09.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.850+0000 7f35eaa00640 1 -- 172.21.15.6:0/2955117970 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e402cca0 con 0x7f35ec0ccca0 2024-03-22T13:05:09.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.850+0000 7f35eaa00640 1 -- 172.21.15.6:0/2955117970 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4034610 con 0x7f35ec0ccca0 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 -- 172.21.15.6:0/2955117970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 msgr2=0x7f35ec0cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 --2- 172.21.15.6:0/2955117970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec0cec20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f35ec08cbc0 tx=0x7f35e402bf60 comp rx=0 tx=0).stop 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 -- 172.21.15.6:0/2955117970 shutdown_connections 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 --2- 172.21.15.6:0/2955117970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec0cec20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 --2- 172.21.15.6:0/2955117970 >> v2:172.21.15.6:3300/0 conn(0x7f35ec0cc860 0x7f35ec06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 --2- 172.21.15.6:0/2955117970 >> v2:172.21.15.92:3300/0 conn(0x7f35ec15af60 0x7f35ec0cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 -- 172.21.15.6:0/2955117970 >> 172.21.15.6:0/2955117970 conn(0x7f35ec0cf9d0 msgr2=0x7f35ec0cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 -- 172.21.15.6:0/2955117970 shutdown_connections 2024-03-22T13:05:09.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.851+0000 7f35f1600640 1 -- 172.21.15.6:0/2955117970 wait complete. 2024-03-22T13:05:09.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.854+0000 7f35f1600640 1 Processor -- start 2024-03-22T13:05:09.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.854+0000 7f35f1600640 1 -- start start 2024-03-22T13:05:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.855+0000 7f35f1600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35ec0cc860 0x7f35ec151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.855+0000 7f35f1600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.855+0000 7f35f1600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35ec15af60 0x7f35ec0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.855+0000 7f35f1600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec06dd60 con 0x7f35ec0cc860 2024-03-22T13:05:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.855+0000 7f35f1600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec15a590 con 0x7f35ec15af60 2024-03-22T13:05:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.855+0000 7f35f1600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec06dee0 con 0x7f35ec0ccca0 2024-03-22T13:05:09.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.856+0000 7f35eb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.856+0000 7f35eb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec14fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51592/0 (socket says 172.21.15.6:51592) 2024-03-22T13:05:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.856+0000 7f35eb400640 1 -- 172.21.15.6:0/1196030889 learned_addr learned my addr 172.21.15.6:0/1196030889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.857+0000 7f35ebe00640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35ec0cc860 0x7f35ec151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.860+0000 7f35eb400640 1 -- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35ec15af60 msgr2=0x7f35ec0af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.860+0000 7f35eb400640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35ec15af60 0x7f35ec0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.860+0000 7f35eb400640 1 -- 172.21.15.6:0/1196030889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35ec0cc860 msgr2=0x7f35ec151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.860+0000 7f35eb400640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35ec0cc860 0x7f35ec151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.860+0000 7f35eb400640 1 -- 172.21.15.6:0/1196030889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0b0fc0 con 0x7f35ec0ccca0 2024-03-22T13:05:09.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.864+0000 7f35eb400640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec14fcf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f35d800b4a0 tx=0x7f35d800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:09.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.869+0000 7f35e3e00640 1 -- 172.21.15.6:0/1196030889 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8014070 con 0x7f35ec0ccca0 2024-03-22T13:05:09.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.869+0000 7f35e3e00640 1 -- 172.21.15.6:0/1196030889 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d8010610 con 0x7f35ec0ccca0 2024-03-22T13:05:09.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.869+0000 7f35e3e00640 1 -- 172.21.15.6:0/1196030889 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8010a00 con 0x7f35ec0ccca0 2024-03-22T13:05:09.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.869+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec150230 con 0x7f35ec0ccca0 2024-03-22T13:05:09.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.870+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec1e6ba0 con 0x7f35ec0ccca0 2024-03-22T13:05:09.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.871+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec075650 con 0x7f35ec0ccca0 2024-03-22T13:05:09.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.897+0000 7f35e3e00640 1 -- 172.21.15.6:0/1196030889 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35d8017070 con 0x7f35ec0ccca0 2024-03-22T13:05:09.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.898+0000 7f35e3e00640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35cc06bc20 0x7f35cc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:09.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.899+0000 7f35ebe00640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35cc06bc20 0x7f35cc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:09.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.899+0000 7f35e3e00640 1 -- 172.21.15.6:0/1196030889 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f35d8056d60 con 0x7f35ec0ccca0 2024-03-22T13:05:09.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:09.899+0000 7f35ebe00640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35cc06bc20 0x7f35cc06e0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0098e0 tx=0x7f35dc009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:10.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.174+0000 7f35e3e00640 1 -- 172.21.15.6:0/1196030889 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35d8058cc0 con 0x7f35ec0ccca0 2024-03-22T13:05:10.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.444+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35ec0b5950 con 0x7f35ec0ccca0 2024-03-22T13:05:10.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.466+0000 7f35e3e00640 1 -- 172.21.15.6:0/1196030889 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f35d805cd60 con 0x7f35ec0ccca0 2024-03-22T13:05:10.466 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:10.467 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35cc06bc20 msgr2=0x7f35cc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35cc06bc20 0x7f35cc06e0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0098e0 tx=0x7f35dc009060 comp rx=0 tx=0).stop 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 msgr2=0x7f35ec14fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec14fcf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f35d800b4a0 tx=0x7f35d800b960 comp rx=0 tx=0).stop 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 shutdown_connections 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35cc06bc20 0x7f35cc06e0e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35ec15af60 0x7f35ec0af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35ec0ccca0 0x7f35ec14fcf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 --2- 172.21.15.6:0/1196030889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35ec0cc860 0x7f35ec151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 >> 172.21.15.6:0/1196030889 conn(0x7f35ec0cf9d0 msgr2=0x7f35ec0ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 shutdown_connections 2024-03-22T13:05:10.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:10.469+0000 7f35f1600640 1 -- 172.21.15.6:0/1196030889 wait complete. 2024-03-22T13:05:11.092 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:11.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:12.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.294+0000 7fd0af200640 1 Processor -- start 2024-03-22T13:05:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.294+0000 7fd0af200640 1 -- start start 2024-03-22T13:05:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.295+0000 7fd0af200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd0a8152400 0x7fd0a815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.296+0000 7fd0af200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a81590b0 con 0x7fd0a815bc80 2024-03-22T13:05:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.296+0000 7fd0af200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80721f0 con 0x7fd0a806f8c0 2024-03-22T13:05:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.296+0000 7fd0af200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a80724d0 con 0x7fd0a8152400 2024-03-22T13:05:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.296+0000 7fd0a7e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd0a8152400 0x7fd0a815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.297+0000 7fd09f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd0a815bc80 0x7fd0a80716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58496/0 (socket says 172.21.15.6:58496) 2024-03-22T13:05:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.297+0000 7fd09f400640 1 -- 172.21.15.6:0/3900964693 learned_addr learned my addr 172.21.15.6:0/3900964693 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.300+0000 7fd0a7400640 1 -- 172.21.15.6:0/3900964693 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790928044 0 0) 0x7fd0a81590b0 con 0x7fd0a815bc80 2024-03-22T13:05:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.300+0000 7fd0a7400640 1 -- 172.21.15.6:0/3900964693 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd08c003520 con 0x7fd0a815bc80 2024-03-22T13:05:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.301+0000 7fd0a7e00640 1 -- 172.21.15.6:0/3900964693 >> v1:172.21.15.92:6789/0 conn(0x7fd0a806f8c0 legacy=0x7fd0a80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:12.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.301+0000 7fd0a7e00640 1 -- 172.21.15.6:0/3900964693 >> v1:172.21.15.6:6789/0 conn(0x7fd0a815bc80 legacy=0x7fd0a80716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:12.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.301+0000 7fd0a7e00640 1 -- 172.21.15.6:0/3900964693 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a81582f0 con 0x7fd0a8152400 2024-03-22T13:05:12.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.307+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/3900964693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a8152400 0x7fd0a815b740 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd098009d90 tx=0x7fd09802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fe06d338ec18f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.313+0000 7fd0a7400640 1 -- 172.21.15.6:0/3900964693 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098039070 con 0x7fd0a8152400 2024-03-22T13:05:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.313+0000 7fd0a7400640 1 -- 172.21.15.6:0/3900964693 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09802cd80 con 0x7fd0a8152400 2024-03-22T13:05:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.313+0000 7fd0a7400640 1 -- 172.21.15.6:0/3900964693 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098034660 con 0x7fd0a8152400 2024-03-22T13:05:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.314+0000 7fd0af200640 1 -- 172.21.15.6:0/3900964693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a8152400 msgr2=0x7fd0a815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.314+0000 7fd0af200640 1 --2- 172.21.15.6:0/3900964693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a8152400 0x7fd0a815b740 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd098009d90 tx=0x7fd09802bfd0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.314+0000 7fd0af200640 1 -- 172.21.15.6:0/3900964693 shutdown_connections 2024-03-22T13:05:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.314+0000 7fd0af200640 1 --2- 172.21.15.6:0/3900964693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a8152400 0x7fd0a815b740 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.314+0000 7fd0af200640 1 -- 172.21.15.6:0/3900964693 >> 172.21.15.6:0/3900964693 conn(0x7fd0a80cb930 msgr2=0x7fd0a80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.314+0000 7fd0af200640 1 -- 172.21.15.6:0/3900964693 shutdown_connections 2024-03-22T13:05:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.314+0000 7fd0af200640 1 -- 172.21.15.6:0/3900964693 wait complete. 2024-03-22T13:05:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.316+0000 7fd0af200640 1 Processor -- start 2024-03-22T13:05:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.316+0000 7fd0af200640 1 -- start start 2024-03-22T13:05:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.317+0000 7fd0af200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a806f8c0 0x7fd0a81ec170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.317+0000 7fd0af200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a8152400 0x7fd0a80caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.317+0000 7fd0af200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0a815bc80 0x7fd0a8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.317+0000 7fd0af200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a8158b90 con 0x7fd0a8152400 2024-03-22T13:05:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.317+0000 7fd0af200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a81588b0 con 0x7fd0a806f8c0 2024-03-22T13:05:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.317+0000 7fd0af200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a81527e0 con 0x7fd0a815bc80 2024-03-22T13:05:12.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.318+0000 7fd0a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a806f8c0 0x7fd0a81ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:12.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.318+0000 7fd09f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a8152400 0x7fd0a80caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:12.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.318+0000 7fd0ad200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0a815bc80 0x7fd0a8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:12.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.318+0000 7fd0a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a806f8c0 0x7fd0a81ec170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47914/0 (socket says 172.21.15.6:47914) 2024-03-22T13:05:12.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.318+0000 7fd0a7e00640 1 -- 172.21.15.6:0/1114830452 learned_addr learned my addr 172.21.15.6:0/1114830452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.322+0000 7fd09f400640 1 -- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0a815bc80 msgr2=0x7fd0a8159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.322+0000 7fd09f400640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0a815bc80 0x7fd0a8159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.322+0000 7fd09f400640 1 -- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a806f8c0 msgr2=0x7fd0a81ec170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.322+0000 7fd09f400640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a806f8c0 0x7fd0a81ec170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.322+0000 7fd09f400640 1 -- 172.21.15.6:0/1114830452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80c4660 con 0x7fd0a8152400 2024-03-22T13:05:12.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.325+0000 7fd09f400640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a8152400 0x7fd0a80caf90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8071680 tx=0x7fd090001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:12.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.329+0000 7fd0a4c00640 1 -- 172.21.15.6:0/1114830452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090017070 con 0x7fd0a8152400 2024-03-22T13:05:12.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.329+0000 7fd0a4c00640 1 -- 172.21.15.6:0/1114830452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090005ce0 con 0x7fd0a8152400 2024-03-22T13:05:12.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.329+0000 7fd0a4c00640 1 -- 172.21.15.6:0/1114830452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0900072f0 con 0x7fd0a8152400 2024-03-22T13:05:12.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.329+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a81ec6b0 con 0x7fd0a8152400 2024-03-22T13:05:12.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.329+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a80c4c00 con 0x7fd0a8152400 2024-03-22T13:05:12.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.330+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a80c4dd0 con 0x7fd0a8152400 2024-03-22T13:05:12.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.634+0000 7fd0a4c00640 1 -- 172.21.15.6:0/1114830452 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd09001a070 con 0x7fd0a8152400 2024-03-22T13:05:12.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.636+0000 7fd0a4c00640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd08406bc20 0x7fd08406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:12.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.637+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd08406bc20 0x7fd08406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:12.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.638+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd08406bc20 0x7fd08406e0e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd098009bd0 tx=0x7fd098030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:12.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.673+0000 7fd0a4c00640 1 -- 172.21.15.6:0/1114830452 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd090056620 con 0x7fd0a8152400 2024-03-22T13:05:12.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.714+0000 7fd0a4c00640 1 -- 172.21.15.6:0/1114830452 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd090021020 con 0x7fd0a8152400 2024-03-22T13:05:12.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.969+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0a80756c0 con 0x7fd0a8152400 2024-03-22T13:05:12.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.985+0000 7fd0a4c00640 1 -- 172.21.15.6:0/1114830452 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fd090007e20 con 0x7fd0a8152400 2024-03-22T13:05:12.985 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:12.985 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:12.986 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:12.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd08406bc20 msgr2=0x7fd08406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:12.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd08406bc20 0x7fd08406e0e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd098009bd0 tx=0x7fd098030000 comp rx=0 tx=0).stop 2024-03-22T13:05:12.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a8152400 msgr2=0x7fd0a80caf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a8152400 0x7fd0a80caf90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8071680 tx=0x7fd090001e70 comp rx=0 tx=0).stop 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 shutdown_connections 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0a815bc80 0x7fd0a8159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a8152400 0x7fd0a80caf90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd08406bc20 0x7fd08406e0e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 --2- 172.21.15.6:0/1114830452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0a806f8c0 0x7fd0a81ec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 >> 172.21.15.6:0/1114830452 conn(0x7fd0a80cb930 msgr2=0x7fd0a80c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 shutdown_connections 2024-03-22T13:05:12.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:12.988+0000 7fd0af200640 1 -- 172.21.15.6:0/1114830452 wait complete. 2024-03-22T13:05:13.610 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:13.610 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:14.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.812+0000 7fa346c00640 1 Processor -- start 2024-03-22T13:05:14.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.812+0000 7fa346c00640 1 -- start start 2024-03-22T13:05:14.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.814+0000 7fa346c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3400c5210 0x7fa340158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.814+0000 7fa346c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34006e0f0 con 0x7fa3400c55f0 2024-03-22T13:05:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.814+0000 7fa346c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34006d350 con 0x7fa34006ede0 2024-03-22T13:05:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.814+0000 7fa346c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400c6f90 con 0x7fa3400c5210 2024-03-22T13:05:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.814+0000 7fa345800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3400c5210 0x7fa340158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.815+0000 7fa345800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3400c5210 0x7fa340158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35130/0 (socket says 172.21.15.6:35130) 2024-03-22T13:05:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.815+0000 7fa345800640 1 -- 172.21.15.6:0/576405335 learned_addr learned my addr 172.21.15.6:0/576405335 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.817+0000 7fa345800640 1 -- 172.21.15.6:0/576405335 >> v1:172.21.15.92:6789/0 conn(0x7fa34006ede0 legacy=0x7fa34014e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.817+0000 7fa345800640 1 -- 172.21.15.6:0/576405335 >> v1:172.21.15.6:6789/0 conn(0x7fa3400c55f0 legacy=0x7fa3400c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:14.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.817+0000 7fa345800640 1 -- 172.21.15.6:0/576405335 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400c6cb0 con 0x7fa3400c5210 2024-03-22T13:05:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.819+0000 7fa345800640 1 --2- 172.21.15.6:0/576405335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3400c5210 0x7fa340158a60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa3340098a0 tx=0x7fa33402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d75e0dbae23a1675 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:14.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.822+0000 7fa33fe00640 1 -- 172.21.15.6:0/576405335 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33402f830 con 0x7fa3400c5210 2024-03-22T13:05:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.824+0000 7fa33fe00640 1 -- 172.21.15.6:0/576405335 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33402f9d0 con 0x7fa3400c5210 2024-03-22T13:05:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.824+0000 7fa33fe00640 1 -- 172.21.15.6:0/576405335 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33402fda0 con 0x7fa3400c5210 2024-03-22T13:05:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.825+0000 7fa346c00640 1 -- 172.21.15.6:0/576405335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3400c5210 msgr2=0x7fa340158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.825+0000 7fa346c00640 1 --2- 172.21.15.6:0/576405335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3400c5210 0x7fa340158a60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa3340098a0 tx=0x7fa33402ba10 comp rx=0 tx=0).stop 2024-03-22T13:05:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.825+0000 7fa346c00640 1 -- 172.21.15.6:0/576405335 shutdown_connections 2024-03-22T13:05:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.825+0000 7fa346c00640 1 --2- 172.21.15.6:0/576405335 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3400c5210 0x7fa340158a60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.825+0000 7fa346c00640 1 -- 172.21.15.6:0/576405335 >> 172.21.15.6:0/576405335 conn(0x7fa3400cb890 msgr2=0x7fa3400cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.825+0000 7fa346c00640 1 -- 172.21.15.6:0/576405335 shutdown_connections 2024-03-22T13:05:14.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.825+0000 7fa346c00640 1 -- 172.21.15.6:0/576405335 wait complete. 2024-03-22T13:05:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.827+0000 7fa346c00640 1 Processor -- start 2024-03-22T13:05:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.827+0000 7fa346c00640 1 -- start start 2024-03-22T13:05:14.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.828+0000 7fa346c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa34006ede0 0x7fa3401493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:14.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.828+0000 7fa346c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3400c5210 0x7fa34006f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:14.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.828+0000 7fa346c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400c55f0 0x7fa3401f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:14.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.828+0000 7fa346c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34006dbd0 con 0x7fa3400c55f0 2024-03-22T13:05:14.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.828+0000 7fa346c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34006d8f0 con 0x7fa3400c5210 2024-03-22T13:05:14.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.828+0000 7fa346c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400c69a0 con 0x7fa34006ede0 2024-03-22T13:05:14.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.828+0000 7fa345800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa34006ede0 0x7fa3401493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:14.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.829+0000 7fa345800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa34006ede0 0x7fa3401493b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35140/0 (socket says 172.21.15.6:35140) 2024-03-22T13:05:14.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.829+0000 7fa345800640 1 -- 172.21.15.6:0/240530498 learned_addr learned my addr 172.21.15.6:0/240530498 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:14.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.829+0000 7fa344e00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3400c5210 0x7fa34006f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:14.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.829+0000 7fa346200640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400c55f0 0x7fa3401f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:14.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.832+0000 7fa345800640 1 -- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3400c5210 msgr2=0x7fa34006f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:14.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.832+0000 7fa345800640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3400c5210 0x7fa34006f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:14.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.832+0000 7fa345800640 1 -- 172.21.15.6:0/240530498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400c55f0 msgr2=0x7fa3401f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:14.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.832+0000 7fa345800640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400c55f0 0x7fa3401f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:14.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.832+0000 7fa345800640 1 -- 172.21.15.6:0/240530498 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3401498f0 con 0x7fa34006ede0 2024-03-22T13:05:14.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.835+0000 7fa345800640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa34006ede0 0x7fa3401493b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa334000c00 tx=0x7fa334004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.839+0000 7fa33d600640 1 -- 172.21.15.6:0/240530498 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334047070 con 0x7fa34006ede0 2024-03-22T13:05:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.839+0000 7fa33d600640 1 -- 172.21.15.6:0/240530498 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33402f7e0 con 0x7fa34006ede0 2024-03-22T13:05:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.839+0000 7fa33d600640 1 -- 172.21.15.6:0/240530498 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3340426c0 con 0x7fa34006ede0 2024-03-22T13:05:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.839+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa340148df0 con 0x7fa34006ede0 2024-03-22T13:05:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.839+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34014a5f0 con 0x7fa34006ede0 2024-03-22T13:05:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.840+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa340075650 con 0x7fa34006ede0 2024-03-22T13:05:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.859+0000 7fa33d600640 1 -- 172.21.15.6:0/240530498 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa33404a070 con 0x7fa34006ede0 2024-03-22T13:05:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.860+0000 7fa33d600640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa32006bc50 0x7fa32006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.861+0000 7fa344e00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa32006bc50 0x7fa32006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:14.861+0000 7fa344e00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa32006bc50 0x7fa32006e110 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa34006fe80 tx=0x7fa32c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.102+0000 7fa33d600640 1 -- 172.21.15.6:0/240530498 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa334079680 con 0x7fa34006ede0 2024-03-22T13:05:15.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.114+0000 7fa33d600640 1 -- 172.21.15.6:0/240530498 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa334051080 con 0x7fa34006ede0 2024-03-22T13:05:15.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.370+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3400c9f00 con 0x7fa34006ede0 2024-03-22T13:05:15.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.400+0000 7fa33d600640 1 -- 172.21.15.6:0/240530498 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7fa334033030 con 0x7fa34006ede0 2024-03-22T13:05:15.401 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:15.401 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:15.401 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa32006bc50 msgr2=0x7fa32006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa32006bc50 0x7fa32006e110 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa34006fe80 tx=0x7fa32c009040 comp rx=0 tx=0).stop 2024-03-22T13:05:15.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa34006ede0 msgr2=0x7fa3401493b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa34006ede0 0x7fa3401493b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa334000c00 tx=0x7fa334004730 comp rx=0 tx=0).stop 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 shutdown_connections 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3400c55f0 0x7fa3401f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa32006bc50 0x7fa32006e110 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3400c5210 0x7fa34006f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 --2- 172.21.15.6:0/240530498 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa34006ede0 0x7fa3401493b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.404+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 >> 172.21.15.6:0/240530498 conn(0x7fa3400cb890 msgr2=0x7fa34014d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.405+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 shutdown_connections 2024-03-22T13:05:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:15.405+0000 7fa346c00640 1 -- 172.21.15.6:0/240530498 wait complete. 2024-03-22T13:05:16.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:16.025 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:17.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:17.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.223+0000 7f8894c00640 1 Processor -- start 2024-03-22T13:05:17.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.224+0000 7f8894c00640 1 -- start start 2024-03-22T13:05:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.225+0000 7f8894c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f889006c230 0x7f889015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.225+0000 7f8894c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900cee00 con 0x7f88900ce8a0 2024-03-22T13:05:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.225+0000 7f8894c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900cf0c0 con 0x7f88900cc380 2024-03-22T13:05:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.225+0000 7f8894c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889006c660 con 0x7f889006c230 2024-03-22T13:05:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.226+0000 7f888fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f889006c230 0x7f889015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:17.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.226+0000 7f888fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f889006c230 0x7f889015ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35150/0 (socket says 172.21.15.6:35150) 2024-03-22T13:05:17.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.226+0000 7f888fe00640 1 -- 172.21.15.6:0/1516724784 learned_addr learned my addr 172.21.15.6:0/1516724784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:17.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.229+0000 7f888e000640 1 -- 172.21.15.6:0/1516724784 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775599110 0 0) 0x7f88900cee00 con 0x7f88900ce8a0 2024-03-22T13:05:17.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.230+0000 7f888e000640 1 -- 172.21.15.6:0/1516724784 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8874003520 con 0x7f88900ce8a0 2024-03-22T13:05:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.231+0000 7f888e000640 1 -- 172.21.15.6:0/1516724784 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099865254 0 0) 0x7f88900cf0c0 con 0x7f88900cc380 2024-03-22T13:05:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.231+0000 7f888e000640 1 -- 172.21.15.6:0/1516724784 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88900cee00 con 0x7f88900cc380 2024-03-22T13:05:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.232+0000 7f888fe00640 1 -- 172.21.15.6:0/1516724784 >> v1:172.21.15.92:6789/0 conn(0x7f88900cc380 legacy=0x7f88900cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.232+0000 7f888fe00640 1 -- 172.21.15.6:0/1516724784 >> v1:172.21.15.6:6789/0 conn(0x7f88900ce8a0 legacy=0x7f8890162b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.232+0000 7f888fe00640 1 -- 172.21.15.6:0/1516724784 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900cf3a0 con 0x7f889006c230 2024-03-22T13:05:17.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.235+0000 7f888fe00640 1 --2- 172.21.15.6:0/1516724784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f889006c230 0x7f889015ae30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8888009e80 tx=0x7f888802bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fef46f72c7d99a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:17.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.240+0000 7f888e000640 1 -- 172.21.15.6:0/1516724784 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888802cb70 con 0x7f889006c230 2024-03-22T13:05:17.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.240+0000 7f888e000640 1 -- 172.21.15.6:0/1516724784 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888802cd10 con 0x7f889006c230 2024-03-22T13:05:17.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.240+0000 7f888e000640 1 -- 172.21.15.6:0/1516724784 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888034610 con 0x7f889006c230 2024-03-22T13:05:17.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.241+0000 7f8894c00640 1 -- 172.21.15.6:0/1516724784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f889006c230 msgr2=0x7f889015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:17.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.241+0000 7f8894c00640 1 --2- 172.21.15.6:0/1516724784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f889006c230 0x7f889015ae30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8888009e80 tx=0x7f888802bfd0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.241+0000 7f8894c00640 1 -- 172.21.15.6:0/1516724784 shutdown_connections 2024-03-22T13:05:17.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.241+0000 7f8894c00640 1 --2- 172.21.15.6:0/1516724784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f889006c230 0x7f889015ae30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.241+0000 7f8894c00640 1 -- 172.21.15.6:0/1516724784 >> 172.21.15.6:0/1516724784 conn(0x7f88900b1480 msgr2=0x7f88900b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:17.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.241+0000 7f8894c00640 1 -- 172.21.15.6:0/1516724784 shutdown_connections 2024-03-22T13:05:17.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.241+0000 7f8894c00640 1 -- 172.21.15.6:0/1516724784 wait complete. 2024-03-22T13:05:17.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.244+0000 7f8894c00640 1 Processor -- start 2024-03-22T13:05:17.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.244+0000 7f8894c00640 1 -- start start 2024-03-22T13:05:17.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.245+0000 7f8894c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f889006c230 0x7f8890152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:17.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.245+0000 7f8894c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88900cc380 0x7f889015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:17.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.245+0000 7f8894c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88900ce8a0 0x7f889015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:17.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.245+0000 7f8894c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889015a910 con 0x7f889006c230 2024-03-22T13:05:17.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.245+0000 7f8894c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889015a630 con 0x7f88900ce8a0 2024-03-22T13:05:17.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.245+0000 7f8894c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88901549d0 con 0x7f88900cc380 2024-03-22T13:05:17.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.246+0000 7f888ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88900cc380 0x7f889015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:17.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.246+0000 7f888fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88900ce8a0 0x7f889015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:17.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.246+0000 7f888f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f889006c230 0x7f8890152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:17.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.246+0000 7f888fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88900ce8a0 0x7f889015c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:33102/0 (socket says 172.21.15.6:33102) 2024-03-22T13:05:17.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.246+0000 7f888ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88900cc380 0x7f889015c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35152/0 (socket says 172.21.15.6:35152) 2024-03-22T13:05:17.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.246+0000 7f888fe00640 1 -- 172.21.15.6:0/681795068 learned_addr learned my addr 172.21.15.6:0/681795068 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:17.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.250+0000 7f888fe00640 1 -- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88900cc380 msgr2=0x7f889015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:17.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.250+0000 7f888fe00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88900cc380 0x7f889015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.250+0000 7f888fe00640 1 -- 172.21.15.6:0/681795068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f889006c230 msgr2=0x7f8890152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:17.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.250+0000 7f888fe00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f889006c230 0x7f8890152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.250+0000 7f888fe00640 1 -- 172.21.15.6:0/681795068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88901f1400 con 0x7f88900ce8a0 2024-03-22T13:05:17.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.253+0000 7f888fe00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88900ce8a0 0x7f889015c850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8888036040 tx=0x7f8888004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.258+0000 7f8887400640 1 -- 172.21.15.6:0/681795068 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888043020 con 0x7f88900ce8a0 2024-03-22T13:05:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.258+0000 7f8887400640 1 -- 172.21.15.6:0/681795068 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888803e450 con 0x7f88900ce8a0 2024-03-22T13:05:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.258+0000 7f8887400640 1 -- 172.21.15.6:0/681795068 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888803e840 con 0x7f88900ce8a0 2024-03-22T13:05:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.258+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8890152e10 con 0x7f88900ce8a0 2024-03-22T13:05:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.258+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889015b550 con 0x7f88900ce8a0 2024-03-22T13:05:17.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.260+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88900c2e10 con 0x7f88900ce8a0 2024-03-22T13:05:17.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.294+0000 7f8887400640 1 -- 172.21.15.6:0/681795068 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8888046020 con 0x7f88900ce8a0 2024-03-22T13:05:17.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.295+0000 7f8887400640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f887006bc50 0x7f887006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.295+0000 7f888f400640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f887006bc50 0x7f887006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.296+0000 7f888f400640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f887006bc50 0x7f887006e110 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f88800011e0 tx=0x7f8880009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:17.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.297+0000 7f8887400640 1 -- 172.21.15.6:0/681795068 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8888079910 con 0x7f88900ce8a0 2024-03-22T13:05:17.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.570+0000 7f8887400640 1 -- 172.21.15.6:0/681795068 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f88900c2e10 con 0x7f88900ce8a0 2024-03-22T13:05:17.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.843+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88901f0e80 con 0x7f88900ce8a0 2024-03-22T13:05:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.856+0000 7f8887400640 1 -- 172.21.15.6:0/681795068 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6028 (secure 0 0 0) 0x7f888804d030 con 0x7f88900ce8a0 2024-03-22T13:05:17.856 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:17.856 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:01:19.921071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:01:19.921065+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:05:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.859+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f887006bc50 msgr2=0x7f887006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f887006bc50 0x7f887006e110 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f88800011e0 tx=0x7f8880009040 comp rx=0 tx=0).stop 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88900ce8a0 msgr2=0x7f889015c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88900ce8a0 0x7f889015c850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8888036040 tx=0x7f8888004a40 comp rx=0 tx=0).stop 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 shutdown_connections 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f887006bc50 0x7f887006e110 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88900ce8a0 0x7f889015c850 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88900cc380 0x7f889015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 --2- 172.21.15.6:0/681795068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f889006c230 0x7f8890152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 >> 172.21.15.6:0/681795068 conn(0x7f88900b1480 msgr2=0x7f88901589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 shutdown_connections 2024-03-22T13:05:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:17.860+0000 7f8894c00640 1 -- 172.21.15.6:0/681795068 wait complete. 2024-03-22T13:05:18.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:01:19.921071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:01:19.921065+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:05:18.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:19.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.683+0000 7ff1e5200640 1 Processor -- start 2024-03-22T13:05:19.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.683+0000 7ff1e5200640 1 -- start start 2024-03-22T13:05:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.684+0000 7ff1e5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.684+0000 7ff1e5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1e015ae30 0x7ff1e00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.684+0000 7ff1e5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e015b260 con 0x7ff1e00ce8a0 2024-03-22T13:05:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.684+0000 7ff1e5200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e00cee00 con 0x7ff1e015ae30 2024-03-22T13:05:19.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.684+0000 7ff1e5200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e00cf0c0 con 0x7ff1e00cc380 2024-03-22T13:05:19.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.685+0000 7ff1dfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1e015ae30 0x7ff1e00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:19.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.685+0000 7ff1df400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:19.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.686+0000 7ff1df400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35156/0 (socket says 172.21.15.6:35156) 2024-03-22T13:05:19.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.686+0000 7ff1df400640 1 -- 172.21.15.6:0/191485195 learned_addr learned my addr 172.21.15.6:0/191485195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.690+0000 7ff1df400640 1 -- 172.21.15.6:0/191485195 >> v2:172.21.15.6:3300/0 conn(0x7ff1e015ae30 msgr2=0x7ff1e00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.690+0000 7ff1df400640 1 --2- 172.21.15.6:0/191485195 >> v2:172.21.15.6:3300/0 conn(0x7ff1e015ae30 0x7ff1e00cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.690+0000 7ff1df400640 1 -- 172.21.15.6:0/191485195 >> v1:172.21.15.6:6789/0 conn(0x7ff1e00ce8a0 legacy=0x7ff1e00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:19.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.690+0000 7ff1df400640 1 -- 172.21.15.6:0/191485195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e00cf3a0 con 0x7ff1e00cc380 2024-03-22T13:05:19.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.690+0000 7ff1dfe00640 1 --2- 172.21.15.6:0/191485195 >> v2:172.21.15.6:3300/0 conn(0x7ff1e015ae30 0x7ff1e00cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:05:19.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.693+0000 7ff1df400640 1 --2- 172.21.15.6:0/191485195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e006c230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009870 tx=0x7ff1d002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4cfb0cc2b84d956 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:19.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.698+0000 7ff1de000640 1 -- 172.21.15.6:0/191485195 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0034070 con 0x7ff1e00cc380 2024-03-22T13:05:19.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.698+0000 7ff1de000640 1 -- 172.21.15.6:0/191485195 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d0030840 con 0x7ff1e00cc380 2024-03-22T13:05:19.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.698+0000 7ff1de000640 1 -- 172.21.15.6:0/191485195 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0030c10 con 0x7ff1e00cc380 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 -- 172.21.15.6:0/191485195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 msgr2=0x7ff1e006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 --2- 172.21.15.6:0/191485195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e006c230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009870 tx=0x7ff1d002b980 comp rx=0 tx=0).stop 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 -- 172.21.15.6:0/191485195 shutdown_connections 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 --2- 172.21.15.6:0/191485195 >> v2:172.21.15.6:3300/0 conn(0x7ff1e015ae30 0x7ff1e00cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 --2- 172.21.15.6:0/191485195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e006c230 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 -- 172.21.15.6:0/191485195 >> 172.21.15.6:0/191485195 conn(0x7ff1e00b1480 msgr2=0x7ff1e00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 -- 172.21.15.6:0/191485195 shutdown_connections 2024-03-22T13:05:19.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.699+0000 7ff1e5200640 1 -- 172.21.15.6:0/191485195 wait complete. 2024-03-22T13:05:19.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.701+0000 7ff1e5200640 1 Processor -- start 2024-03-22T13:05:19.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.701+0000 7ff1e5200640 1 -- start start 2024-03-22T13:05:19.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.702+0000 7ff1e5200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e0145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:19.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.702+0000 7ff1e5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e00ce8a0 0x7ff1e015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:19.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.702+0000 7ff1e5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1e015ae30 0x7ff1e0148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:19.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.702+0000 7ff1e5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e015a910 con 0x7ff1e00ce8a0 2024-03-22T13:05:19.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.702+0000 7ff1e5200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e015a630 con 0x7ff1e015ae30 2024-03-22T13:05:19.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.702+0000 7ff1e5200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01549d0 con 0x7ff1e00cc380 2024-03-22T13:05:19.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.703+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1e015ae30 0x7ff1e0148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:19.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.703+0000 7ff1df400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e0145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:19.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.703+0000 7ff1dea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e00ce8a0 0x7ff1e015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:19.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.703+0000 7ff1dea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e00ce8a0 0x7ff1e015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60182/0 (socket says 172.21.15.6:60182) 2024-03-22T13:05:19.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.703+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1e015ae30 0x7ff1e0148e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:33106/0 (socket says 172.21.15.6:33106) 2024-03-22T13:05:19.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.703+0000 7ff1dfe00640 1 -- 172.21.15.6:0/1375859662 learned_addr learned my addr 172.21.15.6:0/1375859662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:19.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.703+0000 7ff1df400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e0145000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35158/0 (socket says 172.21.15.6:35158) 2024-03-22T13:05:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.707+0000 7ff1dfe00640 1 -- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 msgr2=0x7ff1e0145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.707+0000 7ff1dfe00640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e0145000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.707+0000 7ff1dfe00640 1 -- 172.21.15.6:0/1375859662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e00ce8a0 msgr2=0x7ff1e015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.707+0000 7ff1dfe00640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e00ce8a0 0x7ff1e015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.707+0000 7ff1dfe00640 1 -- 172.21.15.6:0/1375859662 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e00ca430 con 0x7ff1e015ae30 2024-03-22T13:05:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.711+0000 7ff1dfe00640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1e015ae30 0x7ff1e0148e80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800efc0 tx=0x7ff1d8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:19.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.714+0000 7ff1d7400640 1 -- 172.21.15.6:0/1375859662 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8018030 con 0x7ff1e015ae30 2024-03-22T13:05:19.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.714+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e01477b0 con 0x7ff1e015ae30 2024-03-22T13:05:19.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.714+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e01474d0 con 0x7ff1e015ae30 2024-03-22T13:05:19.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.715+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e00684e0 con 0x7ff1e015ae30 2024-03-22T13:05:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.716+0000 7ff1d7400640 1 -- 172.21.15.6:0/1375859662 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d800cde0 con 0x7ff1e015ae30 2024-03-22T13:05:19.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.716+0000 7ff1d7400640 1 -- 172.21.15.6:0/1375859662 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8003b90 con 0x7ff1e015ae30 2024-03-22T13:05:19.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:19.997+0000 7ff1d7400640 1 -- 172.21.15.6:0/1375859662 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff1d8005d10 con 0x7ff1e015ae30 2024-03-22T13:05:19.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.000+0000 7ff1d7400640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1c006bbd0 0x7ff1c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.000+0000 7ff1df400640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1c006bbd0 0x7ff1c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:20.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.001+0000 7ff1df400640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1c006bbd0 0x7ff1c006e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0036000 tx=0x7ff1d003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:20.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.002+0000 7ff1d7400640 1 -- 172.21.15.6:0/1375859662 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff1d8058050 con 0x7ff1e015ae30 2024-03-22T13:05:20.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.022+0000 7ff1d7400640 1 -- 172.21.15.6:0/1375859662 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1e00684e0 con 0x7ff1e015ae30 2024-03-22T13:05:20.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.291+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1e0075650 con 0x7ff1e015ae30 2024-03-22T13:05:20.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.363+0000 7ff1d7400640 1 -- 172.21.15.6:0/1375859662 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7ff1d8022070 con 0x7ff1e015ae30 2024-03-22T13:05:20.363 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:20.363 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:20.363 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:20.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1c006bbd0 msgr2=0x7ff1c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1c006bbd0 0x7ff1c006e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0036000 tx=0x7ff1d003b040 comp rx=0 tx=0).stop 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1e015ae30 msgr2=0x7ff1e0148e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1e015ae30 0x7ff1e0148e80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800efc0 tx=0x7ff1d8009f90 comp rx=0 tx=0).stop 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 shutdown_connections 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1c006bbd0 0x7ff1c006e090 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1e015ae30 0x7ff1e0148e80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e00ce8a0 0x7ff1e015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 --2- 172.21.15.6:0/1375859662 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1e00cc380 0x7ff1e0145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 >> 172.21.15.6:0/1375859662 conn(0x7ff1e00b1480 msgr2=0x7ff1e0149f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 shutdown_connections 2024-03-22T13:05:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:20.366+0000 7ff1e5200640 1 -- 172.21.15.6:0/1375859662 wait complete. 2024-03-22T13:05:20.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:20.986 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:21.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.189+0000 7f28efa00640 1 Processor -- start 2024-03-22T13:05:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.190+0000 7f28efa00640 1 -- start start 2024-03-22T13:05:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.191+0000 7f28efa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e80ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.191+0000 7f28efa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28e80cd300 0x7f28e80ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.191+0000 7f28efa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e80cc380 con 0x7f28e815a9b0 2024-03-22T13:05:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.191+0000 7f28efa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e815a520 con 0x7f28e80cd300 2024-03-22T13:05:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.191+0000 7f28efa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e80cc660 con 0x7f28e8154980 2024-03-22T13:05:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.192+0000 7f28ed000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e80ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.192+0000 7f28e7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28e80cd300 0x7f28e80ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.192+0000 7f28ed000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e80ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35162/0 (socket says 172.21.15.6:35162) 2024-03-22T13:05:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.192+0000 7f28ed000640 1 -- 172.21.15.6:0/406406673 learned_addr learned my addr 172.21.15.6:0/406406673 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.195+0000 7f28ed000640 1 -- 172.21.15.6:0/406406673 >> v2:172.21.15.6:3300/0 conn(0x7f28e80cd300 msgr2=0x7f28e80ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.195+0000 7f28ed000640 1 --2- 172.21.15.6:0/406406673 >> v2:172.21.15.6:3300/0 conn(0x7f28e80cd300 0x7f28e80ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.195+0000 7f28ed000640 1 -- 172.21.15.6:0/406406673 >> v1:172.21.15.6:6789/0 conn(0x7f28e815a9b0 legacy=0x7f28e80ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.195+0000 7f28ed000640 1 -- 172.21.15.6:0/406406673 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e80cc940 con 0x7f28e8154980 2024-03-22T13:05:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.197+0000 7f28ed000640 1 --2- 172.21.15.6:0/406406673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e80ce300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009870 tx=0x7f28dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b48e1206e457501 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.201+0000 7f28e7400640 1 -- 172.21.15.6:0/406406673 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc034070 con 0x7f28e8154980 2024-03-22T13:05:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.201+0000 7f28e7400640 1 -- 172.21.15.6:0/406406673 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc030840 con 0x7f28e8154980 2024-03-22T13:05:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.201+0000 7f28e7400640 1 -- 172.21.15.6:0/406406673 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc030c10 con 0x7f28e8154980 2024-03-22T13:05:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.202+0000 7f28efa00640 1 -- 172.21.15.6:0/406406673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 msgr2=0x7f28e80ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.202+0000 7f28efa00640 1 --2- 172.21.15.6:0/406406673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e80ce300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009870 tx=0x7f28dc02b980 comp rx=0 tx=0).stop 2024-03-22T13:05:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.203+0000 7f28efa00640 1 -- 172.21.15.6:0/406406673 shutdown_connections 2024-03-22T13:05:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.203+0000 7f28efa00640 1 --2- 172.21.15.6:0/406406673 >> v2:172.21.15.6:3300/0 conn(0x7f28e80cd300 0x7f28e80ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.203+0000 7f28efa00640 1 --2- 172.21.15.6:0/406406673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e80ce300 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.203+0000 7f28efa00640 1 -- 172.21.15.6:0/406406673 >> 172.21.15.6:0/406406673 conn(0x7f28e80b16c0 msgr2=0x7f28e80b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.203+0000 7f28efa00640 1 -- 172.21.15.6:0/406406673 shutdown_connections 2024-03-22T13:05:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.203+0000 7f28efa00640 1 -- 172.21.15.6:0/406406673 wait complete. 2024-03-22T13:05:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.205+0000 7f28efa00640 1 Processor -- start 2024-03-22T13:05:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.205+0000 7f28efa00640 1 -- start start 2024-03-22T13:05:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.206+0000 7f28efa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28e80cd300 0x7f28e814dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.206+0000 7f28efa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e81f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.206+0000 7f28efa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e815a9b0 0x7f28e81f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.206+0000 7f28efa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e80cf5b0 con 0x7f28e80cd300 2024-03-22T13:05:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.206+0000 7f28efa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e80ccf00 con 0x7f28e815a9b0 2024-03-22T13:05:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.206+0000 7f28efa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e80cf730 con 0x7f28e8154980 2024-03-22T13:05:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.207+0000 7f28ed000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28e80cd300 0x7f28e814dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.207+0000 7f28ed000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28e80cd300 0x7f28e814dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60202/0 (socket says 172.21.15.6:60202) 2024-03-22T13:05:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.207+0000 7f28ed000640 1 -- 172.21.15.6:0/3255374764 learned_addr learned my addr 172.21.15.6:0/3255374764 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.208+0000 7f28e7e00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e81f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:22.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.209+0000 7f28eda00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e815a9b0 0x7f28e81f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:22.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.211+0000 7f28ed000640 1 -- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 msgr2=0x7f28e81f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:22.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.211+0000 7f28ed000640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e81f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.211+0000 7f28ed000640 1 -- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e815a9b0 msgr2=0x7f28e81f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:22.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.211+0000 7f28ed000640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e815a9b0 0x7f28e81f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.211+0000 7f28ed000640 1 -- 172.21.15.6:0/3255374764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e81514e0 con 0x7f28e80cd300 2024-03-22T13:05:22.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.216+0000 7f28ed000640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28e80cd300 0x7f28e814dbc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:22.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.221+0000 7f28e4c00640 1 -- 172.21.15.6:0/3255374764 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc048070 con 0x7f28e80cd300 2024-03-22T13:05:22.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.222+0000 7f28e4c00640 1 -- 172.21.15.6:0/3255374764 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc030640 con 0x7f28e80cd300 2024-03-22T13:05:22.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.222+0000 7f28e4c00640 1 -- 172.21.15.6:0/3255374764 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc034040 con 0x7f28e80cd300 2024-03-22T13:05:22.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.222+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e81f4010 con 0x7f28e80cd300 2024-03-22T13:05:22.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.222+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e81f3d30 con 0x7f28e80cd300 2024-03-22T13:05:22.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.223+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e80c8550 con 0x7f28e80cd300 2024-03-22T13:05:22.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.510+0000 7f28e4c00640 1 -- 172.21.15.6:0/3255374764 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f28dc04b070 con 0x7f28e80cd300 2024-03-22T13:05:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.512+0000 7f28e4c00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28c806bc20 0x7f28c806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:22.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.513+0000 7f28e7e00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28c806bc20 0x7f28c806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:22.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.514+0000 7f28e7e00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28c806bc20 0x7f28c806e0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f28e8153460 tx=0x7f28d4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:22.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.535+0000 7f28e4c00640 1 -- 172.21.15.6:0/3255374764 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f28dc0307f0 con 0x7f28e80cd300 2024-03-22T13:05:22.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.537+0000 7f28e4c00640 1 -- 172.21.15.6:0/3255374764 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f28e80c8550 con 0x7f28e80cd300 2024-03-22T13:05:22.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.796+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28e80756c0 con 0x7f28e80cd300 2024-03-22T13:05:22.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.821+0000 7f28e4c00640 1 -- 172.21.15.6:0/3255374764 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f28dc052020 con 0x7f28e80cd300 2024-03-22T13:05:22.822 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:22.822 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:22.822 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28c806bc20 msgr2=0x7f28c806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28c806bc20 0x7f28c806e0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f28e8153460 tx=0x7f28d4009190 comp rx=0 tx=0).stop 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28e80cd300 msgr2=0x7f28e814dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28e80cd300 0x7f28e814dbc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc004730 comp rx=0 tx=0).stop 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 shutdown_connections 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28c806bc20 0x7f28c806e0e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e815a9b0 0x7f28e81f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e8154980 0x7f28e81f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 --2- 172.21.15.6:0/3255374764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28e80cd300 0x7f28e814dbc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 >> 172.21.15.6:0/3255374764 conn(0x7f28e80b16c0 msgr2=0x7f28e80744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 shutdown_connections 2024-03-22T13:05:22.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:22.825+0000 7f28efa00640 1 -- 172.21.15.6:0/3255374764 wait complete. 2024-03-22T13:05:23.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:23.445 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:24.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:24.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.647+0000 7f059d400640 1 Processor -- start 2024-03-22T13:05:24.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.647+0000 7f059d400640 1 -- start start 2024-03-22T13:05:24.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.650+0000 7f059d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0598154540 0x7f059815e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:24.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.650+0000 7f059d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05980c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:24.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.650+0000 7f059d400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059815e7f0 con 0x7f059806ede0 2024-03-22T13:05:24.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.650+0000 7f059d400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05980c79e0 con 0x7f0598154540 2024-03-22T13:05:24.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.650+0000 7f059d400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059815b2d0 con 0x7f05980c91e0 2024-03-22T13:05:24.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.651+0000 7f059ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05980c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:24.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.651+0000 7f059ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05980c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56766/0 (socket says 172.21.15.6:56766) 2024-03-22T13:05:24.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.651+0000 7f059ca00640 1 -- 172.21.15.6:0/2063872442 learned_addr learned my addr 172.21.15.6:0/2063872442 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:24.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.655+0000 7f059ca00640 1 -- 172.21.15.6:0/2063872442 >> v2:172.21.15.6:3300/0 conn(0x7f0598154540 msgr2=0x7f059815e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:24.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.655+0000 7f059ca00640 1 --2- 172.21.15.6:0/2063872442 >> v2:172.21.15.6:3300/0 conn(0x7f0598154540 0x7f059815e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:24.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.655+0000 7f059ca00640 1 -- 172.21.15.6:0/2063872442 >> v1:172.21.15.92:6789/0 conn(0x7f059806ede0 legacy=0x7f059815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:24.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.655+0000 7f059ca00640 1 -- 172.21.15.6:0/2063872442 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598070890 con 0x7f05980c91e0 2024-03-22T13:05:24.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.658+0000 7f059ca00640 1 --2- 172.21.15.6:0/2063872442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05980c9670 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0590009e50 tx=0x7f059002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4c2e75dfcc2dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:24.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.661+0000 7f0596a00640 1 -- 172.21.15.6:0/2063872442 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059002cb00 con 0x7f05980c91e0 2024-03-22T13:05:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.664+0000 7f0596a00640 1 -- 172.21.15.6:0/2063872442 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059002cca0 con 0x7f05980c91e0 2024-03-22T13:05:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.664+0000 7f0596a00640 1 -- 172.21.15.6:0/2063872442 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590034610 con 0x7f05980c91e0 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 -- 172.21.15.6:0/2063872442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 msgr2=0x7f05980c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 --2- 172.21.15.6:0/2063872442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05980c9670 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0590009e50 tx=0x7f059002bf60 comp rx=0 tx=0).stop 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 -- 172.21.15.6:0/2063872442 shutdown_connections 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 --2- 172.21.15.6:0/2063872442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05980c9670 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 --2- 172.21.15.6:0/2063872442 >> v2:172.21.15.6:3300/0 conn(0x7f0598154540 0x7f059815e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 -- 172.21.15.6:0/2063872442 >> 172.21.15.6:0/2063872442 conn(0x7f059806c190 msgr2=0x7f059806c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 -- 172.21.15.6:0/2063872442 shutdown_connections 2024-03-22T13:05:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.665+0000 7f059d400640 1 -- 172.21.15.6:0/2063872442 wait complete. 2024-03-22T13:05:24.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.668+0000 7f059d400640 1 Processor -- start 2024-03-22T13:05:24.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.668+0000 7f059d400640 1 -- start start 2024-03-22T13:05:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.669+0000 7f059d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f059806ede0 0x7f0598144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.669+0000 7f059d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05981f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.669+0000 7f059d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0598154540 0x7f059814f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.669+0000 7f059d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059815a730 con 0x7f0598154540 2024-03-22T13:05:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.669+0000 7f059d400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05980c9bb0 con 0x7f059806ede0 2024-03-22T13:05:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.669+0000 7f059d400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059815a3c0 con 0x7f05980c91e0 2024-03-22T13:05:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.670+0000 7f0597400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05981f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.670+0000 7f0597400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05981f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56774/0 (socket says 172.21.15.6:56774) 2024-03-22T13:05:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.670+0000 7f0597400640 1 -- 172.21.15.6:0/2041842076 learned_addr learned my addr 172.21.15.6:0/2041842076 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:24.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.676+0000 7f0597400640 1 -- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f059806ede0 msgr2=0x7f0598144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:24.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.676+0000 7f0597400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f059806ede0 0x7f0598144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:24.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.676+0000 7f0597400640 1 -- 172.21.15.6:0/2041842076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0598154540 msgr2=0x7f059814f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:24.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.676+0000 7f0597400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0598154540 0x7f059814f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.676+0000 7f0597400640 1 -- 172.21.15.6:0/2041842076 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598150270 con 0x7f05980c91e0 2024-03-22T13:05:24.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.679+0000 7f0597400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05981f3ae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f058000c440 tx=0x7f058000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:24.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.685+0000 7f058fe00640 1 -- 172.21.15.6:0/2041842076 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580017070 con 0x7f05980c91e0 2024-03-22T13:05:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.685+0000 7f058fe00640 1 -- 172.21.15.6:0/2041842076 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05800035f0 con 0x7f05980c91e0 2024-03-22T13:05:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.685+0000 7f058fe00640 1 -- 172.21.15.6:0/2041842076 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800039e0 con 0x7f05980c91e0 2024-03-22T13:05:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.685+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059814fcb0 con 0x7f05980c91e0 2024-03-22T13:05:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.685+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059814e940 con 0x7f05980c91e0 2024-03-22T13:05:24.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.687+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0598075650 con 0x7f05980c91e0 2024-03-22T13:05:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.721+0000 7f058fe00640 1 -- 172.21.15.6:0/2041842076 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f058001a070 con 0x7f05980c91e0 2024-03-22T13:05:24.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.723+0000 7f058fe00640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f057406bc50 0x7f057406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:24.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.724+0000 7f0597e00640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f057406bc50 0x7f057406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.725+0000 7f0597e00640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f057406bc50 0x7f057406e110 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0588009670 tx=0x7f0588006bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:24.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.970+0000 7f058fe00640 1 -- 172.21.15.6:0/2041842076 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f05800566b0 con 0x7f05980c91e0 2024-03-22T13:05:24.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:24.989+0000 7f058fe00640 1 -- 172.21.15.6:0/2041842076 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0580021020 con 0x7f05980c91e0 2024-03-22T13:05:25.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.290+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05980c3840 con 0x7f05980c91e0 2024-03-22T13:05:25.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.312+0000 7f058fe00640 1 -- 172.21.15.6:0/2041842076 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f0580058d10 con 0x7f05980c91e0 2024-03-22T13:05:25.312 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:25.312 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:25.312 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f057406bc50 msgr2=0x7f057406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f057406bc50 0x7f057406e110 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0588009670 tx=0x7f0588006bf0 comp rx=0 tx=0).stop 2024-03-22T13:05:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 msgr2=0x7f05981f3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05981f3ae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f058000c440 tx=0x7f058000c900 comp rx=0 tx=0).stop 2024-03-22T13:05:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 shutdown_connections 2024-03-22T13:05:25.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0598154540 0x7f059814f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:25.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05980c91e0 0x7f05981f3ae0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:25.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f057406bc50 0x7f057406e110 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:25.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 --2- 172.21.15.6:0/2041842076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f059806ede0 0x7f0598144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:25.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 >> 172.21.15.6:0/2041842076 conn(0x7f059806c190 msgr2=0x7f05980c5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:25.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 shutdown_connections 2024-03-22T13:05:25.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:25.315+0000 7f059d400640 1 -- 172.21.15.6:0/2041842076 wait complete. 2024-03-22T13:05:25.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:25.943 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:26.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.141+0000 7fbdfb600640 1 Processor -- start 2024-03-22T13:05:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.141+0000 7fbdfb600640 1 -- start start 2024-03-22T13:05:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.143+0000 7fbdfb600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbdf406b080 0x7fbdf406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.143+0000 7fbdfb600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406fea0 0x7fbdf415d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.143+0000 7fbdfb600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40c3d30 con 0x7fbdf415de30 2024-03-22T13:05:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.143+0000 7fbdfb600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf40c3a50 con 0x7fbdf406b080 2024-03-22T13:05:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.143+0000 7fbdfb600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf415a450 con 0x7fbdf406fea0 2024-03-22T13:05:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.144+0000 7fbdf3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406fea0 0x7fbdf415d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.144+0000 7fbdf8c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbdf406b080 0x7fbdf406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.144+0000 7fbdf3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406fea0 0x7fbdf415d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56788/0 (socket says 172.21.15.6:56788) 2024-03-22T13:05:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.144+0000 7fbdf3e00640 1 -- 172.21.15.6:0/4094776610 learned_addr learned my addr 172.21.15.6:0/4094776610 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.147+0000 7fbdf3e00640 1 -- 172.21.15.6:0/4094776610 >> v2:172.21.15.92:3300/0 conn(0x7fbdf406b080 msgr2=0x7fbdf406b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.147+0000 7fbdf3e00640 1 --2- 172.21.15.6:0/4094776610 >> v2:172.21.15.92:3300/0 conn(0x7fbdf406b080 0x7fbdf406b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.147+0000 7fbdf3e00640 1 -- 172.21.15.6:0/4094776610 >> v1:172.21.15.92:6789/0 conn(0x7fbdf415de30 legacy=0x7fbdf40c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.147+0000 7fbdf3e00640 1 -- 172.21.15.6:0/4094776610 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf415a730 con 0x7fbdf406fea0 2024-03-22T13:05:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.149+0000 7fbdf3e00640 1 --2- 172.21.15.6:0/4094776610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406fea0 0x7fbdf415d8f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009870 tx=0x7fbde002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa44e195975cca36 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.154+0000 7fbdf3400640 1 -- 172.21.15.6:0/4094776610 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde002f840 con 0x7fbdf406fea0 2024-03-22T13:05:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.154+0000 7fbdf3400640 1 -- 172.21.15.6:0/4094776610 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde002f9e0 con 0x7fbdf406fea0 2024-03-22T13:05:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.154+0000 7fbdf3400640 1 -- 172.21.15.6:0/4094776610 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde002fdb0 con 0x7fbdf406fea0 2024-03-22T13:05:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 -- 172.21.15.6:0/4094776610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406fea0 msgr2=0x7fbdf415d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 --2- 172.21.15.6:0/4094776610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406fea0 0x7fbdf415d8f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009870 tx=0x7fbde002ba20 comp rx=0 tx=0).stop 2024-03-22T13:05:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 -- 172.21.15.6:0/4094776610 shutdown_connections 2024-03-22T13:05:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 --2- 172.21.15.6:0/4094776610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406fea0 0x7fbdf415d8f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 --2- 172.21.15.6:0/4094776610 >> v2:172.21.15.92:3300/0 conn(0x7fbdf406b080 0x7fbdf406b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 -- 172.21.15.6:0/4094776610 >> 172.21.15.6:0/4094776610 conn(0x7fbdf40ccb50 msgr2=0x7fbdf40ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 -- 172.21.15.6:0/4094776610 shutdown_connections 2024-03-22T13:05:27.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.155+0000 7fbdfb600640 1 -- 172.21.15.6:0/4094776610 wait complete. 2024-03-22T13:05:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.157+0000 7fbdfb600640 1 Processor -- start 2024-03-22T13:05:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.158+0000 7fbdfb600640 1 -- start start 2024-03-22T13:05:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdfb600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406b080 0x7fbdf41529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdfb600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdf406fea0 0x7fbdf41f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdfb600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdf415de30 0x7fbdf4143160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdfb600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf415b210 con 0x7fbdf406fea0 2024-03-22T13:05:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdfb600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf415acf0 con 0x7fbdf415de30 2024-03-22T13:05:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdfb600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf4154930 con 0x7fbdf406b080 2024-03-22T13:05:27.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdf8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406b080 0x7fbdf41529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdf9600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdf415de30 0x7fbdf4143160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdf8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406b080 0x7fbdf41529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56790/0 (socket says 172.21.15.6:56790) 2024-03-22T13:05:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.159+0000 7fbdf8c00640 1 -- 172.21.15.6:0/1392583139 learned_addr learned my addr 172.21.15.6:0/1392583139 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:27.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.160+0000 7fbdf3e00640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdf406fea0 0x7fbdf41f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:27.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.164+0000 7fbdf8c00640 1 -- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdf415de30 msgr2=0x7fbdf4143160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:27.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.164+0000 7fbdf8c00640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdf415de30 0x7fbdf4143160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.164+0000 7fbdf8c00640 1 -- 172.21.15.6:0/1392583139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdf406fea0 msgr2=0x7fbdf41f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:27.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.164+0000 7fbdf8c00640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdf406fea0 0x7fbdf41f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.164+0000 7fbdf8c00640 1 -- 172.21.15.6:0/1392583139 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40c6d70 con 0x7fbdf406b080 2024-03-22T13:05:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.168+0000 7fbdf8c00640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406b080 0x7fbdf41529b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbde800e9e0 tx=0x7fbde800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:27.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.173+0000 7fbdf0c00640 1 -- 172.21.15.6:0/1392583139 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8017070 con 0x7fbdf406b080 2024-03-22T13:05:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.173+0000 7fbdf0c00640 1 -- 172.21.15.6:0/1392583139 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde800c9d0 con 0x7fbdf406b080 2024-03-22T13:05:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.173+0000 7fbdf0c00640 1 -- 172.21.15.6:0/1392583139 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde800cdc0 con 0x7fbdf406b080 2024-03-22T13:05:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.173+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf415b6f0 con 0x7fbdf406b080 2024-03-22T13:05:27.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.174+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf40c6290 con 0x7fbdf406b080 2024-03-22T13:05:27.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.175+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf40756c0 con 0x7fbdf406b080 2024-03-22T13:05:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.212+0000 7fbdf0c00640 1 -- 172.21.15.6:0/1392583139 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbde801a070 con 0x7fbdf406b080 2024-03-22T13:05:27.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.213+0000 7fbdf0c00640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbdd406bc20 0x7fbdd406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:27.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.214+0000 7fbdf3e00640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbdd406bc20 0x7fbdd406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:27.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.215+0000 7fbdf3e00640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbdd406bc20 0x7fbdd406e0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbde0036000 tx=0x7fbde0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:27.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.215+0000 7fbdf0c00640 1 -- 172.21.15.6:0/1392583139 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fbde8057760 con 0x7fbdf406b080 2024-03-22T13:05:27.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.479+0000 7fbdf0c00640 1 -- 172.21.15.6:0/1392583139 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbde80596c0 con 0x7fbdf406b080 2024-03-22T13:05:27.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.734+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf40ccf30 con 0x7fbdf406b080 2024-03-22T13:05:27.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.820+0000 7fbdf0c00640 1 -- 172.21.15.6:0/1392583139 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fbde805d760 con 0x7fbdf406b080 2024-03-22T13:05:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:27.820 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:27.821 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:27.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.822+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbdd406bc20 msgr2=0x7fbdd406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:27.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.822+0000 7fbdfb600640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbdd406bc20 0x7fbdd406e0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbde0036000 tx=0x7fbde0008030 comp rx=0 tx=0).stop 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.822+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406b080 msgr2=0x7fbdf41529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.822+0000 7fbdfb600640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406b080 0x7fbdf41529b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbde800e9e0 tx=0x7fbde800eea0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 shutdown_connections 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fbdd406bc20 0x7fbdd406e0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdf415de30 0x7fbdf4143160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdf406fea0 0x7fbdf41f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 --2- 172.21.15.6:0/1392583139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbdf406b080 0x7fbdf41529b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 >> 172.21.15.6:0/1392583139 conn(0x7fbdf40ccb50 msgr2=0x7fbdf41537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 shutdown_connections 2024-03-22T13:05:27.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:27.823+0000 7fbdfb600640 1 -- 172.21.15.6:0/1392583139 wait complete. 2024-03-22T13:05:28.452 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:28.452 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:29.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:29.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.652+0000 7efd2de00640 1 Processor -- start 2024-03-22T13:05:29.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.653+0000 7efd2de00640 1 -- start start 2024-03-22T13:05:29.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.654+0000 7efd2de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:29.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.654+0000 7efd2de00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd280c4740 con 0x7efd280c68c0 2024-03-22T13:05:29.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.654+0000 7efd2de00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd28154620 con 0x7efd280c7ee0 2024-03-22T13:05:29.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.654+0000 7efd2de00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2815e130 con 0x7efd28154240 2024-03-22T13:05:29.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.655+0000 7efd27400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efd280c68c0 0x7efd2806b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49484/0 (socket says 172.21.15.6:49484) 2024-03-22T13:05:29.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.655+0000 7efd27400640 1 -- 172.21.15.6:0/1511121730 learned_addr learned my addr 172.21.15.6:0/1511121730 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:29.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.655+0000 7efd26a00640 1 --2- 172.21.15.6:0/1511121730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:29.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.658+0000 7efd25600640 1 -- 172.21.15.6:0/1511121730 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195942188 0 0) 0x7efd280c4740 con 0x7efd280c68c0 2024-03-22T13:05:29.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.658+0000 7efd25600640 1 -- 172.21.15.6:0/1511121730 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd0c003520 con 0x7efd280c68c0 2024-03-22T13:05:29.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.658+0000 7efd25600640 1 -- 172.21.15.6:0/1511121730 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738256169 0 0) 0x7efd28154620 con 0x7efd280c7ee0 2024-03-22T13:05:29.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.658+0000 7efd25600640 1 -- 172.21.15.6:0/1511121730 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd280c4740 con 0x7efd280c7ee0 2024-03-22T13:05:29.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.658+0000 7efd26a00640 1 -- 172.21.15.6:0/1511121730 >> v1:172.21.15.92:6789/0 conn(0x7efd280c7ee0 legacy=0x7efd280c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:29.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.658+0000 7efd26a00640 1 -- 172.21.15.6:0/1511121730 >> v1:172.21.15.6:6789/0 conn(0x7efd280c68c0 legacy=0x7efd2806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:29.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.658+0000 7efd26a00640 1 -- 172.21.15.6:0/1511121730 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd280c6e00 con 0x7efd28154240 2024-03-22T13:05:29.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.661+0000 7efd26a00640 1 --2- 172.21.15.6:0/1511121730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815ac90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efd180097d0 tx=0x7efd1802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66cc41533c7a0cae server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:29.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.666+0000 7efd25600640 1 -- 172.21.15.6:0/1511121730 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18034070 con 0x7efd28154240 2024-03-22T13:05:29.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.666+0000 7efd25600640 1 -- 172.21.15.6:0/1511121730 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd18030880 con 0x7efd28154240 2024-03-22T13:05:29.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.666+0000 7efd25600640 1 -- 172.21.15.6:0/1511121730 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18030c50 con 0x7efd28154240 2024-03-22T13:05:29.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.667+0000 7efd2de00640 1 -- 172.21.15.6:0/1511121730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 msgr2=0x7efd2815ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:29.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.667+0000 7efd2de00640 1 --2- 172.21.15.6:0/1511121730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815ac90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efd180097d0 tx=0x7efd1802b9c0 comp rx=0 tx=0).stop 2024-03-22T13:05:29.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.667+0000 7efd2de00640 1 -- 172.21.15.6:0/1511121730 shutdown_connections 2024-03-22T13:05:29.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.667+0000 7efd2de00640 1 --2- 172.21.15.6:0/1511121730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815ac90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:29.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.667+0000 7efd2de00640 1 -- 172.21.15.6:0/1511121730 >> 172.21.15.6:0/1511121730 conn(0x7efd280cb930 msgr2=0x7efd280cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.667+0000 7efd2de00640 1 -- 172.21.15.6:0/1511121730 shutdown_connections 2024-03-22T13:05:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.667+0000 7efd2de00640 1 -- 172.21.15.6:0/1511121730 wait complete. 2024-03-22T13:05:29.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.669+0000 7efd2de00640 1 Processor -- start 2024-03-22T13:05:29.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.669+0000 7efd2de00640 1 -- start start 2024-03-22T13:05:29.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.670+0000 7efd2de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd280c68c0 0x7efd281f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:29.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.670+0000 7efd2de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd280c7ee0 0x7efd28151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:29.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.670+0000 7efd2de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:29.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.670+0000 7efd2de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2815dae0 con 0x7efd280c68c0 2024-03-22T13:05:29.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.670+0000 7efd2de00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2815d800 con 0x7efd280c7ee0 2024-03-22T13:05:29.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.670+0000 7efd2de00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2815ddc0 con 0x7efd28154240 2024-03-22T13:05:29.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.671+0000 7efd26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd280c68c0 0x7efd281f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:29.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.671+0000 7efd27400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:29.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.671+0000 7efd26000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd280c7ee0 0x7efd28151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:29.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.671+0000 7efd26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd280c68c0 0x7efd281f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54662/0 (socket says 172.21.15.6:54662) 2024-03-22T13:05:29.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.671+0000 7efd26a00640 1 -- 172.21.15.6:0/1824710031 learned_addr learned my addr 172.21.15.6:0/1824710031 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:29.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.675+0000 7efd26a00640 1 -- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 msgr2=0x7efd2815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:29.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.675+0000 7efd26a00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:29.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.675+0000 7efd26a00640 1 -- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd280c7ee0 msgr2=0x7efd28151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:29.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.675+0000 7efd26a00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd280c7ee0 0x7efd28151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:29.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.675+0000 7efd26a00640 1 -- 172.21.15.6:0/1824710031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28151ea0 con 0x7efd280c68c0 2024-03-22T13:05:29.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.675+0000 7efd26000640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd280c7ee0 0x7efd28151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:05:29.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.677+0000 7efd26a00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd280c68c0 0x7efd281f1440 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd1802bfd0 tx=0x7efd180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:29.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.682+0000 7efd1f400640 1 -- 172.21.15.6:0/1824710031 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18034040 con 0x7efd280c68c0 2024-03-22T13:05:29.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.682+0000 7efd1f400640 1 -- 172.21.15.6:0/1824710031 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1803ad20 con 0x7efd280c68c0 2024-03-22T13:05:29.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.682+0000 7efd1f400640 1 -- 172.21.15.6:0/1824710031 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18044630 con 0x7efd280c68c0 2024-03-22T13:05:29.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.682+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd28152140 con 0x7efd280c68c0 2024-03-22T13:05:29.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.682+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd28150860 con 0x7efd280c68c0 2024-03-22T13:05:29.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.684+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd280756c0 con 0x7efd280c68c0 2024-03-22T13:05:29.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.720+0000 7efd1f400640 1 -- 172.21.15.6:0/1824710031 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd18037070 con 0x7efd280c68c0 2024-03-22T13:05:29.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.724+0000 7efd1f400640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:29.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.724+0000 7efd26000640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:29.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:29.725+0000 7efd26000640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efd14007fd0 tx=0x7efd14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:30.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.034+0000 7efd1f400640 1 -- 172.21.15.6:0/1824710031 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7efd18079560 con 0x7efd280c68c0 2024-03-22T13:05:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.035+0000 7efd1f400640 1 -- 172.21.15.6:0/1824710031 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd1803f070 con 0x7efd280c68c0 2024-03-22T13:05:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.323+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd2814a710 con 0x7efd280c68c0 2024-03-22T13:05:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.351+0000 7efd1f400640 1 -- 172.21.15.6:0/1824710031 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7efd1807b4c0 con 0x7efd280c68c0 2024-03-22T13:05:30.351 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:30.352 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:30.352 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 msgr2=0x7efd0806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efd14007fd0 tx=0x7efd14009040 comp rx=0 tx=0).stop 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd280c68c0 msgr2=0x7efd281f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd280c68c0 0x7efd281f1440 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd1802bfd0 tx=0x7efd180046c0 comp rx=0 tx=0).stop 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 shutdown_connections 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd28154240 0x7efd2815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd0806bc20 0x7efd0806e0e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd280c7ee0 0x7efd28151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 --2- 172.21.15.6:0/1824710031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd280c68c0 0x7efd281f1440 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.354+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 >> 172.21.15.6:0/1824710031 conn(0x7efd280cb930 msgr2=0x7efd280cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.355+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 shutdown_connections 2024-03-22T13:05:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:30.355+0000 7efd2de00640 1 -- 172.21.15.6:0/1824710031 wait complete. 2024-03-22T13:05:30.975 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:30.976 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:31.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:05:31.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:05:31.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:32.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.173+0000 7f1209e00640 1 Processor -- start 2024-03-22T13:05:32.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.173+0000 7f1209e00640 1 -- start start 2024-03-22T13:05:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.175+0000 7f1209e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1204152240 0x7f12041582a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:32.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.175+0000 7f1209e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040cad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:32.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.175+0000 7f1209e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f12040c9070 0x7f12040cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:32.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.175+0000 7f1209e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040ca8b0 con 0x7f1204152240 2024-03-22T13:05:32.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.175+0000 7f1209e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1204158b50 con 0x7f12040c9070 2024-03-22T13:05:32.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.175+0000 7f1209e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1204158e30 con 0x7f12040c8c30 2024-03-22T13:05:32.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.176+0000 7f1202000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:32.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.176+0000 7f1202a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1204152240 0x7f12041582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:32.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.176+0000 7f1203400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f12040c9070 0x7f12040cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:32.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.176+0000 7f1202000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56820/0 (socket says 172.21.15.6:56820) 2024-03-22T13:05:32.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.176+0000 7f1202000640 1 -- 172.21.15.6:0/3117464493 learned_addr learned my addr 172.21.15.6:0/3117464493 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:32.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.179+0000 7f1202000640 1 -- 172.21.15.6:0/3117464493 >> v2:172.21.15.92:3300/0 conn(0x7f12040c9070 msgr2=0x7f12040cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:32.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.179+0000 7f1202000640 1 --2- 172.21.15.6:0/3117464493 >> v2:172.21.15.92:3300/0 conn(0x7f12040c9070 0x7f12040cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.179+0000 7f1202000640 1 -- 172.21.15.6:0/3117464493 >> v2:172.21.15.6:3300/0 conn(0x7f1204152240 msgr2=0x7f12041582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:32.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.179+0000 7f1202000640 1 --2- 172.21.15.6:0/3117464493 >> v2:172.21.15.6:3300/0 conn(0x7f1204152240 0x7f12041582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.179+0000 7f1202000640 1 -- 172.21.15.6:0/3117464493 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204159110 con 0x7f12040c8c30 2024-03-22T13:05:32.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.181+0000 7f1202000640 1 --2- 172.21.15.6:0/3117464493 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040cad40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1204068e60 tx=0x7f11f002b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42d2de7bb8cecd57 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:32.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.186+0000 7f1201600640 1 -- 172.21.15.6:0/3117464493 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0034070 con 0x7f12040c8c30 2024-03-22T13:05:32.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.186+0000 7f1201600640 1 -- 172.21.15.6:0/3117464493 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f0030860 con 0x7f12040c8c30 2024-03-22T13:05:32.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.186+0000 7f1201600640 1 -- 172.21.15.6:0/3117464493 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0030c30 con 0x7f12040c8c30 2024-03-22T13:05:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 -- 172.21.15.6:0/3117464493 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 msgr2=0x7f12040cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 --2- 172.21.15.6:0/3117464493 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040cad40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1204068e60 tx=0x7f11f002b9a0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 -- 172.21.15.6:0/3117464493 shutdown_connections 2024-03-22T13:05:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 --2- 172.21.15.6:0/3117464493 >> v2:172.21.15.92:3300/0 conn(0x7f12040c9070 0x7f12040cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 --2- 172.21.15.6:0/3117464493 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040cad40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 --2- 172.21.15.6:0/3117464493 >> v2:172.21.15.6:3300/0 conn(0x7f1204152240 0x7f12041582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 -- 172.21.15.6:0/3117464493 >> 172.21.15.6:0/3117464493 conn(0x7f1204071420 msgr2=0x7f1204071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:32.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 -- 172.21.15.6:0/3117464493 shutdown_connections 2024-03-22T13:05:32.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.187+0000 7f1209e00640 1 -- 172.21.15.6:0/3117464493 wait complete. 2024-03-22T13:05:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.189+0000 7f1209e00640 1 Processor -- start 2024-03-22T13:05:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.189+0000 7f1209e00640 1 -- start start 2024-03-22T13:05:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.190+0000 7f1209e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.190+0000 7f1209e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12040c9070 0x7f12040cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.190+0000 7f1209e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1204152240 0x7f12040cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.190+0000 7f1209e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041587e0 con 0x7f1204152240 2024-03-22T13:05:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.190+0000 7f1209e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040ca540 con 0x7f12040c9070 2024-03-22T13:05:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.190+0000 7f1209e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1204152620 con 0x7f12040c8c30 2024-03-22T13:05:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.191+0000 7f1202a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.191+0000 7f1203400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1204152240 0x7f12040cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.191+0000 7f1202a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:56830/0 (socket says 172.21.15.6:56830) 2024-03-22T13:05:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.191+0000 7f1202a00640 1 -- 172.21.15.6:0/710304003 learned_addr learned my addr 172.21.15.6:0/710304003 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.191+0000 7f1202000640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12040c9070 0x7f12040cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.195+0000 7f1202a00640 1 -- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12040c9070 msgr2=0x7f12040cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.195+0000 7f1202a00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12040c9070 0x7f12040cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.195+0000 7f1202a00640 1 -- 172.21.15.6:0/710304003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1204152240 msgr2=0x7f12040cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.195+0000 7f1202a00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1204152240 0x7f12040cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.195+0000 7f1202a00640 1 -- 172.21.15.6:0/710304003 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040cbc60 con 0x7f12040c8c30 2024-03-22T13:05:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.198+0000 7f1202a00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040ce040 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e8a0 tx=0x7f11f400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.202+0000 7f11fb400640 1 -- 172.21.15.6:0/710304003 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4017070 con 0x7f12040c8c30 2024-03-22T13:05:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.202+0000 7f11fb400640 1 -- 172.21.15.6:0/710304003 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f400c940 con 0x7f12040c8c30 2024-03-22T13:05:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.202+0000 7f11fb400640 1 -- 172.21.15.6:0/710304003 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400cd30 con 0x7f12040c8c30 2024-03-22T13:05:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.202+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120414cad0 con 0x7f12040c8c30 2024-03-22T13:05:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.202+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120414d090 con 0x7f12040c8c30 2024-03-22T13:05:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.203+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12040756c0 con 0x7f12040c8c30 2024-03-22T13:05:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.224+0000 7f11fb400640 1 -- 172.21.15.6:0/710304003 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11f401a070 con 0x7f12040c8c30 2024-03-22T13:05:32.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.225+0000 7f11fb400640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11e406bc20 0x7f11e406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:32.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.226+0000 7f1202000640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11e406bc20 0x7f11e406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:32.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.227+0000 7f1202000640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11e406bc20 0x7f11e406e0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f11f0036000 tx=0x7f11f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:32.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.465+0000 7f11fb400640 1 -- 172.21.15.6:0/710304003 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f11f40575a0 con 0x7f12040c8c30 2024-03-22T13:05:32.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.476+0000 7f11fb400640 1 -- 172.21.15.6:0/710304003 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11f4059500 con 0x7f12040c8c30 2024-03-22T13:05:32.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.762+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f120414d2d0 con 0x7f12040c8c30 2024-03-22T13:05:32.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.778+0000 7f11fb400640 1 -- 172.21.15.6:0/710304003 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f11f405d5a0 con 0x7f12040c8c30 2024-03-22T13:05:32.778 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:32.778 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:32.778 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11e406bc20 msgr2=0x7f11e406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11e406bc20 0x7f11e406e0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f11f0036000 tx=0x7f11f0008030 comp rx=0 tx=0).stop 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 msgr2=0x7f12040ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040ce040 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e8a0 tx=0x7f11f400ed60 comp rx=0 tx=0).stop 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 shutdown_connections 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1204152240 0x7f12040cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f11e406bc20 0x7f11e406e0e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12040c9070 0x7f12040cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 --2- 172.21.15.6:0/710304003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12040c8c30 0x7f12040ce040 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.781+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 >> 172.21.15.6:0/710304003 conn(0x7f1204071420 msgr2=0x7f120406d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.782+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 shutdown_connections 2024-03-22T13:05:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:32.782+0000 7f1209e00640 1 -- 172.21.15.6:0/710304003 wait complete. 2024-03-22T13:05:33.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:33.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:34.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:34.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.605+0000 7f138de00640 1 Processor -- start 2024-03-22T13:05:34.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.605+0000 7f138de00640 1 -- start start 2024-03-22T13:05:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.606+0000 7f138de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.606+0000 7f138de00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f138815ae30 0x7f13880cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.606+0000 7f138de00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138815b260 con 0x7f13880cc380 2024-03-22T13:05:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.606+0000 7f138de00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13880cee00 con 0x7f138815ae30 2024-03-22T13:05:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.606+0000 7f138de00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13880cf0c0 con 0x7f13880ce8a0 2024-03-22T13:05:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.607+0000 7f1387e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.608+0000 7f1387e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138806c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46662/0 (socket says 172.21.15.6:46662) 2024-03-22T13:05:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.608+0000 7f1387e00640 1 -- 172.21.15.6:0/2206510150 learned_addr learned my addr 172.21.15.6:0/2206510150 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.611+0000 7f1387e00640 1 -- 172.21.15.6:0/2206510150 >> v2:172.21.15.92:3300/0 conn(0x7f138815ae30 msgr2=0x7f13880cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.611+0000 7f1387e00640 1 --2- 172.21.15.6:0/2206510150 >> v2:172.21.15.92:3300/0 conn(0x7f138815ae30 0x7f13880cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.611+0000 7f1387e00640 1 -- 172.21.15.6:0/2206510150 >> v1:172.21.15.6:6789/0 conn(0x7f13880cc380 legacy=0x7f13880cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.611+0000 7f1387e00640 1 -- 172.21.15.6:0/2206510150 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13880cf3a0 con 0x7f13880ce8a0 2024-03-22T13:05:34.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.614+0000 7f1387e00640 1 --2- 172.21.15.6:0/2206510150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138806c230 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1374009870 tx=0x7f137402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf05a4505aecaad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:34.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.619+0000 7f1387400640 1 -- 172.21.15.6:0/2206510150 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1374035020 con 0x7f13880ce8a0 2024-03-22T13:05:34.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.619+0000 7f1387400640 1 -- 172.21.15.6:0/2206510150 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1374031640 con 0x7f13880ce8a0 2024-03-22T13:05:34.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.619+0000 7f1387400640 1 -- 172.21.15.6:0/2206510150 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137402d040 con 0x7f13880ce8a0 2024-03-22T13:05:34.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.619+0000 7f138de00640 1 -- 172.21.15.6:0/2206510150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 msgr2=0x7f138806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:34.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.619+0000 7f138de00640 1 --2- 172.21.15.6:0/2206510150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138806c230 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1374009870 tx=0x7f137402b8f0 comp rx=0 tx=0).stop 2024-03-22T13:05:34.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.620+0000 7f138de00640 1 -- 172.21.15.6:0/2206510150 shutdown_connections 2024-03-22T13:05:34.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.620+0000 7f138de00640 1 --2- 172.21.15.6:0/2206510150 >> v2:172.21.15.92:3300/0 conn(0x7f138815ae30 0x7f13880cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:34.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.620+0000 7f138de00640 1 --2- 172.21.15.6:0/2206510150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138806c230 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:34.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.620+0000 7f138de00640 1 -- 172.21.15.6:0/2206510150 >> 172.21.15.6:0/2206510150 conn(0x7f13880b1480 msgr2=0x7f13880b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:34.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.620+0000 7f138de00640 1 -- 172.21.15.6:0/2206510150 shutdown_connections 2024-03-22T13:05:34.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.620+0000 7f138de00640 1 -- 172.21.15.6:0/2206510150 wait complete. 2024-03-22T13:05:34.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.622+0000 7f138de00640 1 Processor -- start 2024-03-22T13:05:34.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.622+0000 7f138de00640 1 -- start start 2024-03-22T13:05:34.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.623+0000 7f138de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13880cc380 0x7f13880b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:34.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.623+0000 7f138de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138815ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:34.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.623+0000 7f138de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f138815ae30 0x7f138815b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:34.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.623+0000 7f138de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138815a910 con 0x7f138815ae30 2024-03-22T13:05:34.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.623+0000 7f138de00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138815a630 con 0x7f13880cc380 2024-03-22T13:05:34.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.623+0000 7f138de00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13881549d0 con 0x7f13880ce8a0 2024-03-22T13:05:34.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.623+0000 7f1387e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138815ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:34.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.624+0000 7f1387e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138815ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46664/0 (socket says 172.21.15.6:46664) 2024-03-22T13:05:34.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.624+0000 7f1387e00640 1 -- 172.21.15.6:0/657453089 learned_addr learned my addr 172.21.15.6:0/657453089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:34.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.629+0000 7f1387e00640 1 -- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13880cc380 msgr2=0x7f13880b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:34.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.629+0000 7f1387e00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13880cc380 0x7f13880b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:34.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.629+0000 7f1387e00640 1 -- 172.21.15.6:0/657453089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f138815ae30 msgr2=0x7f138815b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:34.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.629+0000 7f1387e00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f138815ae30 0x7f138815b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:34.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.629+0000 7f1387e00640 1 -- 172.21.15.6:0/657453089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1388153ad0 con 0x7f13880ce8a0 2024-03-22T13:05:34.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.633+0000 7f1387e00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138815ddf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f137402b930 tx=0x7f13740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.638+0000 7f1384c00640 1 -- 172.21.15.6:0/657453089 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137403ac60 con 0x7f13880ce8a0 2024-03-22T13:05:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.638+0000 7f1384c00640 1 -- 172.21.15.6:0/657453089 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137403ae20 con 0x7f13880ce8a0 2024-03-22T13:05:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.638+0000 7f1384c00640 1 -- 172.21.15.6:0/657453089 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1374044630 con 0x7f13880ce8a0 2024-03-22T13:05:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.638+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1388153d90 con 0x7f13880ce8a0 2024-03-22T13:05:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.638+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1388152210 con 0x7f13880ce8a0 2024-03-22T13:05:34.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.640+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1388075650 con 0x7f13880ce8a0 2024-03-22T13:05:34.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.675+0000 7f1384c00640 1 -- 172.21.15.6:0/657453089 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f137404b060 con 0x7f13880ce8a0 2024-03-22T13:05:34.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.676+0000 7f1384c00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f136806bc20 0x7f136806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:34.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.676+0000 7f138ca00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f136806bc20 0x7f136806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:34.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.677+0000 7f138ca00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f136806bc20 0x7f136806e0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f137c009650 tx=0x7f137c006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:34.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.680+0000 7f1384c00640 1 -- 172.21.15.6:0/657453089 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f137404be30 con 0x7f13880ce8a0 2024-03-22T13:05:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:34.946+0000 7f1384c00640 1 -- 172.21.15.6:0/657453089 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f137403f070 con 0x7f13880ce8a0 2024-03-22T13:05:35.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.239+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f138815e680 con 0x7f13880ce8a0 2024-03-22T13:05:35.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.256+0000 7f1384c00640 1 -- 172.21.15.6:0/657453089 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f138815b260 con 0x7f13880ce8a0 2024-03-22T13:05:35.256 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:35.256 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:35.256 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.259+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f136806bc20 msgr2=0x7f136806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.259+0000 7f138de00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f136806bc20 0x7f136806e0e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f137c009650 tx=0x7f137c006ba0 comp rx=0 tx=0).stop 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.259+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 msgr2=0x7f138815ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.259+0000 7f138de00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138815ddf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f137402b930 tx=0x7f13740046c0 comp rx=0 tx=0).stop 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 shutdown_connections 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f138815ae30 0x7f138815b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13880ce8a0 0x7f138815ddf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f136806bc20 0x7f136806e0e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 --2- 172.21.15.6:0/657453089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13880cc380 0x7f13880b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 >> 172.21.15.6:0/657453089 conn(0x7f13880b1480 msgr2=0x7f13880c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 shutdown_connections 2024-03-22T13:05:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:35.260+0000 7f138de00640 1 -- 172.21.15.6:0/657453089 wait complete. 2024-03-22T13:05:35.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:35.881 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:36.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.082+0000 7fed62a00640 1 Processor -- start 2024-03-22T13:05:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.082+0000 7fed62a00640 1 -- start start 2024-03-22T13:05:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.083+0000 7fed62a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed5c0cc380 0x7fed5c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.083+0000 7fed62a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5c15b240 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.083+0000 7fed62a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5c0cee00 con 0x7fed5c15ae30 2024-03-22T13:05:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.083+0000 7fed62a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5c0cf0c0 con 0x7fed5c0cc380 2024-03-22T13:05:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.084+0000 7fed61600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fed5c0cc380 0x7fed5c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.084+0000 7fed60c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fed5c0ce8a0 0x7fed5c0cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38252/0 (socket says 172.21.15.6:38252) 2024-03-22T13:05:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.084+0000 7fed60c00640 1 -- 172.21.15.6:0/832077259 learned_addr learned my addr 172.21.15.6:0/832077259 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.088+0000 7fed5be00640 1 -- 172.21.15.6:0/832077259 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108008712 0 0) 0x7fed5c15b240 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.088+0000 7fed5be00640 1 -- 172.21.15.6:0/832077259 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed44003520 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.089+0000 7fed61600640 1 -- 172.21.15.6:0/832077259 >> v1:172.21.15.92:6789/0 conn(0x7fed5c15ae30 legacy=0x7fed5c164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.089+0000 7fed61600640 1 -- 172.21.15.6:0/832077259 >> v1:172.21.15.6:6789/0 conn(0x7fed5c0ce8a0 legacy=0x7fed5c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.089+0000 7fed61600640 1 -- 172.21.15.6:0/832077259 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c0cf3a0 con 0x7fed5c0cc380 2024-03-22T13:05:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.095+0000 7fed61600640 1 --2- 172.21.15.6:0/832077259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 0x7fed5c06c230 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed500097a0 tx=0x7fed5002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f78bfba63064a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.099+0000 7fed5be00640 1 -- 172.21.15.6:0/832077259 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed50034070 con 0x7fed5c0cc380 2024-03-22T13:05:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.099+0000 7fed5be00640 1 -- 172.21.15.6:0/832077259 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed50030610 con 0x7fed5c0cc380 2024-03-22T13:05:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.099+0000 7fed5be00640 1 -- 172.21.15.6:0/832077259 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed500309e0 con 0x7fed5c0cc380 2024-03-22T13:05:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.100+0000 7fed62a00640 1 -- 172.21.15.6:0/832077259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 msgr2=0x7fed5c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.100+0000 7fed62a00640 1 --2- 172.21.15.6:0/832077259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 0x7fed5c06c230 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed500097a0 tx=0x7fed5002b960 comp rx=0 tx=0).stop 2024-03-22T13:05:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.100+0000 7fed62a00640 1 -- 172.21.15.6:0/832077259 shutdown_connections 2024-03-22T13:05:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.100+0000 7fed62a00640 1 --2- 172.21.15.6:0/832077259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 0x7fed5c06c230 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.100+0000 7fed62a00640 1 -- 172.21.15.6:0/832077259 >> 172.21.15.6:0/832077259 conn(0x7fed5c0b1480 msgr2=0x7fed5c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.100+0000 7fed62a00640 1 -- 172.21.15.6:0/832077259 shutdown_connections 2024-03-22T13:05:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.100+0000 7fed62a00640 1 -- 172.21.15.6:0/832077259 wait complete. 2024-03-22T13:05:37.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.102+0000 7fed62a00640 1 Processor -- start 2024-03-22T13:05:37.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.102+0000 7fed62a00640 1 -- start start 2024-03-22T13:05:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.103+0000 7fed62a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 0x7fed5c1e6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.103+0000 7fed62a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed5c0ce8a0 0x7fed5c15be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.103+0000 7fed62a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed5c15ae30 0x7fed5c152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.103+0000 7fed62a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c15a910 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.103+0000 7fed62a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c15a630 con 0x7fed5c0cc380 2024-03-22T13:05:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.103+0000 7fed62a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c1549d0 con 0x7fed5c15ae30 2024-03-22T13:05:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.104+0000 7fed60c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed5c0ce8a0 0x7fed5c15be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.104+0000 7fed61600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 0x7fed5c1e6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.104+0000 7fed62000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed5c15ae30 0x7fed5c152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.104+0000 7fed60c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed5c0ce8a0 0x7fed5c15be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41566/0 (socket says 172.21.15.6:41566) 2024-03-22T13:05:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.104+0000 7fed60c00640 1 -- 172.21.15.6:0/3864140430 learned_addr learned my addr 172.21.15.6:0/3864140430 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.108+0000 7fed60c00640 1 -- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed5c15ae30 msgr2=0x7fed5c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.108+0000 7fed60c00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed5c15ae30 0x7fed5c152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.108+0000 7fed60c00640 1 -- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 msgr2=0x7fed5c1e6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.108+0000 7fed60c00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 0x7fed5c1e6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.108+0000 7fed60c00640 1 -- 172.21.15.6:0/3864140430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c1496d0 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.111+0000 7fed60c00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed5c0ce8a0 0x7fed5c15be30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fed4800c460 tx=0x7fed4800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.113+0000 7fed59600640 1 -- 172.21.15.6:0/3864140430 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48017070 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.113+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5c1541b0 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.113+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5c149c30 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.115+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5c075650 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.450+0000 7fed59600640 1 -- 172.21.15.6:0/3864140430 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48005ce0 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.450+0000 7fed59600640 1 -- 172.21.15.6:0/3864140430 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48021060 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.452+0000 7fed59600640 1 -- 172.21.15.6:0/3864140430 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fed4801a070 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.453+0000 7fed59600640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed3c06bbd0 0x7fed3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:37.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.453+0000 7fed61600640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed3c06bbd0 0x7fed3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:37.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.454+0000 7fed61600640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed3c06bbd0 0x7fed3c06e090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fed50036000 tx=0x7fed5003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:37.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.475+0000 7fed59600640 1 -- 172.21.15.6:0/3864140430 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fed48056ad0 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.475+0000 7fed59600640 1 -- 172.21.15.6:0/3864140430 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fed5c1496d0 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.784+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed5c14ad60 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.803+0000 7fed59600640 1 -- 172.21.15.6:0/3864140430 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fed48058a30 con 0x7fed5c0ce8a0 2024-03-22T13:05:37.803 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:37.803 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:37.803 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.805+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed3c06bbd0 msgr2=0x7fed3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.805+0000 7fed62a00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed3c06bbd0 0x7fed3c06e090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fed50036000 tx=0x7fed5003b040 comp rx=0 tx=0).stop 2024-03-22T13:05:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.805+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed5c0ce8a0 msgr2=0x7fed5c15be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.805+0000 7fed62a00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed5c0ce8a0 0x7fed5c15be30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fed4800c460 tx=0x7fed4800c920 comp rx=0 tx=0).stop 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 shutdown_connections 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fed5c15ae30 0x7fed5c152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fed5c0ce8a0 0x7fed5c15be30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fed3c06bbd0 0x7fed3c06e090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 --2- 172.21.15.6:0/3864140430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5c0cc380 0x7fed5c1e6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 >> 172.21.15.6:0/3864140430 conn(0x7fed5c0b1480 msgr2=0x7fed5c1e35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 shutdown_connections 2024-03-22T13:05:37.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:37.806+0000 7fed62a00640 1 -- 172.21.15.6:0/3864140430 wait complete. 2024-03-22T13:05:38.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:38.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:39.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:39.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.626+0000 7f5f5ce00640 1 Processor -- start 2024-03-22T13:05:39.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.627+0000 7f5f5ce00640 1 -- start start 2024-03-22T13:05:39.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.628+0000 7f5f5ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f580ccad0 0x7f5f5806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:39.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.628+0000 7f5f5ce00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5806c7a0 con 0x7f5f5815b0e0 2024-03-22T13:05:39.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.628+0000 7f5f5ce00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5806ca80 con 0x7f5f580ca520 2024-03-22T13:05:39.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.628+0000 7f5f5ce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f580cd000 con 0x7f5f580ccad0 2024-03-22T13:05:39.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.629+0000 7f5f55e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5f580ca520 0x7f5f580cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48492/0 (socket says 172.21.15.6:48492) 2024-03-22T13:05:39.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.629+0000 7f5f55e00640 1 -- 172.21.15.6:0/698736791 learned_addr learned my addr 172.21.15.6:0/698736791 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:39.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.629+0000 7f5f55400640 1 --2- 172.21.15.6:0/698736791 >> v2:172.21.15.6:3300/0 conn(0x7f5f580ccad0 0x7f5f5806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:39.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.633+0000 7f5f54a00640 1 -- 172.21.15.6:0/698736791 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870611377 0 0) 0x7f5f5806ca80 con 0x7f5f580ca520 2024-03-22T13:05:39.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.633+0000 7f5f54a00640 1 -- 172.21.15.6:0/698736791 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3c003520 con 0x7f5f580ca520 2024-03-22T13:05:39.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.633+0000 7f5f55400640 1 -- 172.21.15.6:0/698736791 >> v1:172.21.15.92:6789/0 conn(0x7f5f580ca520 legacy=0x7f5f580cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:39.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.633+0000 7f5f55400640 1 -- 172.21.15.6:0/698736791 >> v1:172.21.15.6:6789/0 conn(0x7f5f5815b0e0 legacy=0x7f5f58164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:39.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.633+0000 7f5f55400640 1 -- 172.21.15.6:0/698736791 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5806cd60 con 0x7f5f580ccad0 2024-03-22T13:05:39.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.637+0000 7f5f55400640 1 --2- 172.21.15.6:0/698736791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 0x7f5f5806d840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5f440097a0 tx=0x7f5f4402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46ae06937bd4f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:39.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.643+0000 7f5f54a00640 1 -- 172.21.15.6:0/698736791 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44034070 con 0x7f5f580ccad0 2024-03-22T13:05:39.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.643+0000 7f5f54a00640 1 -- 172.21.15.6:0/698736791 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f44030610 con 0x7f5f580ccad0 2024-03-22T13:05:39.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.643+0000 7f5f54a00640 1 -- 172.21.15.6:0/698736791 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440309e0 con 0x7f5f580ccad0 2024-03-22T13:05:39.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.643+0000 7f5f5ce00640 1 -- 172.21.15.6:0/698736791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 msgr2=0x7f5f5806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:39.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.643+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/698736791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 0x7f5f5806d840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5f440097a0 tx=0x7f5f4402b960 comp rx=0 tx=0).stop 2024-03-22T13:05:39.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.644+0000 7f5f5ce00640 1 -- 172.21.15.6:0/698736791 shutdown_connections 2024-03-22T13:05:39.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.644+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/698736791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 0x7f5f5806d840 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:39.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.644+0000 7f5f5ce00640 1 -- 172.21.15.6:0/698736791 >> 172.21.15.6:0/698736791 conn(0x7f5f580b16b0 msgr2=0x7f5f580b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:39.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.644+0000 7f5f5ce00640 1 -- 172.21.15.6:0/698736791 shutdown_connections 2024-03-22T13:05:39.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.644+0000 7f5f5ce00640 1 -- 172.21.15.6:0/698736791 wait complete. 2024-03-22T13:05:39.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.646+0000 7f5f5ce00640 1 Processor -- start 2024-03-22T13:05:39.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.646+0000 7f5f5ce00640 1 -- start start 2024-03-22T13:05:39.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.647+0000 7f5f5ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f580ca520 0x7f5f58157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:39.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.647+0000 7f5f5ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 0x7f5f580b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:39.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.647+0000 7f5f5ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f5815b0e0 0x7f5f581e7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:39.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.647+0000 7f5f5ce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f58154c20 con 0x7f5f580ccad0 2024-03-22T13:05:39.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.647+0000 7f5f5ce00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5806d320 con 0x7f5f5815b0e0 2024-03-22T13:05:39.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.647+0000 7f5f5ce00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5815b980 con 0x7f5f580ca520 2024-03-22T13:05:39.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.648+0000 7f5f56800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f5815b0e0 0x7f5f581e7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:39.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.648+0000 7f5f55400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 0x7f5f580b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:39.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.648+0000 7f5f55e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f580ca520 0x7f5f58157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:39.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.648+0000 7f5f56800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f5815b0e0 0x7f5f581e7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47708/0 (socket says 172.21.15.6:47708) 2024-03-22T13:05:39.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.648+0000 7f5f56800640 1 -- 172.21.15.6:0/1629937428 learned_addr learned my addr 172.21.15.6:0/1629937428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:39.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.654+0000 7f5f56800640 1 -- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f580ca520 msgr2=0x7f5f58157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:39.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.654+0000 7f5f56800640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f580ca520 0x7f5f58157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:39.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.654+0000 7f5f56800640 1 -- 172.21.15.6:0/1629937428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 msgr2=0x7f5f580b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:39.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.654+0000 7f5f56800640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 0x7f5f580b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:39.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.654+0000 7f5f56800640 1 -- 172.21.15.6:0/1629937428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f581f3c60 con 0x7f5f5815b0e0 2024-03-22T13:05:39.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.657+0000 7f5f56800640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f5815b0e0 0x7f5f581e7300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c6b0 tx=0x7f5f5000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:39.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.662+0000 7f5f4e000640 1 -- 172.21.15.6:0/1629937428 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50003b70 con 0x7f5f5815b0e0 2024-03-22T13:05:39.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.662+0000 7f5f4e000640 1 -- 172.21.15.6:0/1629937428 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f50003d30 con 0x7f5f5815b0e0 2024-03-22T13:05:39.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.662+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5815a350 con 0x7f5f5815b0e0 2024-03-22T13:05:39.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.662+0000 7f5f4e000640 1 -- 172.21.15.6:0/1629937428 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50005310 con 0x7f5f5815b0e0 2024-03-22T13:05:39.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.662+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f580b55e0 con 0x7f5f5815b0e0 2024-03-22T13:05:39.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.663+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f580756c0 con 0x7f5f5815b0e0 2024-03-22T13:05:39.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.695+0000 7f5f4e000640 1 -- 172.21.15.6:0/1629937428 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5f500054b0 con 0x7f5f5815b0e0 2024-03-22T13:05:39.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.697+0000 7f5f4e000640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f3806bc50 0x7f5f3806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:39.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.697+0000 7f5f55e00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f3806bc50 0x7f5f3806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:39.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.698+0000 7f5f55e00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f3806bc50 0x7f5f3806e110 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5f480011e0 tx=0x7f5f48007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:39.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.959+0000 7f5f4e000640 1 -- 172.21.15.6:0/1629937428 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5f500057a0 con 0x7f5f5815b0e0 2024-03-22T13:05:39.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:39.970+0000 7f5f4e000640 1 -- 172.21.15.6:0/1629937428 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5f50059700 con 0x7f5f5815b0e0 2024-03-22T13:05:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.231+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f580cb080 con 0x7f5f5815b0e0 2024-03-22T13:05:40.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.312+0000 7f5f4e000640 1 -- 172.21.15.6:0/1629937428 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f5f5005d7a0 con 0x7f5f5815b0e0 2024-03-22T13:05:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:40.313 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:40.313 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f3806bc50 msgr2=0x7f5f3806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f3806bc50 0x7f5f3806e110 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5f480011e0 tx=0x7f5f48007d30 comp rx=0 tx=0).stop 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f5815b0e0 msgr2=0x7f5f581e7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f5815b0e0 0x7f5f581e7300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c6b0 tx=0x7f5f5000cb70 comp rx=0 tx=0).stop 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 shutdown_connections 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5f3806bc50 0x7f5f3806e110 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f5815b0e0 0x7f5f581e7300 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f580ccad0 0x7f5f580b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 --2- 172.21.15.6:0/1629937428 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f580ca520 0x7f5f58157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:40.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 >> 172.21.15.6:0/1629937428 conn(0x7f5f580b16b0 msgr2=0x7f5f580b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:40.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 shutdown_connections 2024-03-22T13:05:40.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:40.315+0000 7f5f5ce00640 1 -- 172.21.15.6:0/1629937428 wait complete. 2024-03-22T13:05:40.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:40.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:41.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:42.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.132+0000 7f982fe00640 1 Processor -- start 2024-03-22T13:05:42.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.132+0000 7f982fe00640 1 -- start start 2024-03-22T13:05:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.133+0000 7f982fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f983006c230 0x7f983015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.133+0000 7f982fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983006c640 con 0x7f98300ce8a0 2024-03-22T13:05:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.133+0000 7f982fe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98300cee00 con 0x7f98300cc380 2024-03-22T13:05:42.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.133+0000 7f982fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98300cf0c0 con 0x7f983006c230 2024-03-22T13:05:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.134+0000 7f982ea00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f98300cc380 0x7f98300cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48494/0 (socket says 172.21.15.6:48494) 2024-03-22T13:05:42.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.134+0000 7f982ea00640 1 -- 172.21.15.6:0/2349728476 learned_addr learned my addr 172.21.15.6:0/2349728476 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:42.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.137+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931140658 0 0) 0x7f98300cee00 con 0x7f98300cc380 2024-03-22T13:05:42.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.137+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9814003520 con 0x7f98300cc380 2024-03-22T13:05:42.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1406326555 0 0) 0x7f9814003520 con 0x7f98300cc380 2024-03-22T13:05:42.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 >> v2:172.21.15.6:3300/0 conn(0x7f983006c230 msgr2=0x7f983015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:42.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 --2- 172.21.15.6:0/2349728476 >> v2:172.21.15.6:3300/0 conn(0x7f983006c230 0x7f983015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 >> v1:172.21.15.6:6789/0 conn(0x7f98300ce8a0 legacy=0x7f9830162b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:42.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98300cf3a0 con 0x7f98300cc380 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f9820001670 con 0x7f98300cc380 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 >> v1:172.21.15.92:6789/0 conn(0x7f98300cc380 legacy=0x7f98300cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140039b0 0x7f9814003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9814004800 0x7f9814004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98140052e0 0x7f981400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98300cf0c0 con 0x7f98140039b0 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981400c1e0 con 0x7f9814004800 2024-03-22T13:05:42.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.141+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981400c3e0 con 0x7f98140052e0 2024-03-22T13:05:42.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.142+0000 7f982f400640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98140052e0 0x7f981400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:42.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.143+0000 7f982e000640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9814004800 0x7f9814004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:42.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.144+0000 7f982ea00640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140039b0 0x7f9814003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:42.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.146+0000 7f982f400640 1 -- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9814004800 msgr2=0x7f9814004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:42.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.146+0000 7f982f400640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9814004800 0x7f9814004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.146+0000 7f982f400640 1 -- 172.21.15.6:0/2349728476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140039b0 msgr2=0x7f9814003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:42.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.146+0000 7f982f400640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140039b0 0x7f9814003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.146+0000 7f982f400640 1 -- 172.21.15.6:0/2349728476 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98300cf3a0 con 0x7f98140052e0 2024-03-22T13:05:42.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.147+0000 7f982f400640 1 -- 172.21.15.6:0/2349728476 reap_dead start 2024-03-22T13:05:42.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.149+0000 7f982f400640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98140052e0 0x7f981400b9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9828009e40 tx=0x7f9814005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e2d56e66c957fec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:42.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.153+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828009a10 con 0x7f98140052e0 2024-03-22T13:05:42.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.153+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9828006e50 con 0x7f98140052e0 2024-03-22T13:05:42.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.153+0000 7f982d600640 1 -- 172.21.15.6:0/2349728476 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828003970 con 0x7f98140052e0 2024-03-22T13:05:42.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.154+0000 7f982fe00640 1 -- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98140052e0 msgr2=0x7f981400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.154+0000 7f982fe00640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98140052e0 0x7f981400b9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9828009e40 tx=0x7f9814005830 comp rx=0 tx=0).stop 2024-03-22T13:05:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.154+0000 7f982fe00640 1 -- 172.21.15.6:0/2349728476 shutdown_connections 2024-03-22T13:05:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.154+0000 7f982fe00640 1 --2- 172.21.15.6:0/2349728476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98140052e0 0x7f981400b9c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.154+0000 7f982fe00640 1 -- 172.21.15.6:0/2349728476 >> 172.21.15.6:0/2349728476 conn(0x7f98300b1480 msgr2=0x7f98300b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.154+0000 7f982fe00640 1 -- 172.21.15.6:0/2349728476 shutdown_connections 2024-03-22T13:05:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.154+0000 7f982fe00640 1 -- 172.21.15.6:0/2349728476 wait complete. 2024-03-22T13:05:42.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.157+0000 7f982fe00640 1 Processor -- start 2024-03-22T13:05:42.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.157+0000 7f982fe00640 1 -- start start 2024-03-22T13:05:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.158+0000 7f982fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140052e0 0x7f98301f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.158+0000 7f982fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98300c34a0 0x7f983014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.158+0000 7f982fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98300c4390 0x7f98300c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.158+0000 7f982fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983015a910 con 0x7f98140052e0 2024-03-22T13:05:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.158+0000 7f982fe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983015a630 con 0x7f98300c34a0 2024-03-22T13:05:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.158+0000 7f982fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98301549d0 con 0x7f98300c4390 2024-03-22T13:05:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.159+0000 7f982ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140052e0 0x7f98301f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.159+0000 7f982f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98300c4390 0x7f98300c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.159+0000 7f982e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98300c34a0 0x7f983014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.160+0000 7f982f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98300c4390 0x7f98300c4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46694/0 (socket says 172.21.15.6:46694) 2024-03-22T13:05:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.160+0000 7f982f400640 1 -- 172.21.15.6:0/1289241584 learned_addr learned my addr 172.21.15.6:0/1289241584 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.164+0000 7f982e000640 1 -- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98300c4390 msgr2=0x7f98300c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.164+0000 7f982e000640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98300c4390 0x7f98300c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.164+0000 7f982e000640 1 -- 172.21.15.6:0/1289241584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140052e0 msgr2=0x7f98301f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.164+0000 7f982e000640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140052e0 0x7f98301f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.164+0000 7f982e000640 1 -- 172.21.15.6:0/1289241584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98301f4130 con 0x7f98300c34a0 2024-03-22T13:05:42.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.166+0000 7f982e000640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98300c34a0 0x7f983014e140 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f981c00f500 tx=0x7f981c00f9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:42.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.170+0000 7f9826a00640 1 -- 172.21.15.6:0/1289241584 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c01a070 con 0x7f98300c34a0 2024-03-22T13:05:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.171+0000 7f9826a00640 1 -- 172.21.15.6:0/1289241584 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c005320 con 0x7f98300c34a0 2024-03-22T13:05:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.171+0000 7f9826a00640 1 -- 172.21.15.6:0/1289241584 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c007270 con 0x7f98300c34a0 2024-03-22T13:05:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.171+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983014e680 con 0x7f98300c34a0 2024-03-22T13:05:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.171+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9830159410 con 0x7f98300c34a0 2024-03-22T13:05:42.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.172+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9830075650 con 0x7f98300c34a0 2024-03-22T13:05:42.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.341+0000 7f9826a00640 1 -- 172.21.15.6:0/1289241584 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f981c01d070 con 0x7f98300c34a0 2024-03-22T13:05:42.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.343+0000 7f9826a00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f981006bc20 0x7f981006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:42.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.343+0000 7f982ea00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f981006bc20 0x7f981006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:42.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.344+0000 7f982ea00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f981006bc20 0x7f981006e0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9820001eb0 tx=0x7f9820009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:42.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.451+0000 7f9826a00640 1 -- 172.21.15.6:0/1289241584 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f981c056b30 con 0x7f98300c34a0 2024-03-22T13:05:42.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.470+0000 7f9826a00640 1 -- 172.21.15.6:0/1289241584 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f981c024020 con 0x7f98300c34a0 2024-03-22T13:05:42.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.718+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9830148e40 con 0x7f98300c34a0 2024-03-22T13:05:42.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.742+0000 7f9826a00640 1 -- 172.21.15.6:0/1289241584 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f981c058a90 con 0x7f98300c34a0 2024-03-22T13:05:42.742 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:42.743 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:42.743 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f981006bc20 msgr2=0x7f981006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f981006bc20 0x7f981006e0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9820001eb0 tx=0x7f9820009040 comp rx=0 tx=0).stop 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98300c34a0 msgr2=0x7f983014e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98300c34a0 0x7f983014e140 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f981c00f500 tx=0x7f981c00f9c0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 shutdown_connections 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98300c4390 0x7f98300c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f981006bc20 0x7f981006e0e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98300c34a0 0x7f983014e140 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 --2- 172.21.15.6:0/1289241584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98140052e0 0x7f98301f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 >> 172.21.15.6:0/1289241584 conn(0x7f98300b1480 msgr2=0x7f98300b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:42.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 shutdown_connections 2024-03-22T13:05:42.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:42.745+0000 7f982fe00640 1 -- 172.21.15.6:0/1289241584 wait complete. 2024-03-22T13:05:43.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:43.368 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:44.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:44.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.564+0000 7f20c8e00640 1 Processor -- start 2024-03-22T13:05:44.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.564+0000 7f20c8e00640 1 -- start start 2024-03-22T13:05:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.565+0000 7f20c8e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20c4152400 0x7f20c415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.565+0000 7f20c8e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c41590b0 con 0x7f20c406f8c0 2024-03-22T13:05:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.565+0000 7f20c8e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c40721f0 con 0x7f20c415bc80 2024-03-22T13:05:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.565+0000 7f20c8e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40724d0 con 0x7f20c4152400 2024-03-22T13:05:44.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.609+0000 7f20c1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20c4152400 0x7f20c415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:44.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.609+0000 7f20c1e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20c4152400 0x7f20c415b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47962/0 (socket says 172.21.15.6:47962) 2024-03-22T13:05:44.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.609+0000 7f20c1e00640 1 -- 172.21.15.6:0/3079284772 learned_addr learned my addr 172.21.15.6:0/3079284772 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:44.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.613+0000 7f20c1e00640 1 -- 172.21.15.6:0/3079284772 >> v1:172.21.15.92:6789/0 conn(0x7f20c415bc80 legacy=0x7f20c40716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:44.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.613+0000 7f20c1e00640 1 -- 172.21.15.6:0/3079284772 >> v1:172.21.15.6:6789/0 conn(0x7f20c406f8c0 legacy=0x7f20c40c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:44.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.613+0000 7f20c1e00640 1 -- 172.21.15.6:0/3079284772 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c41582f0 con 0x7f20c4152400 2024-03-22T13:05:44.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.618+0000 7f20c1e00640 1 --2- 172.21.15.6:0/3079284772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c4152400 0x7f20c415b740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009870 tx=0x7f20b402ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c000b821de986c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:44.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.622+0000 7f20c0a00640 1 -- 172.21.15.6:0/3079284772 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b402f840 con 0x7f20c4152400 2024-03-22T13:05:44.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.622+0000 7f20c0a00640 1 -- 172.21.15.6:0/3079284772 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b402f9e0 con 0x7f20c4152400 2024-03-22T13:05:44.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.622+0000 7f20c0a00640 1 -- 172.21.15.6:0/3079284772 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b402fdb0 con 0x7f20c4152400 2024-03-22T13:05:44.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.623+0000 7f20c8e00640 1 -- 172.21.15.6:0/3079284772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c4152400 msgr2=0x7f20c415b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:44.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.623+0000 7f20c8e00640 1 --2- 172.21.15.6:0/3079284772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c4152400 0x7f20c415b740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009870 tx=0x7f20b402ba20 comp rx=0 tx=0).stop 2024-03-22T13:05:44.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.623+0000 7f20c8e00640 1 -- 172.21.15.6:0/3079284772 shutdown_connections 2024-03-22T13:05:44.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.623+0000 7f20c8e00640 1 --2- 172.21.15.6:0/3079284772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c4152400 0x7f20c415b740 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:44.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.623+0000 7f20c8e00640 1 -- 172.21.15.6:0/3079284772 >> 172.21.15.6:0/3079284772 conn(0x7f20c40cb930 msgr2=0x7f20c40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:44.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.623+0000 7f20c8e00640 1 -- 172.21.15.6:0/3079284772 shutdown_connections 2024-03-22T13:05:44.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.624+0000 7f20c8e00640 1 -- 172.21.15.6:0/3079284772 wait complete. 2024-03-22T13:05:44.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.626+0000 7f20c8e00640 1 Processor -- start 2024-03-22T13:05:44.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.626+0000 7f20c8e00640 1 -- start start 2024-03-22T13:05:44.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.628+0000 7f20c8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c406f8c0 0x7f20c413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:44.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.628+0000 7f20c8e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c4152400 0x7f20c414d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:44.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.628+0000 7f20c8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20c415bc80 0x7f20c414d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:44.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.628+0000 7f20c8e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c4158b90 con 0x7f20c415bc80 2024-03-22T13:05:44.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.628+0000 7f20c8e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c41588b0 con 0x7f20c406f8c0 2024-03-22T13:05:44.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.628+0000 7f20c8e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c41527e0 con 0x7f20c4152400 2024-03-22T13:05:44.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.629+0000 7f20c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20c415bc80 0x7f20c414d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:44.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.629+0000 7f20c1400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c4152400 0x7f20c414d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:44.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.629+0000 7f20c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20c415bc80 0x7f20c414d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57212/0 (socket says 172.21.15.6:57212) 2024-03-22T13:05:44.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.629+0000 7f20c2800640 1 -- 172.21.15.6:0/1645206103 learned_addr learned my addr 172.21.15.6:0/1645206103 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:44.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.632+0000 7f20c1e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c406f8c0 0x7f20c413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.633+0000 7f20c1400640 1 -- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c406f8c0 msgr2=0x7f20c413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.633+0000 7f20c1400640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c406f8c0 0x7f20c413ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.633+0000 7f20c1400640 1 -- 172.21.15.6:0/1645206103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20c415bc80 msgr2=0x7f20c414d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.633+0000 7f20c1400640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20c415bc80 0x7f20c414d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.633+0000 7f20c1400640 1 -- 172.21.15.6:0/1645206103 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4156f80 con 0x7f20c4152400 2024-03-22T13:05:44.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.633+0000 7f20c1e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c406f8c0 0x7f20c413ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T13:05:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.636+0000 7f20c1400640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c4152400 0x7f20c414d4b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f20b000c390 tx=0x7f20b000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.641+0000 7f20ba000640 1 -- 172.21.15.6:0/1645206103 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b00052d0 con 0x7f20c4152400 2024-03-22T13:05:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.641+0000 7f20ba000640 1 -- 172.21.15.6:0/1645206103 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b0005490 con 0x7f20c4152400 2024-03-22T13:05:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.641+0000 7f20ba000640 1 -- 172.21.15.6:0/1645206103 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b0007260 con 0x7f20c4152400 2024-03-22T13:05:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.641+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c4157260 con 0x7f20c4152400 2024-03-22T13:05:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.641+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c414ccb0 con 0x7f20c4152400 2024-03-22T13:05:44.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.643+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c406fed0 con 0x7f20c4152400 2024-03-22T13:05:44.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.913+0000 7f20ba000640 1 -- 172.21.15.6:0/1645206103 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f20b0007400 con 0x7f20c4152400 2024-03-22T13:05:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.915+0000 7f20ba000640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20a406bc20 0x7f20a406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.916+0000 7f20c1e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20a406bc20 0x7f20a406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:44.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.916+0000 7f20ba000640 1 -- 172.21.15.6:0/1645206103 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f20b0020cb0 con 0x7f20c4152400 2024-03-22T13:05:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.917+0000 7f20c1e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20a406bc20 0x7f20a406e0e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f20b4036000 tx=0x7f20b4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:44.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:44.934+0000 7f20ba000640 1 -- 172.21.15.6:0/1645206103 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f20c406fed0 con 0x7f20c4152400 2024-03-22T13:05:45.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.198+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20c4072d10 con 0x7f20c4152400 2024-03-22T13:05:45.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.221+0000 7f20ba000640 1 -- 172.21.15.6:0/1645206103 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f20c4072d10 con 0x7f20c4152400 2024-03-22T13:05:45.221 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:45.222 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:45.222 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20a406bc20 msgr2=0x7f20a406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20a406bc20 0x7f20a406e0e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f20b4036000 tx=0x7f20b4008030 comp rx=0 tx=0).stop 2024-03-22T13:05:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c4152400 msgr2=0x7f20c414d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c4152400 0x7f20c414d4b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f20b000c390 tx=0x7f20b000c850 comp rx=0 tx=0).stop 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 shutdown_connections 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20c415bc80 0x7f20c414d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c4152400 0x7f20c414d4b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f20a406bc20 0x7f20a406e0e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 --2- 172.21.15.6:0/1645206103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c406f8c0 0x7f20c413ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 >> 172.21.15.6:0/1645206103 conn(0x7f20c40cb930 msgr2=0x7f20c40716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 shutdown_connections 2024-03-22T13:05:45.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:45.224+0000 7f20c8e00640 1 -- 172.21.15.6:0/1645206103 wait complete. 2024-03-22T13:05:45.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:45.846 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:46.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.052+0000 7f83e0c00640 1 Processor -- start 2024-03-22T13:05:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.052+0000 7f83e0c00640 1 -- start start 2024-03-22T13:05:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.054+0000 7f83e0c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc150f50 0x7f83dc0c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.054+0000 7f83e0c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f83dc0c5c20 0x7f83dc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.054+0000 7f83e0c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83dc0c3bf0 con 0x7f83dc157410 2024-03-22T13:05:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.054+0000 7f83e0c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83dc0c3760 con 0x7f83dc0c5c20 2024-03-22T13:05:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.054+0000 7f83e0c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc071e70 con 0x7f83dc150f50 2024-03-22T13:05:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.055+0000 7f83d9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc150f50 0x7f83dc0c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.056+0000 7f83d9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc150f50 0x7f83dc0c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46878/0 (socket says 172.21.15.6:46878) 2024-03-22T13:05:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.056+0000 7f83d9e00640 1 -- 172.21.15.6:0/3405816404 learned_addr learned my addr 172.21.15.6:0/3405816404 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:47.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.056+0000 7f83d9400640 1 --2- 172.21.15.6:0/3405816404 >> v2:172.21.15.6:3300/0 conn(0x7f83dc0c5c20 0x7f83dc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.059+0000 7f83d9e00640 1 -- 172.21.15.6:0/3405816404 >> v2:172.21.15.6:3300/0 conn(0x7f83dc0c5c20 msgr2=0x7f83dc156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.059+0000 7f83d9e00640 1 --2- 172.21.15.6:0/3405816404 >> v2:172.21.15.6:3300/0 conn(0x7f83dc0c5c20 0x7f83dc156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.059+0000 7f83d9e00640 1 -- 172.21.15.6:0/3405816404 >> v1:172.21.15.6:6789/0 conn(0x7f83dc157410 legacy=0x7f83dc06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:47.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.059+0000 7f83d9e00640 1 -- 172.21.15.6:0/3405816404 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc072150 con 0x7f83dc150f50 2024-03-22T13:05:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.062+0000 7f83d9e00640 1 --2- 172.21.15.6:0/3405816404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc150f50 0x7f83dc0c79e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009870 tx=0x7f83cc02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=693d98f7f116d3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:47.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.065+0000 7f83d8a00640 1 -- 172.21.15.6:0/3405816404 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc02f830 con 0x7f83dc150f50 2024-03-22T13:05:47.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.067+0000 7f83d8a00640 1 -- 172.21.15.6:0/3405816404 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc02f9d0 con 0x7f83dc150f50 2024-03-22T13:05:47.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.067+0000 7f83d8a00640 1 -- 172.21.15.6:0/3405816404 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc02fda0 con 0x7f83dc150f50 2024-03-22T13:05:47.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.067+0000 7f83e0c00640 1 -- 172.21.15.6:0/3405816404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc150f50 msgr2=0x7f83dc0c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:47.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.067+0000 7f83e0c00640 1 --2- 172.21.15.6:0/3405816404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc150f50 0x7f83dc0c79e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009870 tx=0x7f83cc02ba10 comp rx=0 tx=0).stop 2024-03-22T13:05:47.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.068+0000 7f83e0c00640 1 -- 172.21.15.6:0/3405816404 shutdown_connections 2024-03-22T13:05:47.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.068+0000 7f83e0c00640 1 --2- 172.21.15.6:0/3405816404 >> v2:172.21.15.6:3300/0 conn(0x7f83dc0c5c20 0x7f83dc156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.068+0000 7f83e0c00640 1 --2- 172.21.15.6:0/3405816404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc150f50 0x7f83dc0c79e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.068+0000 7f83e0c00640 1 -- 172.21.15.6:0/3405816404 >> 172.21.15.6:0/3405816404 conn(0x7f83dc0cb930 msgr2=0x7f83dc0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:47.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.068+0000 7f83e0c00640 1 -- 172.21.15.6:0/3405816404 shutdown_connections 2024-03-22T13:05:47.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.068+0000 7f83e0c00640 1 -- 172.21.15.6:0/3405816404 wait complete. 2024-03-22T13:05:47.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.069+0000 7f83e0c00640 1 Processor -- start 2024-03-22T13:05:47.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.070+0000 7f83e0c00640 1 -- start start 2024-03-22T13:05:47.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.070+0000 7f83e0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83dc0c5c20 0x7f83dc0ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:47.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.070+0000 7f83e0c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83dc150f50 0x7f83dc14bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:47.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.070+0000 7f83e0c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc157410 0x7f83dc14c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.070+0000 7f83e0c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc0c33f0 con 0x7f83dc0c5c20 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.070+0000 7f83e0c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc0c3110 con 0x7f83dc150f50 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.070+0000 7f83e0c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc151330 con 0x7f83dc157410 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.071+0000 7f83d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83dc0c5c20 0x7f83dc0ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.071+0000 7f83da800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc157410 0x7f83dc14c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.071+0000 7f83d9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83dc150f50 0x7f83dc14bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.071+0000 7f83d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83dc0c5c20 0x7f83dc0ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57230/0 (socket says 172.21.15.6:57230) 2024-03-22T13:05:47.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.072+0000 7f83d9e00640 1 -- 172.21.15.6:0/2130974227 learned_addr learned my addr 172.21.15.6:0/2130974227 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.075+0000 7f83d9e00640 1 -- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc157410 msgr2=0x7f83dc14c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.075+0000 7f83d9e00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc157410 0x7f83dc14c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.075+0000 7f83d9e00640 1 -- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83dc150f50 msgr2=0x7f83dc14bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.075+0000 7f83d9e00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83dc150f50 0x7f83dc14bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.075+0000 7f83d9e00640 1 -- 172.21.15.6:0/2130974227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc14ba70 con 0x7f83dc0c5c20 2024-03-22T13:05:47.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.078+0000 7f83d9e00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83dc0c5c20 0x7f83dc0ca9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.082+0000 7f83d2000640 1 -- 172.21.15.6:0/2130974227 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc037c40 con 0x7f83dc0c5c20 2024-03-22T13:05:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.082+0000 7f83d2000640 1 -- 172.21.15.6:0/2130974227 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc046070 con 0x7f83dc0c5c20 2024-03-22T13:05:47.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.082+0000 7f83d2000640 1 -- 172.21.15.6:0/2130974227 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc041400 con 0x7f83dc0c5c20 2024-03-22T13:05:47.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.082+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83dc156850 con 0x7f83dc0c5c20 2024-03-22T13:05:47.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.082+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83dc156e10 con 0x7f83dc0c5c20 2024-03-22T13:05:47.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.083+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83dc0756c0 con 0x7f83dc0c5c20 2024-03-22T13:05:47.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.416+0000 7f83d2000640 1 -- 172.21.15.6:0/2130974227 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f83cc040070 con 0x7f83dc0c5c20 2024-03-22T13:05:47.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.417+0000 7f83d2000640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f83bc06bc20 0x7f83bc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:47.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.418+0000 7f83d9400640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f83bc06bc20 0x7f83bc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:47.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.419+0000 7f83d9400640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f83bc06bc20 0x7f83bc06e0e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0c6ae0 tx=0x7f83c8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:47.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.454+0000 7f83d2000640 1 -- 172.21.15.6:0/2130974227 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f83cc079f40 con 0x7f83dc0c5c20 2024-03-22T13:05:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.455+0000 7f83d2000640 1 -- 172.21.15.6:0/2130974227 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f83cc033030 con 0x7f83dc0c5c20 2024-03-22T13:05:47.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.725+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83dc0c2690 con 0x7f83dc0c5c20 2024-03-22T13:05:47.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.747+0000 7f83d2000640 1 -- 172.21.15.6:0/2130974227 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f83cc07bea0 con 0x7f83dc0c5c20 2024-03-22T13:05:47.747 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:47.747 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:47.747 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f83bc06bc20 msgr2=0x7f83bc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f83bc06bc20 0x7f83bc06e0e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0c6ae0 tx=0x7f83c8009190 comp rx=0 tx=0).stop 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83dc0c5c20 msgr2=0x7f83dc0ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83dc0c5c20 0x7f83dc0ca9f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc004730 comp rx=0 tx=0).stop 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 shutdown_connections 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83dc157410 0x7f83dc14c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f83bc06bc20 0x7f83bc06e0e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.749+0000 7f83e0c00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83dc150f50 0x7f83dc14bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.750+0000 7f83e0c00640 1 --2- 172.21.15.6:0/2130974227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f83dc0c5c20 0x7f83dc0ca9f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.750+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 >> 172.21.15.6:0/2130974227 conn(0x7f83dc0cb930 msgr2=0x7f83dc0744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.750+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 shutdown_connections 2024-03-22T13:05:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:47.750+0000 7f83e0c00640 1 -- 172.21.15.6:0/2130974227 wait complete. 2024-03-22T13:05:48.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:48.370 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:49.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.573+0000 7f865e600640 1 Processor -- start 2024-03-22T13:05:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.574+0000 7f865e600640 1 -- start start 2024-03-22T13:05:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.575+0000 7f865e600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8658152400 0x7f865815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.575+0000 7f865e600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865815bc80 0x7f86580716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.575+0000 7f865e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f865806f8c0 0x7f8658071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.575+0000 7f865e600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580724d0 con 0x7f865806f8c0 2024-03-22T13:05:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.575+0000 7f865e600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86581590b0 con 0x7f8658152400 2024-03-22T13:05:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.575+0000 7f865e600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86580721f0 con 0x7f865815bc80 2024-03-22T13:05:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.576+0000 7f8656a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865815bc80 0x7f86580716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.576+0000 7f8657400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8658152400 0x7f865815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.576+0000 7f8657e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f865806f8c0 0x7f8658071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.576+0000 7f8656a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865815bc80 0x7f86580716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46900/0 (socket says 172.21.15.6:46900) 2024-03-22T13:05:49.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.576+0000 7f8656a00640 1 -- 172.21.15.6:0/3546151129 learned_addr learned my addr 172.21.15.6:0/3546151129 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:49.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.579+0000 7f8656a00640 1 -- 172.21.15.6:0/3546151129 >> v2:172.21.15.92:3300/0 conn(0x7f8658152400 msgr2=0x7f865815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:49.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.579+0000 7f8656a00640 1 --2- 172.21.15.6:0/3546151129 >> v2:172.21.15.92:3300/0 conn(0x7f8658152400 0x7f865815b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.579+0000 7f8656a00640 1 -- 172.21.15.6:0/3546151129 >> v2:172.21.15.6:3300/0 conn(0x7f865806f8c0 msgr2=0x7f8658071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:49.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.579+0000 7f8656a00640 1 --2- 172.21.15.6:0/3546151129 >> v2:172.21.15.6:3300/0 conn(0x7f865806f8c0 0x7f8658071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.579+0000 7f8656a00640 1 -- 172.21.15.6:0/3546151129 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86581582f0 con 0x7f865815bc80 2024-03-22T13:05:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.581+0000 7f8656a00640 1 --2- 172.21.15.6:0/3546151129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865815bc80 0x7f86580716d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8658071390 tx=0x7f864402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7886b5df4603b4fb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:49.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.585+0000 7f8656000640 1 -- 172.21.15.6:0/3546151129 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644034070 con 0x7f865815bc80 2024-03-22T13:05:49.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.585+0000 7f8656000640 1 -- 172.21.15.6:0/3546151129 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644030860 con 0x7f865815bc80 2024-03-22T13:05:49.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.585+0000 7f8656000640 1 -- 172.21.15.6:0/3546151129 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644030c30 con 0x7f865815bc80 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 -- 172.21.15.6:0/3546151129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865815bc80 msgr2=0x7f86580716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 --2- 172.21.15.6:0/3546151129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865815bc80 0x7f86580716d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8658071390 tx=0x7f864402b9a0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 -- 172.21.15.6:0/3546151129 shutdown_connections 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 --2- 172.21.15.6:0/3546151129 >> v2:172.21.15.6:3300/0 conn(0x7f865806f8c0 0x7f8658071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 --2- 172.21.15.6:0/3546151129 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865815bc80 0x7f86580716d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 --2- 172.21.15.6:0/3546151129 >> v2:172.21.15.92:3300/0 conn(0x7f8658152400 0x7f865815b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 -- 172.21.15.6:0/3546151129 >> 172.21.15.6:0/3546151129 conn(0x7f86580cb930 msgr2=0x7f86580cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 -- 172.21.15.6:0/3546151129 shutdown_connections 2024-03-22T13:05:49.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.586+0000 7f865e600640 1 -- 172.21.15.6:0/3546151129 wait complete. 2024-03-22T13:05:49.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.588+0000 7f865e600640 1 Processor -- start 2024-03-22T13:05:49.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.588+0000 7f865e600640 1 -- start start 2024-03-22T13:05:49.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.589+0000 7f865e600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865806f8c0 0x7f86581f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:49.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.589+0000 7f865e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8658152400 0x7f865814dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:49.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.589+0000 7f865e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865815bc80 0x7f86581f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:49.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.589+0000 7f865e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658158b90 con 0x7f8658152400 2024-03-22T13:05:49.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.589+0000 7f865e600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86581588b0 con 0x7f865815bc80 2024-03-22T13:05:49.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.589+0000 7f865e600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86581527e0 con 0x7f865806f8c0 2024-03-22T13:05:49.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.590+0000 7f8657400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865806f8c0 0x7f86581f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:49.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.590+0000 7f8657e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865815bc80 0x7f86581f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:49.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.590+0000 7f8656a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8658152400 0x7f865814dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:49.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.590+0000 7f8657400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865806f8c0 0x7f86581f0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46902/0 (socket says 172.21.15.6:46902) 2024-03-22T13:05:49.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.590+0000 7f8657e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865815bc80 0x7f86581f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48002/0 (socket says 172.21.15.6:48002) 2024-03-22T13:05:49.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.590+0000 7f8657400640 1 -- 172.21.15.6:0/1770111964 learned_addr learned my addr 172.21.15.6:0/1770111964 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:49.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.593+0000 7f8657e00640 1 -- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865806f8c0 msgr2=0x7f86581f0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:49.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.593+0000 7f8657e00640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865806f8c0 0x7f86581f0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.593+0000 7f8657e00640 1 -- 172.21.15.6:0/1770111964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8658152400 msgr2=0x7f865814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:49.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.593+0000 7f8657e00640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8658152400 0x7f865814dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:49.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.593+0000 7f8657e00640 1 -- 172.21.15.6:0/1770111964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86581f0a40 con 0x7f865815bc80 2024-03-22T13:05:49.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.596+0000 7f8657e00640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865815bc80 0x7f86581f41f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f865000eed0 tx=0x7f865000b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:49.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.600+0000 7f864f400640 1 -- 172.21.15.6:0/1770111964 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650018020 con 0x7f865815bc80 2024-03-22T13:05:49.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.600+0000 7f864f400640 1 -- 172.21.15.6:0/1770111964 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8650014400 con 0x7f865815bc80 2024-03-22T13:05:49.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.600+0000 7f864f400640 1 -- 172.21.15.6:0/1770111964 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650009e60 con 0x7f865815bc80 2024-03-22T13:05:49.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.600+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86581f3a10 con 0x7f865815bc80 2024-03-22T13:05:49.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.601+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86581f0fe0 con 0x7f865815bc80 2024-03-22T13:05:49.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.602+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86580756c0 con 0x7f865815bc80 2024-03-22T13:05:49.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.873+0000 7f864f400640 1 -- 172.21.15.6:0/1770111964 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8650005d10 con 0x7f865815bc80 2024-03-22T13:05:49.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.875+0000 7f864f400640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f863806bc20 0x7f863806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:49.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.875+0000 7f8657400640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f863806bc20 0x7f863806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.876+0000 7f864f400640 1 -- 172.21.15.6:0/1770111964 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8650057de0 con 0x7f865815bc80 2024-03-22T13:05:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.876+0000 7f8657400640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f863806bc20 0x7f863806e0e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8648009c00 tx=0x7f8648009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:49.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:49.897+0000 7f864f400640 1 -- 172.21.15.6:0/1770111964 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8650022030 con 0x7f865815bc80 2024-03-22T13:05:50.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.166+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86580c6400 con 0x7f865815bc80 2024-03-22T13:05:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.250+0000 7f864f400640 1 -- 172.21.15.6:0/1770111964 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f8650059d40 con 0x7f865815bc80 2024-03-22T13:05:50.251 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:50.251 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f863806bc20 msgr2=0x7f863806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:50.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f863806bc20 0x7f863806e0e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8648009c00 tx=0x7f8648009190 comp rx=0 tx=0).stop 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865815bc80 msgr2=0x7f86581f41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865815bc80 0x7f86581f41f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f865000eed0 tx=0x7f865000b480 comp rx=0 tx=0).stop 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 shutdown_connections 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f863806bc20 0x7f863806e0e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865815bc80 0x7f86581f41f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8658152400 0x7f865814dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 --2- 172.21.15.6:0/1770111964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f865806f8c0 0x7f86581f0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.254+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 >> 172.21.15.6:0/1770111964 conn(0x7f86580cb930 msgr2=0x7f8658151440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.255+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 shutdown_connections 2024-03-22T13:05:50.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:50.255+0000 7f865e600640 1 -- 172.21.15.6:0/1770111964 wait complete. 2024-03-22T13:05:50.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:50.878 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:51.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:52.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.076+0000 7fe929800640 1 Processor -- start 2024-03-22T13:05:52.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.076+0000 7fe929800640 1 -- start start 2024-03-22T13:05:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.077+0000 7fe929800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cc3b0 0x7fe9240cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.077+0000 7fe929800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe9240cf770 0x7fe9240cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.077+0000 7fe929800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe9240cccd0 0x7fe92415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.077+0000 7fe929800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9240ce230 con 0x7fe9240cf770 2024-03-22T13:05:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.077+0000 7fe929800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9240ce510 con 0x7fe9240cccd0 2024-03-22T13:05:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.077+0000 7fe929800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9240ce7f0 con 0x7fe9240cc3b0 2024-03-22T13:05:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.078+0000 7fe922a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cc3b0 0x7fe9240cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.078+0000 7fe922000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe9240cf770 0x7fe9240cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.078+0000 7fe923400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe9240cccd0 0x7fe92415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.078+0000 7fe922a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cc3b0 0x7fe9240cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46914/0 (socket says 172.21.15.6:46914) 2024-03-22T13:05:52.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.078+0000 7fe922a00640 1 -- 172.21.15.6:0/2803898326 learned_addr learned my addr 172.21.15.6:0/2803898326 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:52.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.081+0000 7fe922a00640 1 -- 172.21.15.6:0/2803898326 >> v2:172.21.15.92:3300/0 conn(0x7fe9240cccd0 msgr2=0x7fe92415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:52.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.081+0000 7fe922a00640 1 --2- 172.21.15.6:0/2803898326 >> v2:172.21.15.92:3300/0 conn(0x7fe9240cccd0 0x7fe92415ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.081+0000 7fe922a00640 1 -- 172.21.15.6:0/2803898326 >> v2:172.21.15.6:3300/0 conn(0x7fe9240cf770 msgr2=0x7fe9240cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:52.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.081+0000 7fe922a00640 1 --2- 172.21.15.6:0/2803898326 >> v2:172.21.15.6:3300/0 conn(0x7fe9240cf770 0x7fe9240cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.081+0000 7fe922a00640 1 -- 172.21.15.6:0/2803898326 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240cead0 con 0x7fe9240cc3b0 2024-03-22T13:05:52.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.084+0000 7fe922a00640 1 --2- 172.21.15.6:0/2803898326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cc3b0 0x7fe9240cf230 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe92408d7f0 tx=0x7fe91402b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ceb59262ad3666 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:52.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.089+0000 7fe921600640 1 -- 172.21.15.6:0/2803898326 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91402e070 con 0x7fe9240cc3b0 2024-03-22T13:05:52.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.089+0000 7fe921600640 1 -- 172.21.15.6:0/2803898326 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe914037070 con 0x7fe9240cc3b0 2024-03-22T13:05:52.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.089+0000 7fe921600640 1 -- 172.21.15.6:0/2803898326 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914031a00 con 0x7fe9240cc3b0 2024-03-22T13:05:52.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 -- 172.21.15.6:0/2803898326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cc3b0 msgr2=0x7fe9240cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:52.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 --2- 172.21.15.6:0/2803898326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cc3b0 0x7fe9240cf230 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe92408d7f0 tx=0x7fe91402b880 comp rx=0 tx=0).stop 2024-03-22T13:05:52.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 -- 172.21.15.6:0/2803898326 shutdown_connections 2024-03-22T13:05:52.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 --2- 172.21.15.6:0/2803898326 >> v2:172.21.15.92:3300/0 conn(0x7fe9240cccd0 0x7fe92415ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 --2- 172.21.15.6:0/2803898326 >> v2:172.21.15.6:3300/0 conn(0x7fe9240cf770 0x7fe9240cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 --2- 172.21.15.6:0/2803898326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cc3b0 0x7fe9240cf230 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 -- 172.21.15.6:0/2803898326 >> 172.21.15.6:0/2803898326 conn(0x7fe9240b16d0 msgr2=0x7fe9240b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:52.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 -- 172.21.15.6:0/2803898326 shutdown_connections 2024-03-22T13:05:52.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.090+0000 7fe929800640 1 -- 172.21.15.6:0/2803898326 wait complete. 2024-03-22T13:05:52.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.092+0000 7fe929800640 1 Processor -- start 2024-03-22T13:05:52.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.092+0000 7fe929800640 1 -- start start 2024-03-22T13:05:52.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.093+0000 7fe929800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9240cc3b0 0x7fe9241577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:52.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.093+0000 7fe929800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9240cccd0 0x7fe9241f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:52.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.093+0000 7fe929800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cf770 0x7fe9241f0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:52.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.093+0000 7fe929800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92415a5d0 con 0x7fe9240cccd0 2024-03-22T13:05:52.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.093+0000 7fe929800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92406dc10 con 0x7fe9240cc3b0 2024-03-22T13:05:52.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.093+0000 7fe929800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe924154bb0 con 0x7fe9240cf770 2024-03-22T13:05:52.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.094+0000 7fe923400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cf770 0x7fe9241f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:52.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.094+0000 7fe923400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cf770 0x7fe9241f0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46928/0 (socket says 172.21.15.6:46928) 2024-03-22T13:05:52.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.094+0000 7fe923400640 1 -- 172.21.15.6:0/2140089649 learned_addr learned my addr 172.21.15.6:0/2140089649 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:52.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.095+0000 7fe922000640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9240cccd0 0x7fe9241f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:52.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.095+0000 7fe922a00640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9240cc3b0 0x7fe9241577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:52.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.099+0000 7fe922000640 1 -- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cf770 msgr2=0x7fe9241f0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:52.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.099+0000 7fe922000640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cf770 0x7fe9241f0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.099+0000 7fe922000640 1 -- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9240cc3b0 msgr2=0x7fe9241577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:52.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.099+0000 7fe922000640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9240cc3b0 0x7fe9241577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.099+0000 7fe922000640 1 -- 172.21.15.6:0/2140089649 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9240c7110 con 0x7fe9240cccd0 2024-03-22T13:05:52.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.102+0000 7fe922000640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9240cccd0 0x7fe9241f3d30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e870 tx=0x7fe91000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.108+0000 7fe91aa00640 1 -- 172.21.15.6:0/2140089649 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe910017070 con 0x7fe9240cccd0 2024-03-22T13:05:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.108+0000 7fe91aa00640 1 -- 172.21.15.6:0/2140089649 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91000ee60 con 0x7fe9240cccd0 2024-03-22T13:05:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.108+0000 7fe91aa00640 1 -- 172.21.15.6:0/2140089649 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91000cb80 con 0x7fe9240cccd0 2024-03-22T13:05:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.108+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9240c73f0 con 0x7fe9240cccd0 2024-03-22T13:05:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.108+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9240b01b0 con 0x7fe9240cccd0 2024-03-22T13:05:52.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.110+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9240756c0 con 0x7fe9240cccd0 2024-03-22T13:05:52.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.429+0000 7fe91aa00640 1 -- 172.21.15.6:0/2140089649 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe91001a070 con 0x7fe9240cccd0 2024-03-22T13:05:52.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.431+0000 7fe91aa00640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe90406bc20 0x7fe90406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:52.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.431+0000 7fe922a00640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe90406bc20 0x7fe90406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:52.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.432+0000 7fe922a00640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe90406bc20 0x7fe90406e0e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe914036000 tx=0x7fe91403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.467+0000 7fe91aa00640 1 -- 172.21.15.6:0/2140089649 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe910057440 con 0x7fe9240cccd0 2024-03-22T13:05:52.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.468+0000 7fe91aa00640 1 -- 172.21.15.6:0/2140089649 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9100593a0 con 0x7fe9240cccd0 2024-03-22T13:05:52.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.737+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe924153ef0 con 0x7fe9240cccd0 2024-03-22T13:05:52.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.755+0000 7fe91aa00640 1 -- 172.21.15.6:0/2140089649 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fe91005d440 con 0x7fe9240cccd0 2024-03-22T13:05:52.756 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:52.756 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:52.756 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.758+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe90406bc20 msgr2=0x7fe90406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.758+0000 7fe929800640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe90406bc20 0x7fe90406e0e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe914036000 tx=0x7fe91403d040 comp rx=0 tx=0).stop 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.758+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9240cccd0 msgr2=0x7fe9241f3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.758+0000 7fe929800640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9240cccd0 0x7fe9241f3d30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e870 tx=0x7fe91000ed30 comp rx=0 tx=0).stop 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.758+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 shutdown_connections 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.758+0000 7fe929800640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe9240cf770 0x7fe9241f0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.758+0000 7fe929800640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9240cccd0 0x7fe9241f3d30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.759+0000 7fe929800640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe90406bc20 0x7fe90406e0e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.759+0000 7fe929800640 1 --2- 172.21.15.6:0/2140089649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9240cc3b0 0x7fe9241577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.759+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 >> 172.21.15.6:0/2140089649 conn(0x7fe9240b16d0 msgr2=0x7fe9240cab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.759+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 shutdown_connections 2024-03-22T13:05:52.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:52.759+0000 7fe929800640 1 -- 172.21.15.6:0/2140089649 wait complete. 2024-03-22T13:05:53.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:53.380 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:54.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:54.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.596+0000 7f28f8a00640 1 Processor -- start 2024-03-22T13:05:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.596+0000 7f28f8a00640 1 -- start start 2024-03-22T13:05:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.597+0000 7f28f8a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f28f40cc3b0 0x7f28f40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.597+0000 7f28f8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40ccea0 0x7f28f415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.597+0000 7f28f8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f40ce230 con 0x7f28f40cf770 2024-03-22T13:05:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.597+0000 7f28f8a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f40ce510 con 0x7f28f40cc3b0 2024-03-22T13:05:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.597+0000 7f28f8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40ce7f0 con 0x7f28f40ccea0 2024-03-22T13:05:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.598+0000 7f28f1400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f28f40cf770 0x7f28f40cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50380/0 (socket says 172.21.15.6:50380) 2024-03-22T13:05:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.598+0000 7f28f1400640 1 -- 172.21.15.6:0/3469148687 learned_addr learned my addr 172.21.15.6:0/3469148687 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.601+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218853253 0 0) 0x7f28f40ce230 con 0x7f28f40cf770 2024-03-22T13:05:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.602+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d8003520 con 0x7f28f40cf770 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33789911 0 0) 0x7f28d8003520 con 0x7f28f40cf770 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40ccea0 msgr2=0x7f28f415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40ccea0 0x7f28f415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 >> v2:172.21.15.92:3300/0 conn(0x7f28f40cc3b0 msgr2=0x7f28f40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 --2- 172.21.15.6:0/3469148687 >> v2:172.21.15.92:3300/0 conn(0x7f28f40cc3b0 0x7f28f40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40cead0 con 0x7f28f40cf770 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f28dc001670 con 0x7f28f40cf770 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 >> v1:172.21.15.6:6789/0 conn(0x7f28f40cf770 legacy=0x7f28f40cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28d8004800 0x7f28d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40ccea0 0x7f28d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40ce510 con 0x7f28d80039b0 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40ce7f0 con 0x7f28d8004800 2024-03-22T13:05:54.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.606+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d800bf00 con 0x7f28f40ccea0 2024-03-22T13:05:54.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.611+0000 7f28f1e00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:54.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.615+0000 7f28f1e00640 1 -- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40ccea0 msgr2=0x7f28d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:54.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.615+0000 7f28f1e00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40ccea0 0x7f28d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:54.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.615+0000 7f28f1e00640 1 -- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28d8004800 msgr2=0x7f28d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:54.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.615+0000 7f28f1e00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28d8004800 0x7f28d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:54.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.615+0000 7f28f1e00640 1 -- 172.21.15.6:0/3469148687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d8003520 con 0x7f28d80039b0 2024-03-22T13:05:54.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.618+0000 7f28f1e00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28d8003dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009860 tx=0x7f28d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d287389410cf25 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:54.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.623+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4037070 con 0x7f28d80039b0 2024-03-22T13:05:54.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.623+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e4032620 con 0x7f28d80039b0 2024-03-22T13:05:54.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.624+0000 7f28f0a00640 1 -- 172.21.15.6:0/3469148687 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4009460 con 0x7f28d80039b0 2024-03-22T13:05:54.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.624+0000 7f28f8a00640 1 -- 172.21.15.6:0/3469148687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 msgr2=0x7f28d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:54.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.624+0000 7f28f8a00640 1 --2- 172.21.15.6:0/3469148687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28d8003dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009860 tx=0x7f28d8005830 comp rx=0 tx=0).stop 2024-03-22T13:05:54.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.624+0000 7f28f2800640 1 -- 172.21.15.6:0/3469148687 reap_dead start 2024-03-22T13:05:54.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.625+0000 7f28f8a00640 1 -- 172.21.15.6:0/3469148687 shutdown_connections 2024-03-22T13:05:54.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.625+0000 7f28f8a00640 1 -- 172.21.15.6:0/3469148687 >> 172.21.15.6:0/3469148687 conn(0x7f28f40b16d0 msgr2=0x7f28f40b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:54.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.625+0000 7f28f8a00640 1 -- 172.21.15.6:0/3469148687 shutdown_connections 2024-03-22T13:05:54.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.625+0000 7f28f8a00640 1 -- 172.21.15.6:0/3469148687 wait complete. 2024-03-22T13:05:54.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.627+0000 7f28f8a00640 1 Processor -- start 2024-03-22T13:05:54.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.627+0000 7f28f8a00640 1 -- start start 2024-03-22T13:05:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.628+0000 7f28f8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28f40c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.628+0000 7f28f8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40c8920 0x7f28f4148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.628+0000 7f28f8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28f40b2a80 0x7f28f41493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.628+0000 7f28f8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f415a5d0 con 0x7f28d80039b0 2024-03-22T13:05:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.628+0000 7f28f8a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f406dc10 con 0x7f28f40b2a80 2024-03-22T13:05:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.628+0000 7f28f8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4154bb0 con 0x7f28f40c8920 2024-03-22T13:05:54.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.629+0000 7f28f1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28f40c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:54.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.630+0000 7f28f1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28f40c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60124/0 (socket says 172.21.15.6:60124) 2024-03-22T13:05:54.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.630+0000 7f28f1e00640 1 -- 172.21.15.6:0/3480573502 learned_addr learned my addr 172.21.15.6:0/3480573502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.635+0000 7f28f1e00640 1 -- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40c8920 msgr2=0x7f28f4148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.635+0000 7f28f1e00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40c8920 0x7f28f4148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.635+0000 7f28f1e00640 1 -- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28f40b2a80 msgr2=0x7f28f41493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.635+0000 7f28f1e00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28f40b2a80 0x7f28f41493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:54.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.635+0000 7f28f1e00640 1 -- 172.21.15.6:0/3480573502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40b5360 con 0x7f28d80039b0 2024-03-22T13:05:54.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.639+0000 7f28f1e00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28f40c9570 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f28e402bcd0 tx=0x7f28e40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.644+0000 7f28ea000640 1 -- 172.21.15.6:0/3480573502 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4037030 con 0x7f28d80039b0 2024-03-22T13:05:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.645+0000 7f28ea000640 1 -- 172.21.15.6:0/3480573502 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e40044c0 con 0x7f28d80039b0 2024-03-22T13:05:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.645+0000 7f28ea000640 1 -- 172.21.15.6:0/3480573502 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e40037f0 con 0x7f28d80039b0 2024-03-22T13:05:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.645+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40b55d0 con 0x7f28d80039b0 2024-03-22T13:05:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.645+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f41f3860 con 0x7f28d80039b0 2024-03-22T13:05:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.646+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f40756c0 con 0x7f28d80039b0 2024-03-22T13:05:54.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.684+0000 7f28ea000640 1 -- 172.21.15.6:0/3480573502 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f28e4032d10 con 0x7f28d80039b0 2024-03-22T13:05:54.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.686+0000 7f28ea000640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28d006bc20 0x7f28d006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:54.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.686+0000 7f28f1400640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28d006bc20 0x7f28d006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:54.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.687+0000 7f28f1400640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28d006bc20 0x7f28d006e0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f28f40ce100 tx=0x7f28dc001530 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:54.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.692+0000 7f28ea000640 1 -- 172.21.15.6:0/3480573502 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f28e40794b0 con 0x7f28d80039b0 2024-03-22T13:05:54.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:54.979+0000 7f28ea000640 1 -- 172.21.15.6:0/3480573502 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f28e407bb10 con 0x7f28d80039b0 2024-03-22T13:05:55.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.225+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28f40cbba0 con 0x7f28d80039b0 2024-03-22T13:05:55.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.241+0000 7f28ea000640 1 -- 172.21.15.6:0/3480573502 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f28e407b410 con 0x7f28d80039b0 2024-03-22T13:05:55.241 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:55.242 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.244+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28d006bc20 msgr2=0x7f28d006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.244+0000 7f28f8a00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28d006bc20 0x7f28d006e0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f28f40ce100 tx=0x7f28dc001530 comp rx=0 tx=0).stop 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.244+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 msgr2=0x7f28f40c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.244+0000 7f28f8a00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28f40c9570 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f28e402bcd0 tx=0x7f28e40047d0 comp rx=0 tx=0).stop 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 shutdown_connections 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f28d006bc20 0x7f28d006e0e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28f40b2a80 0x7f28f41493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28f40c8920 0x7f28f4148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 --2- 172.21.15.6:0/3480573502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28d80039b0 0x7f28f40c9570 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 >> 172.21.15.6:0/3480573502 conn(0x7f28f40b16d0 msgr2=0x7f28f40b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 shutdown_connections 2024-03-22T13:05:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:55.245+0000 7f28f8a00640 1 -- 172.21.15.6:0/3480573502 wait complete. 2024-03-22T13:05:55.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:55.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:56.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.060+0000 7fd74f200640 1 Processor -- start 2024-03-22T13:05:57.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.060+0000 7fd74f200640 1 -- start start 2024-03-22T13:05:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.062+0000 7fd74f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd7500c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.062+0000 7fd74f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd750154580 0x7fd75006b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.062+0000 7fd74f200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd75006bbc0 0x7fd75015a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.062+0000 7fd74f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7500c90e0 con 0x7fd750154580 2024-03-22T13:05:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.062+0000 7fd74f200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75015b030 con 0x7fd75006bbc0 2024-03-22T13:05:57.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.062+0000 7fd74f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500c9350 con 0x7fd7501541a0 2024-03-22T13:05:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.063+0000 7fd74de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd7500c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.063+0000 7fd74d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd750154580 0x7fd75006b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.063+0000 7fd74e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd75006bbc0 0x7fd75015a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.063+0000 7fd74de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd7500c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44150/0 (socket says 172.21.15.6:44150) 2024-03-22T13:05:57.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.063+0000 7fd74de00640 1 -- 172.21.15.6:0/1067692097 learned_addr learned my addr 172.21.15.6:0/1067692097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:57.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.066+0000 7fd74d400640 1 -- 172.21.15.6:0/1067692097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 msgr2=0x7fd7500c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.066+0000 7fd74d400640 1 --2- 172.21.15.6:0/1067692097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd7500c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.066+0000 7fd74d400640 1 -- 172.21.15.6:0/1067692097 >> v2:172.21.15.92:3300/0 conn(0x7fd75006bbc0 msgr2=0x7fd75015a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.066+0000 7fd74d400640 1 --2- 172.21.15.6:0/1067692097 >> v2:172.21.15.92:3300/0 conn(0x7fd75006bbc0 0x7fd75015a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.066+0000 7fd74d400640 1 -- 172.21.15.6:0/1067692097 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500c9630 con 0x7fd750154580 2024-03-22T13:05:57.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.069+0000 7fd74d400640 1 --2- 172.21.15.6:0/1067692097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd750154580 0x7fd75006b680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd7500c4ea0 tx=0x7fd73c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1938409f1354a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.073+0000 7fd74ca00640 1 -- 172.21.15.6:0/1067692097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c035020 con 0x7fd750154580 2024-03-22T13:05:57.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.073+0000 7fd74ca00640 1 -- 172.21.15.6:0/1067692097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c031830 con 0x7fd750154580 2024-03-22T13:05:57.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.073+0000 7fd74ca00640 1 -- 172.21.15.6:0/1067692097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c031c00 con 0x7fd750154580 2024-03-22T13:05:57.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 -- 172.21.15.6:0/1067692097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd750154580 msgr2=0x7fd75006b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 --2- 172.21.15.6:0/1067692097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd750154580 0x7fd75006b680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd7500c4ea0 tx=0x7fd73c02b8d0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 -- 172.21.15.6:0/1067692097 shutdown_connections 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 --2- 172.21.15.6:0/1067692097 >> v2:172.21.15.92:3300/0 conn(0x7fd75006bbc0 0x7fd75015a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 --2- 172.21.15.6:0/1067692097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd750154580 0x7fd75006b680 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 --2- 172.21.15.6:0/1067692097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd7500c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 -- 172.21.15.6:0/1067692097 >> 172.21.15.6:0/1067692097 conn(0x7fd750070140 msgr2=0x7fd750070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 -- 172.21.15.6:0/1067692097 shutdown_connections 2024-03-22T13:05:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.074+0000 7fd74f200640 1 -- 172.21.15.6:0/1067692097 wait complete. 2024-03-22T13:05:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.076+0000 7fd74f200640 1 Processor -- start 2024-03-22T13:05:57.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.076+0000 7fd74f200640 1 -- start start 2024-03-22T13:05:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd75006bbc0 0x7fd75014c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd75014e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd750154580 0x7fd7501f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500ca110 con 0x7fd75006bbc0 2024-03-22T13:05:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500c9bf0 con 0x7fd750154580 2024-03-22T13:05:57.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd750073170 con 0x7fd7501541a0 2024-03-22T13:05:57.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd75014e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:57.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd75006bbc0 0x7fd75014c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:57.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.077+0000 7fd74e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd750154580 0x7fd7501f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:57.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.078+0000 7fd74d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd75014e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44160/0 (socket says 172.21.15.6:44160) 2024-03-22T13:05:57.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.078+0000 7fd74d400640 1 -- 172.21.15.6:0/2964174181 learned_addr learned my addr 172.21.15.6:0/2964174181 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.082+0000 7fd74d400640 1 -- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd750154580 msgr2=0x7fd7501f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.082+0000 7fd74d400640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd750154580 0x7fd7501f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.082+0000 7fd74d400640 1 -- 172.21.15.6:0/2964174181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd75006bbc0 msgr2=0x7fd75014c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.082+0000 7fd74d400640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd75006bbc0 0x7fd75014c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.082+0000 7fd74d400640 1 -- 172.21.15.6:0/2964174181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75015e070 con 0x7fd7501541a0 2024-03-22T13:05:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.085+0000 7fd74d400640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd75014e7e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009890 tx=0x7fd73c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.090+0000 7fd746000640 1 -- 172.21.15.6:0/2964174181 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c040070 con 0x7fd7501541a0 2024-03-22T13:05:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.090+0000 7fd746000640 1 -- 172.21.15.6:0/2964174181 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c03e070 con 0x7fd7501541a0 2024-03-22T13:05:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.090+0000 7fd746000640 1 -- 172.21.15.6:0/2964174181 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c035040 con 0x7fd7501541a0 2024-03-22T13:05:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.090+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7501f3cd0 con 0x7fd7501541a0 2024-03-22T13:05:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.090+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75015e5f0 con 0x7fd7501541a0 2024-03-22T13:05:57.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.092+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd750075650 con 0x7fd7501541a0 2024-03-22T13:05:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.123+0000 7fd746000640 1 -- 172.21.15.6:0/2964174181 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd73c038020 con 0x7fd7501541a0 2024-03-22T13:05:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.125+0000 7fd746000640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd73006bc20 0x7fd73006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.125+0000 7fd74de00640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd73006bc20 0x7fd73006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.126+0000 7fd74de00640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd73006bc20 0x7fd73006e0e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd740009bd0 tx=0x7fd740009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.126+0000 7fd746000640 1 -- 172.21.15.6:0/2964174181 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd73c0317b0 con 0x7fd7501541a0 2024-03-22T13:05:57.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.414+0000 7fd746000640 1 -- 172.21.15.6:0/2964174181 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd73c07a7c0 con 0x7fd7501541a0 2024-03-22T13:05:57.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.736+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd75015e7c0 con 0x7fd7501541a0 2024-03-22T13:05:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.758+0000 7fd746000640 1 -- 172.21.15.6:0/2964174181 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fd73c07e860 con 0x7fd7501541a0 2024-03-22T13:05:57.758 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:05:57.758 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:05:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:05:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.761+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd73006bc20 msgr2=0x7fd73006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.761+0000 7fd74f200640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd73006bc20 0x7fd73006e0e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd740009bd0 tx=0x7fd740009240 comp rx=0 tx=0).stop 2024-03-22T13:05:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.761+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 msgr2=0x7fd75014e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.761+0000 7fd74f200640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd75014e7e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009890 tx=0x7fd73c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 shutdown_connections 2024-03-22T13:05:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd73006bc20 0x7fd73006e0e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd750154580 0x7fd7501f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd7501541a0 0x7fd75014e7e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 --2- 172.21.15.6:0/2964174181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd75006bbc0 0x7fd75014c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 >> 172.21.15.6:0/2964174181 conn(0x7fd750070140 msgr2=0x7fd7500cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 shutdown_connections 2024-03-22T13:05:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:57.762+0000 7fd74f200640 1 -- 172.21.15.6:0/2964174181 wait complete. 2024-03-22T13:05:58.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:05:58.382 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:05:59.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:05:59.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.571+0000 7f1b7d000640 1 Processor -- start 2024-03-22T13:05:59.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.571+0000 7f1b7d000640 1 -- start start 2024-03-22T13:05:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.573+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780ce8a0 0x7f1b7806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.573+0000 7f1b7d000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b7815ae30 0x7f1b780cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.573+0000 7f1b7d000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7815b260 con 0x7f1b780cc380 2024-03-22T13:05:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.573+0000 7f1b7d000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b780cee00 con 0x7f1b7815ae30 2024-03-22T13:05:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.573+0000 7f1b7d000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b780cf0c0 con 0x7f1b780ce8a0 2024-03-22T13:05:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.574+0000 7f1b77e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b7815ae30 0x7f1b780cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.574+0000 7f1b77400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1b780cc380 0x7f1b780cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50384/0 (socket says 172.21.15.6:50384) 2024-03-22T13:05:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.574+0000 7f1b76a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780ce8a0 0x7f1b7806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.574+0000 7f1b77400640 1 -- 172.21.15.6:0/2797063347 learned_addr learned my addr 172.21.15.6:0/2797063347 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:05:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.577+0000 7f1b77e00640 1 -- 172.21.15.6:0/2797063347 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780ce8a0 msgr2=0x7f1b7806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:05:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.577+0000 7f1b77e00640 1 --2- 172.21.15.6:0/2797063347 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780ce8a0 0x7f1b7806c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.577+0000 7f1b77e00640 1 -- 172.21.15.6:0/2797063347 >> v1:172.21.15.6:6789/0 conn(0x7f1b780cc380 legacy=0x7f1b780cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:59.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.577+0000 7f1b77e00640 1 -- 172.21.15.6:0/2797063347 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b780cf3a0 con 0x7f1b7815ae30 2024-03-22T13:05:59.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.579+0000 7f1b77e00640 1 --2- 172.21.15.6:0/2797063347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 0x7f1b780cc760 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009d90 tx=0x7f1b7002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75336daf322a521 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:59.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.582+0000 7f1b76000640 1 -- 172.21.15.6:0/2797063347 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70039070 con 0x7f1b7815ae30 2024-03-22T13:05:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b76000640 1 -- 172.21.15.6:0/2797063347 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7002cc50 con 0x7f1b7815ae30 2024-03-22T13:05:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b76000640 1 -- 172.21.15.6:0/2797063347 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70034440 con 0x7f1b7815ae30 2024-03-22T13:05:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b7d000640 1 -- 172.21.15.6:0/2797063347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 msgr2=0x7f1b780cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2797063347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 0x7f1b780cc760 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009d90 tx=0x7f1b7002bea0 comp rx=0 tx=0).stop 2024-03-22T13:05:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b7d000640 1 -- 172.21.15.6:0/2797063347 shutdown_connections 2024-03-22T13:05:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2797063347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 0x7f1b780cc760 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2797063347 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780ce8a0 0x7f1b7806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.584+0000 7f1b7d000640 1 -- 172.21.15.6:0/2797063347 >> 172.21.15.6:0/2797063347 conn(0x7f1b780b1480 msgr2=0x7f1b780b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:05:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.585+0000 7f1b7d000640 1 -- 172.21.15.6:0/2797063347 shutdown_connections 2024-03-22T13:05:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.585+0000 7f1b7d000640 1 -- 172.21.15.6:0/2797063347 wait complete. 2024-03-22T13:05:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.587+0000 7f1b7d000640 1 Processor -- start 2024-03-22T13:05:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.587+0000 7f1b7d000640 1 -- start start 2024-03-22T13:05:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.588+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780cc380 0x7f1b780b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.588+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b780ce8a0 0x7f1b781f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.588+0000 7f1b7d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 0x7f1b781f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.588+0000 7f1b7d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7815a910 con 0x7f1b780ce8a0 2024-03-22T13:05:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.588+0000 7f1b7d000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7815a630 con 0x7f1b7815ae30 2024-03-22T13:05:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.588+0000 7f1b7d000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b781549d0 con 0x7f1b780cc380 2024-03-22T13:05:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.589+0000 7f1b77400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780cc380 0x7f1b780b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:59.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.589+0000 7f1b77e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 0x7f1b781f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:59.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.589+0000 7f1b76a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b780ce8a0 0x7f1b781f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:59.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.589+0000 7f1b77400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780cc380 0x7f1b780b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44174/0 (socket says 172.21.15.6:44174) 2024-03-22T13:05:59.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.589+0000 7f1b77400640 1 -- 172.21.15.6:0/2000120136 learned_addr learned my addr 172.21.15.6:0/2000120136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:05:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.594+0000 7f1b76a00640 1 -- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780cc380 msgr2=0x7f1b780b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.594+0000 7f1b76a00640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780cc380 0x7f1b780b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.594+0000 7f1b76a00640 1 -- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 msgr2=0x7f1b781f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:05:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.594+0000 7f1b76a00640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 0x7f1b781f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:05:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.594+0000 7f1b76a00640 1 -- 172.21.15.6:0/2000120136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7815e150 con 0x7f1b780ce8a0 2024-03-22T13:05:59.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.597+0000 7f1b76a00640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b780ce8a0 0x7f1b781f3740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1b6400e820 tx=0x7f1b6400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:59.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.602+0000 7f1b6f400640 1 -- 172.21.15.6:0/2000120136 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64017070 con 0x7f1b780ce8a0 2024-03-22T13:05:59.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.602+0000 7f1b6f400640 1 -- 172.21.15.6:0/2000120136 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6400c940 con 0x7f1b780ce8a0 2024-03-22T13:05:59.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.602+0000 7f1b6f400640 1 -- 172.21.15.6:0/2000120136 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400cd30 con 0x7f1b780ce8a0 2024-03-22T13:05:59.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.602+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7814fcd0 con 0x7f1b780ce8a0 2024-03-22T13:05:59.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.603+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7814ff70 con 0x7f1b780ce8a0 2024-03-22T13:05:59.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.604+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b78075650 con 0x7f1b780ce8a0 2024-03-22T13:05:59.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.630+0000 7f1b6f400640 1 -- 172.21.15.6:0/2000120136 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b6401a070 con 0x7f1b780ce8a0 2024-03-22T13:05:59.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.631+0000 7f1b6f400640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b5806bc20 0x7f1b5806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:05:59.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.632+0000 7f1b77400640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b5806bc20 0x7f1b5806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:05:59.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.632+0000 7f1b77400640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b5806bc20 0x7f1b5806e0e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1b68004e40 tx=0x7f1b68007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:05:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.928+0000 7f1b6f400640 1 -- 172.21.15.6:0/2000120136 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1b64057610 con 0x7f1b780ce8a0 2024-03-22T13:05:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:05:59.929+0000 7f1b6f400640 1 -- 172.21.15.6:0/2000120136 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b64059570 con 0x7f1b780ce8a0 2024-03-22T13:06:00.136 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:18.471 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.146 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:18.480 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.149 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:18.539 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.153 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:18.543 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.160 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:10:18.550 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.165 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:10:18.555 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.219 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:18.609 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.264 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:10:18.654 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.266+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b7806de60 con 0x7f1b780ce8a0 2024-03-22T13:06:00.282 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:10:18.617 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:06:00.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.556+0000 7f1b6f400640 1 -- 172.21.15.6:0/2000120136 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f1b6405d610 con 0x7f1b780ce8a0 2024-03-22T13:06:00.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:00.557 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:00.557 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b5806bc20 msgr2=0x7f1b5806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b5806bc20 0x7f1b5806e0e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1b68004e40 tx=0x7f1b68007f10 comp rx=0 tx=0).stop 2024-03-22T13:06:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b780ce8a0 msgr2=0x7f1b781f3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b780ce8a0 0x7f1b781f3740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1b6400e820 tx=0x7f1b6400ece0 comp rx=0 tx=0).stop 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 shutdown_connections 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b5806bc20 0x7f1b5806e0e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b7815ae30 0x7f1b781f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b780ce8a0 0x7f1b781f3740 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 --2- 172.21.15.6:0/2000120136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b780cc380 0x7f1b780b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 >> 172.21.15.6:0/2000120136 conn(0x7f1b780b1480 msgr2=0x7f1b78152970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 shutdown_connections 2024-03-22T13:06:00.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:00.560+0000 7f1b7d000640 1 -- 172.21.15.6:0/2000120136 wait complete. 2024-03-22T13:06:01.184 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:01.184 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:01.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:06:02.002 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:06:02.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:02.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.381+0000 7fba47e00640 1 Processor -- start 2024-03-22T13:06:02.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.381+0000 7fba47e00640 1 -- start start 2024-03-22T13:06:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.382+0000 7fba47e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fba480cc380 0x7fba4806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.382+0000 7fba47e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba480cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.382+0000 7fba47e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4815b260 con 0x7fba480ce8a0 2024-03-22T13:06:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.383+0000 7fba47e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba480cee00 con 0x7fba480cc380 2024-03-22T13:06:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.383+0000 7fba47e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cf0c0 con 0x7fba4815ae30 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.383+0000 7fba46a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fba480cc380 0x7fba4806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.383+0000 7fba47400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba480cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.384+0000 7fba47400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba480cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44176/0 (socket says 172.21.15.6:44176) 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.384+0000 7fba47400640 1 -- 172.21.15.6:0/363337933 learned_addr learned my addr 172.21.15.6:0/363337933 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.387+0000 7fba47400640 1 -- 172.21.15.6:0/363337933 >> v2:172.21.15.92:3300/0 conn(0x7fba480cc380 msgr2=0x7fba4806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.387+0000 7fba47400640 1 --2- 172.21.15.6:0/363337933 >> v2:172.21.15.92:3300/0 conn(0x7fba480cc380 0x7fba4806c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.387+0000 7fba47400640 1 -- 172.21.15.6:0/363337933 >> v1:172.21.15.92:6789/0 conn(0x7fba480ce8a0 legacy=0x7fba480cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.387+0000 7fba47400640 1 -- 172.21.15.6:0/363337933 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480cf3a0 con 0x7fba4815ae30 2024-03-22T13:06:02.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.389+0000 7fba47400640 1 --2- 172.21.15.6:0/363337933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba480cc760 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fba40009e50 tx=0x7fba4002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3624a47de6c4c142 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:02.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.395+0000 7fba45600640 1 -- 172.21.15.6:0/363337933 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4002cb00 con 0x7fba4815ae30 2024-03-22T13:06:02.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.395+0000 7fba45600640 1 -- 172.21.15.6:0/363337933 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4002cca0 con 0x7fba4815ae30 2024-03-22T13:06:02.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.395+0000 7fba45600640 1 -- 172.21.15.6:0/363337933 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40034610 con 0x7fba4815ae30 2024-03-22T13:06:02.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.395+0000 7fba47e00640 1 -- 172.21.15.6:0/363337933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 msgr2=0x7fba480cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.395+0000 7fba47e00640 1 --2- 172.21.15.6:0/363337933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba480cc760 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fba40009e50 tx=0x7fba4002bf60 comp rx=0 tx=0).stop 2024-03-22T13:06:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.396+0000 7fba47e00640 1 -- 172.21.15.6:0/363337933 shutdown_connections 2024-03-22T13:06:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.396+0000 7fba47e00640 1 --2- 172.21.15.6:0/363337933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba480cc760 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.396+0000 7fba47e00640 1 --2- 172.21.15.6:0/363337933 >> v2:172.21.15.92:3300/0 conn(0x7fba480cc380 0x7fba4806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.396+0000 7fba47e00640 1 -- 172.21.15.6:0/363337933 >> 172.21.15.6:0/363337933 conn(0x7fba480b1480 msgr2=0x7fba480b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.396+0000 7fba47e00640 1 -- 172.21.15.6:0/363337933 shutdown_connections 2024-03-22T13:06:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.396+0000 7fba47e00640 1 -- 172.21.15.6:0/363337933 wait complete. 2024-03-22T13:06:02.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.398+0000 7fba47e00640 1 Processor -- start 2024-03-22T13:06:02.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.398+0000 7fba47e00640 1 -- start start 2024-03-22T13:06:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.400+0000 7fba47e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba480cc380 0x7fba4815d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.400+0000 7fba47e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba480ce8a0 0x7fba480ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.400+0000 7fba47e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba4815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.400+0000 7fba47e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4815a910 con 0x7fba480cc380 2024-03-22T13:06:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.400+0000 7fba47e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4815a630 con 0x7fba480ce8a0 2024-03-22T13:06:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.400+0000 7fba47e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba481549d0 con 0x7fba4815ae30 2024-03-22T13:06:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.401+0000 7fba47400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba4815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.401+0000 7fba46a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba480cc380 0x7fba4815d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.401+0000 7fba46000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba480ce8a0 0x7fba480ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.401+0000 7fba47400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba4815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44190/0 (socket says 172.21.15.6:44190) 2024-03-22T13:06:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.401+0000 7fba47400640 1 -- 172.21.15.6:0/813738662 learned_addr learned my addr 172.21.15.6:0/813738662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.406+0000 7fba47400640 1 -- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba480ce8a0 msgr2=0x7fba480ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.406+0000 7fba47400640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba480ce8a0 0x7fba480ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.406+0000 7fba47400640 1 -- 172.21.15.6:0/813738662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba480cc380 msgr2=0x7fba4815d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.406+0000 7fba47400640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba480cc380 0x7fba4815d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.406+0000 7fba47400640 1 -- 172.21.15.6:0/813738662 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba481e48e0 con 0x7fba4815ae30 2024-03-22T13:06:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.409+0000 7fba47400640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba4815b5f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fba40036040 tx=0x7fba40004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.412+0000 7fba3ea00640 1 -- 172.21.15.6:0/813738662 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40043020 con 0x7fba4815ae30 2024-03-22T13:06:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.412+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba481e4bc0 con 0x7fba4815ae30 2024-03-22T13:06:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.412+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba481e5180 con 0x7fba4815ae30 2024-03-22T13:06:02.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.414+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba48075650 con 0x7fba4815ae30 2024-03-22T13:06:02.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.617+0000 7fba3ea00640 1 -- 172.21.15.6:0/813738662 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4002ce50 con 0x7fba4815ae30 2024-03-22T13:06:02.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.689+0000 7fba3ea00640 1 -- 172.21.15.6:0/813738662 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4003e450 con 0x7fba4815ae30 2024-03-22T13:06:02.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.695+0000 7fba3ea00640 1 -- 172.21.15.6:0/813738662 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fba40046020 con 0x7fba4815ae30 2024-03-22T13:06:02.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.697+0000 7fba3ea00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fba2806bc00 0x7fba2806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:02.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.698+0000 7fba3ea00640 1 -- 172.21.15.6:0/813738662 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fba400ad910 con 0x7fba4815ae30 2024-03-22T13:06:02.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.698+0000 7fba46a00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fba2806bc00 0x7fba2806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:02.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.699+0000 7fba46a00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fba2806bc00 0x7fba2806e0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fba38009aa0 tx=0x7fba38009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:02.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:02.713+0000 7fba3ea00640 1 -- 172.21.15.6:0/813738662 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fba4004d030 con 0x7fba4815ae30 2024-03-22T13:06:03.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.010+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba4806de60 con 0x7fba4815ae30 2024-03-22T13:06:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.024+0000 7fba3ea00640 1 -- 172.21.15.6:0/813738662 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fba4007b9a0 con 0x7fba4815ae30 2024-03-22T13:06:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:03.025 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:03.025 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fba2806bc00 msgr2=0x7fba2806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fba2806bc00 0x7fba2806e0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fba38009aa0 tx=0x7fba38009190 comp rx=0 tx=0).stop 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 msgr2=0x7fba4815b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba4815b5f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fba40036040 tx=0x7fba40004a40 comp rx=0 tx=0).stop 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 shutdown_connections 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba4815ae30 0x7fba4815b5f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fba2806bc00 0x7fba2806e0c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba480ce8a0 0x7fba480ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:03.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 --2- 172.21.15.6:0/813738662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba480cc380 0x7fba4815d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:03.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 >> 172.21.15.6:0/813738662 conn(0x7fba480b1480 msgr2=0x7fba480af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:03.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 shutdown_connections 2024-03-22T13:06:03.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:03.027+0000 7fba47e00640 1 -- 172.21.15.6:0/813738662 wait complete. 2024-03-22T13:06:03.650 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:03.650 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:04.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:04.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.847+0000 7f7be3c00640 1 Processor -- start 2024-03-22T13:06:04.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.847+0000 7f7be3c00640 1 -- start start 2024-03-22T13:06:04.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.848+0000 7f7be3c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:04.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.848+0000 7f7be3c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7bdc15b0e0 0x7f7bdc15b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:04.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.848+0000 7f7be3c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bdc06c7a0 con 0x7f7bdc0ca520 2024-03-22T13:06:04.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.849+0000 7f7be3c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc0cd000 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.849+0000 7f7be3c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc06ca80 con 0x7f7bdc0ccad0 2024-03-22T13:06:04.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.849+0000 7f7be1c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7bdc15b0e0 0x7f7bdc15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:04.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.849+0000 7f7bdbe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:04.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.850+0000 7f7be1c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7bdc15b0e0 0x7f7bdc15b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:49192/0 (socket says 172.21.15.6:49192) 2024-03-22T13:06:04.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.850+0000 7f7be1c00640 1 -- 172.21.15.6:0/731942672 learned_addr learned my addr 172.21.15.6:0/731942672 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:04.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.852+0000 7f7be1c00640 1 -- 172.21.15.6:0/731942672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 msgr2=0x7f7bdc06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:04.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.852+0000 7f7be1c00640 1 --2- 172.21.15.6:0/731942672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc06d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:04.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.852+0000 7f7be1c00640 1 -- 172.21.15.6:0/731942672 >> v1:172.21.15.92:6789/0 conn(0x7f7bdc0ca520 legacy=0x7f7bdc0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:04.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.852+0000 7f7be1c00640 1 -- 172.21.15.6:0/731942672 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc06cd60 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.856+0000 7f7be1c00640 1 --2- 172.21.15.6:0/731942672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc15b980 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009e50 tx=0x7f7bd402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c282074cc35afc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:04.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.860+0000 7f7bdb400640 1 -- 172.21.15.6:0/731942672 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd402ca70 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.860+0000 7f7bdb400640 1 -- 172.21.15.6:0/731942672 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd402cc10 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.860+0000 7f7bdb400640 1 -- 172.21.15.6:0/731942672 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd40343f0 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.861+0000 7f7be3c00640 1 -- 172.21.15.6:0/731942672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 msgr2=0x7f7bdc15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:04.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.861+0000 7f7be3c00640 1 --2- 172.21.15.6:0/731942672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc15b980 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009e50 tx=0x7f7bd402bed0 comp rx=0 tx=0).stop 2024-03-22T13:06:04.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.861+0000 7f7be3c00640 1 -- 172.21.15.6:0/731942672 shutdown_connections 2024-03-22T13:06:04.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.861+0000 7f7be3c00640 1 --2- 172.21.15.6:0/731942672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc15b980 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:04.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.861+0000 7f7be3c00640 1 --2- 172.21.15.6:0/731942672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:04.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.861+0000 7f7be3c00640 1 -- 172.21.15.6:0/731942672 >> 172.21.15.6:0/731942672 conn(0x7f7bdc0b16b0 msgr2=0x7f7bdc0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:04.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.861+0000 7f7be3c00640 1 -- 172.21.15.6:0/731942672 shutdown_connections 2024-03-22T13:06:04.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.862+0000 7f7be3c00640 1 -- 172.21.15.6:0/731942672 wait complete. 2024-03-22T13:06:04.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.864+0000 7f7be3c00640 1 Processor -- start 2024-03-22T13:06:04.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.864+0000 7f7be3c00640 1 -- start start 2024-03-22T13:06:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.865+0000 7f7be3c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bdc0ca520 0x7f7bdc0c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.865+0000 7f7be3c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.865+0000 7f7be3c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.865+0000 7f7be3c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc154c20 con 0x7f7bdc0ca520 2024-03-22T13:06:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.865+0000 7f7be3c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc06d320 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.865+0000 7f7be3c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0ca900 con 0x7f7bdc0ccad0 2024-03-22T13:06:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.866+0000 7f7bdbe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.866+0000 7f7be1c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.866+0000 7f7be1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bdc0ca520 0x7f7bdc0c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.866+0000 7f7bdbe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc153ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:37476/0 (socket says 172.21.15.6:37476) 2024-03-22T13:06:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.866+0000 7f7bdbe00640 1 -- 172.21.15.6:0/2085682497 learned_addr learned my addr 172.21.15.6:0/2085682497 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.870+0000 7f7be1c00640 1 -- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 msgr2=0x7f7bdc153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.870+0000 7f7be1c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:04.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.870+0000 7f7be1c00640 1 -- 172.21.15.6:0/2085682497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bdc0ca520 msgr2=0x7f7bdc0c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:04.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.870+0000 7f7be1c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bdc0ca520 0x7f7bdc0c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:04.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.870+0000 7f7be1c00640 1 -- 172.21.15.6:0/2085682497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc0c8a10 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.873+0000 7f7be1c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc154430 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4032000 tx=0x7f7bd4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:04.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.877+0000 7f7bd8c00640 1 -- 172.21.15.6:0/2085682497 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4044020 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.877+0000 7f7bd8c00640 1 -- 172.21.15.6:0/2085682497 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4040670 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.877+0000 7f7bd8c00640 1 -- 172.21.15.6:0/2085682497 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4009510 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.877+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc0c8d70 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.877+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc1f1280 con 0x7f7bdc15b0e0 2024-03-22T13:06:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:04.879+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc0cef10 con 0x7f7bdc15b0e0 2024-03-22T13:06:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.140+0000 7f7bd8c00640 1 -- 172.21.15.6:0/2085682497 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7bd4047020 con 0x7f7bdc15b0e0 2024-03-22T13:06:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.142+0000 7f7bd8c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7bbc06bc20 0x7f7bbc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:05.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.142+0000 7f7be1200640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7bbc06bc20 0x7f7bbc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:05.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.142+0000 7f7bd8c00640 1 -- 172.21.15.6:0/2085682497 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f7bd4079e50 con 0x7f7bdc15b0e0 2024-03-22T13:06:05.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.143+0000 7f7be1200640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7bbc06bc20 0x7f7bbc06e0e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0004f60 tx=0x7f7bd0007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:05.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.154+0000 7f7bd8c00640 1 -- 172.21.15.6:0/2085682497 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7bdc0cef10 con 0x7f7bdc15b0e0 2024-03-22T13:06:05.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.419+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bdc0756c0 con 0x7f7bdc15b0e0 2024-03-22T13:06:05.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.434+0000 7f7bd8c00640 1 -- 172.21.15.6:0/2085682497 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f7bd404e030 con 0x7f7bdc15b0e0 2024-03-22T13:06:05.434 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:05.434 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:05.434 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.436+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7bbc06bc20 msgr2=0x7f7bbc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.436+0000 7f7be3c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7bbc06bc20 0x7f7bbc06e0e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0004f60 tx=0x7f7bd0007e30 comp rx=0 tx=0).stop 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.436+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 msgr2=0x7f7bdc154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.436+0000 7f7be3c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc154430 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4032000 tx=0x7f7bd4004a40 comp rx=0 tx=0).stop 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 shutdown_connections 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7bbc06bc20 0x7f7bbc06e0e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bdc15b0e0 0x7f7bdc154430 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7bdc0ccad0 0x7f7bdc153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 --2- 172.21.15.6:0/2085682497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bdc0ca520 0x7f7bdc0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 >> 172.21.15.6:0/2085682497 conn(0x7f7bdc0b16b0 msgr2=0x7f7bdc15b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 shutdown_connections 2024-03-22T13:06:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:05.437+0000 7f7be3c00640 1 -- 172.21.15.6:0/2085682497 wait complete. 2024-03-22T13:06:06.057 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:06.057 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:07.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.257+0000 7f56eda00640 1 Processor -- start 2024-03-22T13:06:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.257+0000 7f56eda00640 1 -- start start 2024-03-22T13:06:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.259+0000 7f56eda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56e80ce8a0 0x7f56e806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.259+0000 7f56eda00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f56e815ae30 0x7f56e80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.259+0000 7f56eda00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e815b240 con 0x7f56e80cc380 2024-03-22T13:06:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.259+0000 7f56eda00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e80cee00 con 0x7f56e80ce8a0 2024-03-22T13:06:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.259+0000 7f56eda00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e80cf0c0 con 0x7f56e815ae30 2024-03-22T13:06:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.259+0000 7f56df400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56e80ce8a0 0x7f56e806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.259+0000 7f56ed000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f56e815ae30 0x7f56e80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.260+0000 7f56df400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56e80ce8a0 0x7f56e806c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34252/0 (socket says 172.21.15.6:34252) 2024-03-22T13:06:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.260+0000 7f56df400640 1 -- 172.21.15.6:0/3330178213 learned_addr learned my addr 172.21.15.6:0/3330178213 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:07.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.263+0000 7f56df400640 1 -- 172.21.15.6:0/3330178213 >> v2:172.21.15.92:3300/0 conn(0x7f56e815ae30 msgr2=0x7f56e80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:07.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.263+0000 7f56df400640 1 --2- 172.21.15.6:0/3330178213 >> v2:172.21.15.92:3300/0 conn(0x7f56e815ae30 0x7f56e80cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.263+0000 7f56df400640 1 -- 172.21.15.6:0/3330178213 >> v1:172.21.15.92:6789/0 conn(0x7f56e80cc380 legacy=0x7f56e80cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:07.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.263+0000 7f56df400640 1 -- 172.21.15.6:0/3330178213 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e80cf3a0 con 0x7f56e80ce8a0 2024-03-22T13:06:07.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.265+0000 7f56df400640 1 --2- 172.21.15.6:0/3330178213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80ce8a0 0x7f56e806c230 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f56e80c9f50 tx=0x7f56d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a396f3e18af501 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.271+0000 7f56e7400640 1 -- 172.21.15.6:0/3330178213 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0034070 con 0x7f56e80ce8a0 2024-03-22T13:06:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56e7400640 1 -- 172.21.15.6:0/3330178213 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d0030860 con 0x7f56e80ce8a0 2024-03-22T13:06:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56e7400640 1 -- 172.21.15.6:0/3330178213 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0030c30 con 0x7f56e80ce8a0 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56eda00640 1 -- 172.21.15.6:0/3330178213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80ce8a0 msgr2=0x7f56e806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56eda00640 1 --2- 172.21.15.6:0/3330178213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80ce8a0 0x7f56e806c230 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f56e80c9f50 tx=0x7f56d002b9a0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56eda00640 1 -- 172.21.15.6:0/3330178213 shutdown_connections 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56eda00640 1 --2- 172.21.15.6:0/3330178213 >> v2:172.21.15.92:3300/0 conn(0x7f56e815ae30 0x7f56e80cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56eda00640 1 --2- 172.21.15.6:0/3330178213 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80ce8a0 0x7f56e806c230 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56eda00640 1 -- 172.21.15.6:0/3330178213 >> 172.21.15.6:0/3330178213 conn(0x7f56e80b1480 msgr2=0x7f56e80b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.272+0000 7f56eda00640 1 -- 172.21.15.6:0/3330178213 shutdown_connections 2024-03-22T13:06:07.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.273+0000 7f56eda00640 1 -- 172.21.15.6:0/3330178213 wait complete. 2024-03-22T13:06:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.275+0000 7f56eda00640 1 Processor -- start 2024-03-22T13:06:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.275+0000 7f56eda00640 1 -- start start 2024-03-22T13:06:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.277+0000 7f56eda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80cc380 0x7f56e815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.277+0000 7f56eda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56e80ce8a0 0x7f56e815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.277+0000 7f56eda00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56e815ae30 0x7f56e814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.277+0000 7f56eda00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e815a910 con 0x7f56e80cc380 2024-03-22T13:06:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.277+0000 7f56eda00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e815a630 con 0x7f56e80ce8a0 2024-03-22T13:06:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.277+0000 7f56eda00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e81549d0 con 0x7f56e815ae30 2024-03-22T13:06:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.278+0000 7f56e7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80cc380 0x7f56e815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.278+0000 7f56df400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56e80ce8a0 0x7f56e815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.278+0000 7f56ed000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56e815ae30 0x7f56e814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.278+0000 7f56e7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80cc380 0x7f56e815c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34262/0 (socket says 172.21.15.6:34262) 2024-03-22T13:06:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.278+0000 7f56e7e00640 1 -- 172.21.15.6:0/717107120 learned_addr learned my addr 172.21.15.6:0/717107120 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:07.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.282+0000 7f56e7e00640 1 -- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56e815ae30 msgr2=0x7f56e814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.282+0000 7f56e7e00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56e815ae30 0x7f56e814dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.282+0000 7f56e7e00640 1 -- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56e80ce8a0 msgr2=0x7f56e815c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.282+0000 7f56e7e00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56e80ce8a0 0x7f56e815c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.282+0000 7f56e7e00640 1 -- 172.21.15.6:0/717107120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e815b5e0 con 0x7f56e80cc380 2024-03-22T13:06:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.285+0000 7f56e7e00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80cc380 0x7f56e815c360 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56d800c3c0 tx=0x7f56d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.290+0000 7f56e4c00640 1 -- 172.21.15.6:0/717107120 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80052d0 con 0x7f56e80cc380 2024-03-22T13:06:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.290+0000 7f56e4c00640 1 -- 172.21.15.6:0/717107120 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d8005490 con 0x7f56e80cc380 2024-03-22T13:06:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.290+0000 7f56e4c00640 1 -- 172.21.15.6:0/717107120 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8007260 con 0x7f56e80cc380 2024-03-22T13:06:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.290+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e81f0840 con 0x7f56e80cc380 2024-03-22T13:06:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.290+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e815b880 con 0x7f56e80cc380 2024-03-22T13:06:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.292+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e8075650 con 0x7f56e80cc380 2024-03-22T13:06:07.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.320+0000 7f56e4c00640 1 -- 172.21.15.6:0/717107120 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f56d8007400 con 0x7f56e80cc380 2024-03-22T13:06:07.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.322+0000 7f56e4c00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f56c406bc20 0x7f56c406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.322+0000 7f56df400640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f56c406bc20 0x7f56c406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.323+0000 7f56df400640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f56c406bc20 0x7f56c406e0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f56d0036000 tx=0x7f56d0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:07.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.647+0000 7f56e4c00640 1 -- 172.21.15.6:0/717107120 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f56d80568a0 con 0x7f56e80cc380 2024-03-22T13:06:07.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.647+0000 7f56e4c00640 1 -- 172.21.15.6:0/717107120 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f56d8058800 con 0x7f56e80cc380 2024-03-22T13:06:07.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.902+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56e806de60 con 0x7f56e80cc380 2024-03-22T13:06:07.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.924+0000 7f56e4c00640 1 -- 172.21.15.6:0/717107120 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f56d805c8a0 con 0x7f56e80cc380 2024-03-22T13:06:07.924 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:07.925 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:07.925 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:07.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f56c406bc20 msgr2=0x7f56c406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:07.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f56c406bc20 0x7f56c406e0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f56d0036000 tx=0x7f56d0008030 comp rx=0 tx=0).stop 2024-03-22T13:06:07.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80cc380 msgr2=0x7f56e815c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80cc380 0x7f56e815c360 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56d800c3c0 tx=0x7f56d800c880 comp rx=0 tx=0).stop 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 shutdown_connections 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56e815ae30 0x7f56e814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f56c406bc20 0x7f56c406e0e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56e80ce8a0 0x7f56e815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 --2- 172.21.15.6:0/717107120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56e80cc380 0x7f56e815c360 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.927+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 >> 172.21.15.6:0/717107120 conn(0x7f56e80b1480 msgr2=0x7f56e8158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.928+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 shutdown_connections 2024-03-22T13:06:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:07.928+0000 7f56eda00640 1 -- 172.21.15.6:0/717107120 wait complete. 2024-03-22T13:06:08.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:08.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:09.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.759+0000 7f78dec00640 1 Processor -- start 2024-03-22T13:06:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.759+0000 7f78dec00640 1 -- start start 2024-03-22T13:06:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.761+0000 7f78dec00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f78d8152400 0x7f78d815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.761+0000 7f78dec00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d815bc80 0x7f78d80716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.761+0000 7f78dec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78d806f8c0 0x7f78d8071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.761+0000 7f78dec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d80724d0 con 0x7f78d806f8c0 2024-03-22T13:06:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.761+0000 7f78dec00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d81590b0 con 0x7f78d8152400 2024-03-22T13:06:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.761+0000 7f78dec00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80721f0 con 0x7f78d815bc80 2024-03-22T13:06:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.762+0000 7f78d7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d815bc80 0x7f78d80716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.762+0000 7f78d7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d815bc80 0x7f78d80716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:37484/0 (socket says 172.21.15.6:37484) 2024-03-22T13:06:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.762+0000 7f78d7400640 1 -- 172.21.15.6:0/3396259099 learned_addr learned my addr 172.21.15.6:0/3396259099 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.766+0000 7f78d7400640 1 -- 172.21.15.6:0/3396259099 >> v2:172.21.15.92:3300/0 conn(0x7f78d8152400 msgr2=0x7f78d815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.766+0000 7f78d7400640 1 --2- 172.21.15.6:0/3396259099 >> v2:172.21.15.92:3300/0 conn(0x7f78d8152400 0x7f78d815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.766+0000 7f78d7400640 1 -- 172.21.15.6:0/3396259099 >> v2:172.21.15.6:3300/0 conn(0x7f78d806f8c0 msgr2=0x7f78d8071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:09.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.766+0000 7f78d7400640 1 --2- 172.21.15.6:0/3396259099 >> v2:172.21.15.6:3300/0 conn(0x7f78d806f8c0 0x7f78d8071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.766+0000 7f78d7400640 1 -- 172.21.15.6:0/3396259099 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d81582f0 con 0x7f78d815bc80 2024-03-22T13:06:09.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.769+0000 7f78d7400640 1 --2- 172.21.15.6:0/3396259099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d815bc80 0x7f78d80716d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f78d8071390 tx=0x7f78c402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f204c68be6fed2db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:09.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.773+0000 7f78d6a00640 1 -- 172.21.15.6:0/3396259099 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4034070 con 0x7f78d815bc80 2024-03-22T13:06:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.774+0000 7f78d6a00640 1 -- 172.21.15.6:0/3396259099 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4030860 con 0x7f78d815bc80 2024-03-22T13:06:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.774+0000 7f78d6a00640 1 -- 172.21.15.6:0/3396259099 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4030c30 con 0x7f78d815bc80 2024-03-22T13:06:09.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.775+0000 7f78dec00640 1 -- 172.21.15.6:0/3396259099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d815bc80 msgr2=0x7f78d80716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:09.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.775+0000 7f78dec00640 1 --2- 172.21.15.6:0/3396259099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d815bc80 0x7f78d80716d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f78d8071390 tx=0x7f78c402b9a0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.775+0000 7f78dec00640 1 -- 172.21.15.6:0/3396259099 shutdown_connections 2024-03-22T13:06:09.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.775+0000 7f78dec00640 1 --2- 172.21.15.6:0/3396259099 >> v2:172.21.15.6:3300/0 conn(0x7f78d806f8c0 0x7f78d8071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.775+0000 7f78dec00640 1 --2- 172.21.15.6:0/3396259099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d815bc80 0x7f78d80716d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.775+0000 7f78dec00640 1 --2- 172.21.15.6:0/3396259099 >> v2:172.21.15.92:3300/0 conn(0x7f78d8152400 0x7f78d815b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.775+0000 7f78dec00640 1 -- 172.21.15.6:0/3396259099 >> 172.21.15.6:0/3396259099 conn(0x7f78d80cb930 msgr2=0x7f78d80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.776+0000 7f78dec00640 1 -- 172.21.15.6:0/3396259099 shutdown_connections 2024-03-22T13:06:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.776+0000 7f78dec00640 1 -- 172.21.15.6:0/3396259099 wait complete. 2024-03-22T13:06:09.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.778+0000 7f78dec00640 1 Processor -- start 2024-03-22T13:06:09.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.778+0000 7f78dec00640 1 -- start start 2024-03-22T13:06:09.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.780+0000 7f78dec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78d806f8c0 0x7f78d8156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:09.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.780+0000 7f78dec00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d8152400 0x7f78d8144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:09.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.780+0000 7f78dec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d815bc80 0x7f78d814dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:09.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.780+0000 7f78dec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8158b90 con 0x7f78d815bc80 2024-03-22T13:06:09.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.780+0000 7f78dec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81588b0 con 0x7f78d806f8c0 2024-03-22T13:06:09.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.780+0000 7f78dec00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81527e0 con 0x7f78d8152400 2024-03-22T13:06:09.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.781+0000 7f78d7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d8152400 0x7f78d8144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:09.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.781+0000 7f78d7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d8152400 0x7f78d8144340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:37490/0 (socket says 172.21.15.6:37490) 2024-03-22T13:06:09.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.781+0000 7f78d7400640 1 -- 172.21.15.6:0/1054183162 learned_addr learned my addr 172.21.15.6:0/1054183162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:09.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.786+0000 7f78d7400640 1 -- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78d806f8c0 msgr2=0x7f78d8156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:09.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.786+0000 7f78d7400640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78d806f8c0 0x7f78d8156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.786+0000 7f78d7400640 1 -- 172.21.15.6:0/1054183162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d815bc80 msgr2=0x7f78d814dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:09.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.786+0000 7f78d7400640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d815bc80 0x7f78d814dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:09.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.786+0000 7f78d7400640 1 -- 172.21.15.6:0/1054183162 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d814cdd0 con 0x7f78d8152400 2024-03-22T13:06:09.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.789+0000 7f78d7400640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d8152400 0x7f78d8144340 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f78c40090b0 tx=0x7f78c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.792+0000 7f78cfe00640 1 -- 172.21.15.6:0/1054183162 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4034040 con 0x7f78d8152400 2024-03-22T13:06:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.792+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d8157510 con 0x7f78d8152400 2024-03-22T13:06:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.792+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d814d290 con 0x7f78d8152400 2024-03-22T13:06:09.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.794+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d8072750 con 0x7f78d8152400 2024-03-22T13:06:09.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.794+0000 7f78cfe00640 1 -- 172.21.15.6:0/1054183162 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4030860 con 0x7f78d8152400 2024-03-22T13:06:09.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:09.795+0000 7f78cfe00640 1 -- 172.21.15.6:0/1054183162 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4052060 con 0x7f78d8152400 2024-03-22T13:06:10.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.102+0000 7f78cfe00640 1 -- 172.21.15.6:0/1054183162 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f78c4037070 con 0x7f78d8152400 2024-03-22T13:06:10.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.104+0000 7f78cfe00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f78b806bc00 0x7f78b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:10.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.105+0000 7f78d7e00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f78b806bc00 0x7f78b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:10.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.106+0000 7f78d7e00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f78b806bc00 0x7f78b806e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009710 tx=0x7f78c8006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:10.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.107+0000 7f78cfe00640 1 -- 172.21.15.6:0/1054183162 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f78c40797f0 con 0x7f78d8152400 2024-03-22T13:06:10.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.126+0000 7f78cfe00640 1 -- 172.21.15.6:0/1054183162 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f78d8072750 con 0x7f78d8152400 2024-03-22T13:06:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.384+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78d80756c0 con 0x7f78d8152400 2024-03-22T13:06:10.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.400+0000 7f78cfe00640 1 -- 172.21.15.6:0/1054183162 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f78d814cdd0 con 0x7f78d8152400 2024-03-22T13:06:10.400 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:10.400 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:10.400 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f78b806bc00 msgr2=0x7f78b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f78b806bc00 0x7f78b806e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009710 tx=0x7f78c8006c90 comp rx=0 tx=0).stop 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d8152400 msgr2=0x7f78d8144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d8152400 0x7f78d8144340 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f78c40090b0 tx=0x7f78c40046c0 comp rx=0 tx=0).stop 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 shutdown_connections 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78d815bc80 0x7f78d814dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78d8152400 0x7f78d8144340 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f78b806bc00 0x7f78b806e0c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 --2- 172.21.15.6:0/1054183162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78d806f8c0 0x7f78d8156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 >> 172.21.15.6:0/1054183162 conn(0x7f78d80cb930 msgr2=0x7f78d80c8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 shutdown_connections 2024-03-22T13:06:10.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:10.403+0000 7f78dec00640 1 -- 172.21.15.6:0/1054183162 wait complete. 2024-03-22T13:06:11.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:11.024 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:12.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:12.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.227+0000 7fdc1ea00640 1 Processor -- start 2024-03-22T13:06:12.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.227+0000 7fdc1ea00640 1 -- start start 2024-03-22T13:06:12.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.228+0000 7fdc1ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc180ca590 0x7fdc180ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:12.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.229+0000 7fdc1ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1815a790 con 0x7fdc180cdfd0 2024-03-22T13:06:12.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.229+0000 7fdc1ea00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1815aa70 con 0x7fdc180cc5f0 2024-03-22T13:06:12.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.229+0000 7fdc1ea00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc180cdb40 con 0x7fdc180ca590 2024-03-22T13:06:12.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.229+0000 7fdc17400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdc180cc5f0 0x7fdc180cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:40372/0 (socket says 172.21.15.6:40372) 2024-03-22T13:06:12.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.230+0000 7fdc17400640 1 -- 172.21.15.6:0/193944115 learned_addr learned my addr 172.21.15.6:0/193944115 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:12.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.230+0000 7fdc17e00640 1 --2- 172.21.15.6:0/193944115 >> v2:172.21.15.6:3300/0 conn(0x7fdc180ca590 0x7fdc180ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:12.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.233+0000 7fdc16a00640 1 -- 172.21.15.6:0/193944115 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899164610 0 0) 0x7fdc1815aa70 con 0x7fdc180cc5f0 2024-03-22T13:06:12.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.233+0000 7fdc16a00640 1 -- 172.21.15.6:0/193944115 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbfc003520 con 0x7fdc180cc5f0 2024-03-22T13:06:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.235+0000 7fdc17e00640 1 -- 172.21.15.6:0/193944115 >> v1:172.21.15.92:6789/0 conn(0x7fdc180cc5f0 legacy=0x7fdc180cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:12.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.235+0000 7fdc17e00640 1 -- 172.21.15.6:0/193944115 >> v1:172.21.15.6:6789/0 conn(0x7fdc180cdfd0 legacy=0x7fdc180ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:12.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.235+0000 7fdc17e00640 1 -- 172.21.15.6:0/193944115 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1815ad50 con 0x7fdc180ca590 2024-03-22T13:06:12.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.240+0000 7fdc17e00640 1 --2- 172.21.15.6:0/193944115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ca590 0x7fdc180ca970 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdc080097a0 tx=0x7fdc0802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5bf98ead4ac084 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.244+0000 7fdc16a00640 1 -- 172.21.15.6:0/193944115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0802f810 con 0x7fdc180ca590 2024-03-22T13:06:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.244+0000 7fdc16a00640 1 -- 172.21.15.6:0/193944115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0802f9b0 con 0x7fdc180ca590 2024-03-22T13:06:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.244+0000 7fdc16a00640 1 -- 172.21.15.6:0/193944115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0802fd80 con 0x7fdc180ca590 2024-03-22T13:06:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.245+0000 7fdc1ea00640 1 -- 172.21.15.6:0/193944115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ca590 msgr2=0x7fdc180ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.245+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/193944115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ca590 0x7fdc180ca970 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdc080097a0 tx=0x7fdc0802b9f0 comp rx=0 tx=0).stop 2024-03-22T13:06:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.245+0000 7fdc1ea00640 1 -- 172.21.15.6:0/193944115 shutdown_connections 2024-03-22T13:06:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.245+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/193944115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180ca590 0x7fdc180ca970 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.245+0000 7fdc1ea00640 1 -- 172.21.15.6:0/193944115 >> 172.21.15.6:0/193944115 conn(0x7fdc180b1720 msgr2=0x7fdc180b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.245+0000 7fdc1ea00640 1 -- 172.21.15.6:0/193944115 shutdown_connections 2024-03-22T13:06:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.245+0000 7fdc1ea00640 1 -- 172.21.15.6:0/193944115 wait complete. 2024-03-22T13:06:12.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.247+0000 7fdc1ea00640 1 Processor -- start 2024-03-22T13:06:12.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.247+0000 7fdc1ea00640 1 -- start start 2024-03-22T13:06:12.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.248+0000 7fdc1ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc180ca590 0x7fdc180c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:12.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.248+0000 7fdc1ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180cc5f0 0x7fdc180c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:12.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.248+0000 7fdc1ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cdfd0 0x7fdc180c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:12.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.248+0000 7fdc1ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1815a480 con 0x7fdc180cc5f0 2024-03-22T13:06:12.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.248+0000 7fdc1ea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1815b310 con 0x7fdc180ca590 2024-03-22T13:06:12.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.248+0000 7fdc1ea00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180cd4b0 con 0x7fdc180cdfd0 2024-03-22T13:06:12.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.249+0000 7fdc17e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc180ca590 0x7fdc180c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:12.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.249+0000 7fdc17400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180cc5f0 0x7fdc180c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:12.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.249+0000 7fdc1ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cdfd0 0x7fdc180c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:12.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.249+0000 7fdc17400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180cc5f0 0x7fdc180c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34294/0 (socket says 172.21.15.6:34294) 2024-03-22T13:06:12.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.249+0000 7fdc17400640 1 -- 172.21.15.6:0/3654926283 learned_addr learned my addr 172.21.15.6:0/3654926283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:12.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.253+0000 7fdc17e00640 1 -- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cdfd0 msgr2=0x7fdc180c7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:12.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.253+0000 7fdc17e00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cdfd0 0x7fdc180c7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:12.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.253+0000 7fdc17e00640 1 -- 172.21.15.6:0/3654926283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180cc5f0 msgr2=0x7fdc180c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:12.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.253+0000 7fdc17e00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180cc5f0 0x7fdc180c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:12.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.253+0000 7fdc17e00640 1 -- 172.21.15.6:0/3654926283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180c6640 con 0x7fdc180ca590 2024-03-22T13:06:12.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.255+0000 7fdc17e00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc180ca590 0x7fdc180c6100 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdc08033000 tx=0x7fdc080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:12.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.259+0000 7fdc0fe00640 1 -- 172.21.15.6:0/3654926283 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc080378d0 con 0x7fdc180ca590 2024-03-22T13:06:12.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.259+0000 7fdc0fe00640 1 -- 172.21.15.6:0/3654926283 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08037a70 con 0x7fdc180ca590 2024-03-22T13:06:12.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.259+0000 7fdc0fe00640 1 -- 172.21.15.6:0/3654926283 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08037e60 con 0x7fdc180ca590 2024-03-22T13:06:12.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.260+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc181f3730 con 0x7fdc180ca590 2024-03-22T13:06:12.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.260+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc181f3cf0 con 0x7fdc180ca590 2024-03-22T13:06:12.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.261+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc180756c0 con 0x7fdc180ca590 2024-03-22T13:06:12.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.548+0000 7fdc0fe00640 1 -- 172.21.15.6:0/3654926283 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc08049060 con 0x7fdc180ca590 2024-03-22T13:06:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.550+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf806bc50 0x7fdbf806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.551+0000 7fdc17400640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf806bc50 0x7fdbf806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.551+0000 7fdc0fe00640 1 -- 172.21.15.6:0/3654926283 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdc08079090 con 0x7fdc180ca590 2024-03-22T13:06:12.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.552+0000 7fdc17400640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf806bc50 0x7fdbf806e110 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdc18159d40 tx=0x7fdc04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.566+0000 7fdc0fe00640 1 -- 172.21.15.6:0/3654926283 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc08033030 con 0x7fdc180ca590 2024-03-22T13:06:12.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:12.820+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc180c9bc0 con 0x7fdc180ca590 2024-03-22T13:06:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.044+0000 7fdc0fe00640 1 -- 172.21.15.6:0/3654926283 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fdc0807aff0 con 0x7fdc180ca590 2024-03-22T13:06:13.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:13.044 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf806bc50 msgr2=0x7fdbf806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf806bc50 0x7fdbf806e110 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdc18159d40 tx=0x7fdc04009040 comp rx=0 tx=0).stop 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc180ca590 msgr2=0x7fdc180c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc180ca590 0x7fdc180c6100 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdc08033000 tx=0x7fdc080046c0 comp rx=0 tx=0).stop 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 shutdown_connections 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc180cdfd0 0x7fdc180c7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc180cc5f0 0x7fdc180c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdbf806bc50 0x7fdbf806e110 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 --2- 172.21.15.6:0/3654926283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc180ca590 0x7fdc180c6100 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 >> 172.21.15.6:0/3654926283 conn(0x7fdc180b1720 msgr2=0x7fdc180b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 shutdown_connections 2024-03-22T13:06:13.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:13.048+0000 7fdc1ea00640 1 -- 172.21.15.6:0/3654926283 wait complete. 2024-03-22T13:06:13.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:13.679 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:14.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.884+0000 7f5421800640 1 Processor -- start 2024-03-22T13:06:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.884+0000 7f5421800640 1 -- start start 2024-03-22T13:06:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.885+0000 7f5421800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c15b250 0x7f541c0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.885+0000 7f5421800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0ce230 con 0x7f541c0cc790 2024-03-22T13:06:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.885+0000 7f5421800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0ce510 con 0x7f541c0cc3b0 2024-03-22T13:06:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.885+0000 7f5421800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0cd0f0 con 0x7f541c15b250 2024-03-22T13:06:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.886+0000 7f541a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f541c0cc790 0x7f541c15ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59792/0 (socket says 172.21.15.6:59792) 2024-03-22T13:06:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.886+0000 7f541a000640 1 -- 172.21.15.6:0/849886048 learned_addr learned my addr 172.21.15.6:0/849886048 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.886+0000 7f541b400640 1 --2- 172.21.15.6:0/849886048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c15b250 0x7f541c0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:14.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.889+0000 7f541b400640 1 -- 172.21.15.6:0/849886048 >> v1:172.21.15.92:6789/0 conn(0x7f541c0cc3b0 legacy=0x7f541c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:14.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.889+0000 7f541b400640 1 -- 172.21.15.6:0/849886048 >> v1:172.21.15.6:6789/0 conn(0x7f541c0cc790 legacy=0x7f541c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:14.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.889+0000 7f541b400640 1 -- 172.21.15.6:0/849886048 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0ce7f0 con 0x7f541c15b250 2024-03-22T13:06:14.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.892+0000 7f541b400640 1 --2- 172.21.15.6:0/849886048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c15b250 0x7f541c0ccbb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5414009dc0 tx=0x7f541402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=381d8d24d7c5788e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:14.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.897+0000 7f5419600640 1 -- 172.21.15.6:0/849886048 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414039070 con 0x7f541c15b250 2024-03-22T13:06:14.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.897+0000 7f5419600640 1 -- 172.21.15.6:0/849886048 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541402cc40 con 0x7f541c15b250 2024-03-22T13:06:14.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.897+0000 7f5419600640 1 -- 172.21.15.6:0/849886048 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414034440 con 0x7f541c15b250 2024-03-22T13:06:14.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.898+0000 7f5421800640 1 -- 172.21.15.6:0/849886048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c15b250 msgr2=0x7f541c0ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:14.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.898+0000 7f5421800640 1 --2- 172.21.15.6:0/849886048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c15b250 0x7f541c0ccbb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5414009dc0 tx=0x7f541402be90 comp rx=0 tx=0).stop 2024-03-22T13:06:14.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.898+0000 7f5421800640 1 -- 172.21.15.6:0/849886048 shutdown_connections 2024-03-22T13:06:14.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.898+0000 7f5421800640 1 --2- 172.21.15.6:0/849886048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c15b250 0x7f541c0ccbb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:14.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.898+0000 7f5421800640 1 -- 172.21.15.6:0/849886048 >> 172.21.15.6:0/849886048 conn(0x7f541c0b16d0 msgr2=0x7f541c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:14.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.898+0000 7f5421800640 1 -- 172.21.15.6:0/849886048 shutdown_connections 2024-03-22T13:06:14.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.898+0000 7f5421800640 1 -- 172.21.15.6:0/849886048 wait complete. 2024-03-22T13:06:14.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.900+0000 7f5421800640 1 Processor -- start 2024-03-22T13:06:14.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.900+0000 7f5421800640 1 -- start start 2024-03-22T13:06:14.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.901+0000 7f5421800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f541c0cc3b0 0x7f541c1f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:14.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.901+0000 7f5421800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c0cc790 0x7f541c0c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:14.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.901+0000 7f5421800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f541c15b250 0x7f541c157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:14.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.901+0000 7f5421800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c06dc10 con 0x7f541c15b250 2024-03-22T13:06:14.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.901+0000 7f5421800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0cedb0 con 0x7f541c0cc3b0 2024-03-22T13:06:14.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.901+0000 7f5421800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0cf940 con 0x7f541c0cc790 2024-03-22T13:06:14.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.902+0000 7f541a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c0cc790 0x7f541c0c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:14.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.902+0000 7f541b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f541c15b250 0x7f541c157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:14.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.902+0000 7f541aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f541c0cc3b0 0x7f541c1f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:14.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.902+0000 7f541a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c0cc790 0x7f541c0c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53838/0 (socket says 172.21.15.6:53838) 2024-03-22T13:06:14.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.902+0000 7f541a000640 1 -- 172.21.15.6:0/1246472258 learned_addr learned my addr 172.21.15.6:0/1246472258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:14.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.906+0000 7f541a000640 1 -- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f541c0cc3b0 msgr2=0x7f541c1f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:14.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.906+0000 7f541a000640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f541c0cc3b0 0x7f541c1f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:14.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.906+0000 7f541a000640 1 -- 172.21.15.6:0/1246472258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f541c15b250 msgr2=0x7f541c157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:14.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.906+0000 7f541a000640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f541c15b250 0x7f541c157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:14.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.906+0000 7f541a000640 1 -- 172.21.15.6:0/1246472258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0c8050 con 0x7f541c0cc790 2024-03-22T13:06:14.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.908+0000 7f541a000640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c0cc790 0x7f541c0c9b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f540400c4d0 tx=0x7f540400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.913+0000 7f5412a00640 1 -- 172.21.15.6:0/1246472258 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404017070 con 0x7f541c0cc790 2024-03-22T13:06:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.913+0000 7f5412a00640 1 -- 172.21.15.6:0/1246472258 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5404005ce0 con 0x7f541c0cc790 2024-03-22T13:06:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.913+0000 7f5412a00640 1 -- 172.21.15.6:0/1246472258 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404007270 con 0x7f541c0cc790 2024-03-22T13:06:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.913+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c0c8330 con 0x7f541c0cc790 2024-03-22T13:06:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.913+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c154130 con 0x7f541c0cc790 2024-03-22T13:06:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.914+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c0b2010 con 0x7f541c0cc790 2024-03-22T13:06:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.939+0000 7f5412a00640 1 -- 172.21.15.6:0/1246472258 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f540401a070 con 0x7f541c0cc790 2024-03-22T13:06:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.940+0000 7f5412a00640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f806bc20 0x7f53f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.940+0000 7f541aa00640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f806bc20 0x7f53f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:14.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.941+0000 7f541aa00640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f806bc20 0x7f53f806e0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f540c000f60 tx=0x7f540c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:14.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:14.942+0000 7f5412a00640 1 -- 172.21.15.6:0/1246472258 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5404056740 con 0x7f541c0cc790 2024-03-22T13:06:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.253+0000 7f5412a00640 1 -- 172.21.15.6:0/1246472258 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f541c0b2010 con 0x7f541c0cc790 2024-03-22T13:06:15.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.506+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f541c0756c0 con 0x7f541c0cc790 2024-03-22T13:06:15.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.526+0000 7f5412a00640 1 -- 172.21.15.6:0/1246472258 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f54040586a0 con 0x7f541c0cc790 2024-03-22T13:06:15.526 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:15.526 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:15.527 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:15.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f806bc20 msgr2=0x7f53f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f806bc20 0x7f53f806e0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f540c000f60 tx=0x7f540c009040 comp rx=0 tx=0).stop 2024-03-22T13:06:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c0cc790 msgr2=0x7f541c0c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:15.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c0cc790 0x7f541c0c9b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f540400c4d0 tx=0x7f540400c990 comp rx=0 tx=0).stop 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 shutdown_connections 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f541c15b250 0x7f541c157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f541c0cc790 0x7f541c0c9b60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f806bc20 0x7f53f806e0e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.529+0000 7f5421800640 1 --2- 172.21.15.6:0/1246472258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f541c0cc3b0 0x7f541c1f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.530+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 >> 172.21.15.6:0/1246472258 conn(0x7f541c0b16d0 msgr2=0x7f541c0b06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.530+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 shutdown_connections 2024-03-22T13:06:15.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:15.530+0000 7f5421800640 1 -- 172.21.15.6:0/1246472258 wait complete. 2024-03-22T13:06:16.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:16.151 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:17.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.353+0000 7f7c9ca00640 1 Processor -- start 2024-03-22T13:06:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.353+0000 7f7c9ca00640 1 -- start start 2024-03-22T13:06:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.355+0000 7f7c9ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7c980c9c30 0x7f7c980c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.355+0000 7f7c9ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9815a3c0 con 0x7f7c981545a0 2024-03-22T13:06:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.355+0000 7f7c9ca00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c980c4130 con 0x7f7c980c96e0 2024-03-22T13:06:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.355+0000 7f7c9ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c980c4410 con 0x7f7c980c9c30 2024-03-22T13:06:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.356+0000 7f7c96800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7c980c96e0 0x7f7c98164bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35870/0 (socket says 172.21.15.6:35870) 2024-03-22T13:06:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.356+0000 7f7c96800640 1 -- 172.21.15.6:0/3953995448 learned_addr learned my addr 172.21.15.6:0/3953995448 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.357+0000 7f7c8fe00640 1 --2- 172.21.15.6:0/3953995448 >> v2:172.21.15.6:3300/0 conn(0x7f7c980c9c30 0x7f7c980c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.359+0000 7f7c95400640 1 -- 172.21.15.6:0/3953995448 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021037528 0 0) 0x7f7c980c4130 con 0x7f7c980c96e0 2024-03-22T13:06:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.359+0000 7f7c95400640 1 -- 172.21.15.6:0/3953995448 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c003520 con 0x7f7c980c96e0 2024-03-22T13:06:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.363+0000 7f7c8fe00640 1 -- 172.21.15.6:0/3953995448 >> v1:172.21.15.92:6789/0 conn(0x7f7c980c96e0 legacy=0x7f7c98164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.363+0000 7f7c8fe00640 1 -- 172.21.15.6:0/3953995448 >> v1:172.21.15.6:6789/0 conn(0x7f7c981545a0 legacy=0x7f7c980ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.363+0000 7f7c8fe00640 1 -- 172.21.15.6:0/3953995448 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9815a660 con 0x7f7c980c9c30 2024-03-22T13:06:17.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.366+0000 7f7c8fe00640 1 --2- 172.21.15.6:0/3953995448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c980c9c30 0x7f7c980c3af0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7c800097a0 tx=0x7f7c8002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b145bb19ecfa96b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:17.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.373+0000 7f7c95400640 1 -- 172.21.15.6:0/3953995448 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8002f810 con 0x7f7c980c9c30 2024-03-22T13:06:17.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.373+0000 7f7c95400640 1 -- 172.21.15.6:0/3953995448 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8002f9b0 con 0x7f7c980c9c30 2024-03-22T13:06:17.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.373+0000 7f7c95400640 1 -- 172.21.15.6:0/3953995448 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8002fd80 con 0x7f7c980c9c30 2024-03-22T13:06:17.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.373+0000 7f7c9ca00640 1 -- 172.21.15.6:0/3953995448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c980c9c30 msgr2=0x7f7c980c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:17.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.373+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/3953995448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c980c9c30 0x7f7c980c3af0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7c800097a0 tx=0x7f7c8002b9f0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.374+0000 7f7c9ca00640 1 -- 172.21.15.6:0/3953995448 shutdown_connections 2024-03-22T13:06:17.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.374+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/3953995448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c980c9c30 0x7f7c980c3af0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.374+0000 7f7c9ca00640 1 -- 172.21.15.6:0/3953995448 >> 172.21.15.6:0/3953995448 conn(0x7f7c980714f0 msgr2=0x7f7c98071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:17.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.374+0000 7f7c9ca00640 1 -- 172.21.15.6:0/3953995448 shutdown_connections 2024-03-22T13:06:17.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.374+0000 7f7c9ca00640 1 -- 172.21.15.6:0/3953995448 wait complete. 2024-03-22T13:06:17.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.377+0000 7f7c9ca00640 1 Processor -- start 2024-03-22T13:06:17.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.377+0000 7f7c9ca00640 1 -- start start 2024-03-22T13:06:17.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.378+0000 7f7c9ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c980c96e0 0x7f7c981eff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:17.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.378+0000 7f7c9ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c980c9c30 0x7f7c980c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:17.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.378+0000 7f7c9ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c981545a0 0x7f7c981556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:17.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.378+0000 7f7c9ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c98154980 con 0x7f7c981545a0 2024-03-22T13:06:17.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.378+0000 7f7c9ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c980c46f0 con 0x7f7c980c96e0 2024-03-22T13:06:17.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.378+0000 7f7c9ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c980cadc0 con 0x7f7c980c9c30 2024-03-22T13:06:17.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.379+0000 7f7c95e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c980c96e0 0x7f7c981eff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:17.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.379+0000 7f7c8fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c980c9c30 0x7f7c980c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:17.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.379+0000 7f7c96800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c981545a0 0x7f7c981556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:17.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.379+0000 7f7c8fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c980c9c30 0x7f7c980c72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53846/0 (socket says 172.21.15.6:53846) 2024-03-22T13:06:17.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.379+0000 7f7c8fe00640 1 -- 172.21.15.6:0/2382965796 learned_addr learned my addr 172.21.15.6:0/2382965796 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:17.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.383+0000 7f7c8fe00640 1 -- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c980c96e0 msgr2=0x7f7c981eff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:17.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.383+0000 7f7c8fe00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c980c96e0 0x7f7c981eff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.383+0000 7f7c8fe00640 1 -- 172.21.15.6:0/2382965796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c981545a0 msgr2=0x7f7c981556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:17.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.383+0000 7f7c8fe00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c981545a0 0x7f7c981556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.383+0000 7f7c8fe00640 1 -- 172.21.15.6:0/2382965796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c981576f0 con 0x7f7c980c9c30 2024-03-22T13:06:17.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.385+0000 7f7c8fe00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c980c9c30 0x7f7c980c72d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7c8002c000 tx=0x7f7c800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.390+0000 7f7c8e000640 1 -- 172.21.15.6:0/2382965796 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c800378d0 con 0x7f7c980c9c30 2024-03-22T13:06:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.390+0000 7f7c8e000640 1 -- 172.21.15.6:0/2382965796 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80037a70 con 0x7f7c980c9c30 2024-03-22T13:06:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.390+0000 7f7c8e000640 1 -- 172.21.15.6:0/2382965796 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80037e60 con 0x7f7c980c9c30 2024-03-22T13:06:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.390+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c98157990 con 0x7f7c980c9c30 2024-03-22T13:06:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.390+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c980c7e60 con 0x7f7c980c9c30 2024-03-22T13:06:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.391+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c980c4130 con 0x7f7c980c9c30 2024-03-22T13:06:17.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.678+0000 7f7c8e000640 1 -- 172.21.15.6:0/2382965796 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c80040070 con 0x7f7c980c9c30 2024-03-22T13:06:17.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.680+0000 7f7c8e000640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c7406bc50 0x7f7c7406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:17.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.681+0000 7f7c95e00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c7406bc50 0x7f7c7406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:17.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.681+0000 7f7c8e000640 1 -- 172.21.15.6:0/2382965796 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f7c80079360 con 0x7f7c980c9c30 2024-03-22T13:06:17.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.682+0000 7f7c95e00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c7406bc50 0x7f7c7406e110 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7c880016e0 tx=0x7f7c88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:17.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.700+0000 7f7c8e000640 1 -- 172.21.15.6:0/2382965796 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c980c4130 con 0x7f7c980c9c30 2024-03-22T13:06:17.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.958+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c98074c10 con 0x7f7c980c9c30 2024-03-22T13:06:17.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.979+0000 7f7c8e000640 1 -- 172.21.15.6:0/2382965796 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f7c98074c10 con 0x7f7c980c9c30 2024-03-22T13:06:17.979 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:17.979 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:17.979 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:17.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.981+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c7406bc50 msgr2=0x7f7c7406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.981+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c7406bc50 0x7f7c7406e110 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7c880016e0 tx=0x7f7c88009040 comp rx=0 tx=0).stop 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.981+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c980c9c30 msgr2=0x7f7c980c72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.981+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c980c9c30 0x7f7c980c72d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7c8002c000 tx=0x7f7c800046c0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 shutdown_connections 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c981545a0 0x7f7c981556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c980c9c30 0x7f7c980c72d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c7406bc50 0x7f7c7406e110 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 --2- 172.21.15.6:0/2382965796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c980c96e0 0x7f7c981eff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 >> 172.21.15.6:0/2382965796 conn(0x7f7c980714f0 msgr2=0x7f7c980ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 shutdown_connections 2024-03-22T13:06:17.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:17.982+0000 7f7c9ca00640 1 -- 172.21.15.6:0/2382965796 wait complete. 2024-03-22T13:06:18.604 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:18.604 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:19.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:19.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.802+0000 7fcd13200640 1 Processor -- start 2024-03-22T13:06:19.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.802+0000 7fcd13200640 1 -- start start 2024-03-22T13:06:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.803+0000 7fcd13200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1415b7a0 0x7fcd1415e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.803+0000 7fcd13200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1406c440 con 0x7fcd140cc640 2024-03-22T13:06:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.803+0000 7fcd13200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1406c720 con 0x7fcd140cca20 2024-03-22T13:06:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.803+0000 7fcd13200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1406d330 con 0x7fcd1415b7a0 2024-03-22T13:06:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.818+0000 7fcd12800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1415b7a0 0x7fcd1415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.818+0000 7fcd12800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1415b7a0 0x7fcd1415e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53862/0 (socket says 172.21.15.6:53862) 2024-03-22T13:06:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.818+0000 7fcd12800640 1 -- 172.21.15.6:0/3614065994 learned_addr learned my addr 172.21.15.6:0/3614065994 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.821+0000 7fcd12800640 1 -- 172.21.15.6:0/3614065994 >> v1:172.21.15.92:6789/0 conn(0x7fcd140cca20 legacy=0x7fcd140ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.821+0000 7fcd12800640 1 -- 172.21.15.6:0/3614065994 >> v1:172.21.15.6:6789/0 conn(0x7fcd140cc640 legacy=0x7fcd1406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.821+0000 7fcd12800640 1 -- 172.21.15.6:0/3614065994 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1406ca00 con 0x7fcd1415b7a0 2024-03-22T13:06:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.824+0000 7fcd12800640 1 --2- 172.21.15.6:0/3614065994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1415b7a0 0x7fcd1415e260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c009e80 tx=0x7fcd0c02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0cb90ee6605050b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.829+0000 7fcd10a00640 1 -- 172.21.15.6:0/3614065994 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c02caf0 con 0x7fcd1415b7a0 2024-03-22T13:06:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.829+0000 7fcd10a00640 1 -- 172.21.15.6:0/3614065994 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0c02cc90 con 0x7fcd1415b7a0 2024-03-22T13:06:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.829+0000 7fcd10a00640 1 -- 172.21.15.6:0/3614065994 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c034610 con 0x7fcd1415b7a0 2024-03-22T13:06:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.830+0000 7fcd13200640 1 -- 172.21.15.6:0/3614065994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1415b7a0 msgr2=0x7fcd1415e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.830+0000 7fcd13200640 1 --2- 172.21.15.6:0/3614065994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1415b7a0 0x7fcd1415e260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c009e80 tx=0x7fcd0c02bf50 comp rx=0 tx=0).stop 2024-03-22T13:06:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.830+0000 7fcd13200640 1 -- 172.21.15.6:0/3614065994 shutdown_connections 2024-03-22T13:06:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.830+0000 7fcd13200640 1 --2- 172.21.15.6:0/3614065994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1415b7a0 0x7fcd1415e260 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.830+0000 7fcd13200640 1 -- 172.21.15.6:0/3614065994 >> 172.21.15.6:0/3614065994 conn(0x7fcd140cf9c0 msgr2=0x7fcd140cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.830+0000 7fcd13200640 1 -- 172.21.15.6:0/3614065994 shutdown_connections 2024-03-22T13:06:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.830+0000 7fcd13200640 1 -- 172.21.15.6:0/3614065994 wait complete. 2024-03-22T13:06:19.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.832+0000 7fcd13200640 1 Processor -- start 2024-03-22T13:06:19.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.833+0000 7fcd13200640 1 -- start start 2024-03-22T13:06:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.834+0000 7fcd13200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd140cc640 0x7fcd140af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.834+0000 7fcd13200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd140cca20 0x7fcd141e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.834+0000 7fcd13200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd1415b7a0 0x7fcd14151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.834+0000 7fcd13200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd140caa50 con 0x7fcd1415b7a0 2024-03-22T13:06:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.834+0000 7fcd13200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1406cfc0 con 0x7fcd140cca20 2024-03-22T13:06:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.834+0000 7fcd13200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1415b000 con 0x7fcd140cc640 2024-03-22T13:06:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.835+0000 7fcd11e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd140cc640 0x7fcd140af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.835+0000 7fcd11e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd140cc640 0x7fcd140af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53874/0 (socket says 172.21.15.6:53874) 2024-03-22T13:06:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.835+0000 7fcd11e00640 1 -- 172.21.15.6:0/1066017740 learned_addr learned my addr 172.21.15.6:0/1066017740 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:19.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.841+0000 7fcd11e00640 1 -- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd140cca20 msgr2=0x7fcd141e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:19.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.841+0000 7fcd11e00640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd140cca20 0x7fcd141e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:19.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.841+0000 7fcd11e00640 1 -- 172.21.15.6:0/1066017740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd1415b7a0 msgr2=0x7fcd14151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:19.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.841+0000 7fcd11e00640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd1415b7a0 0x7fcd14151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:19.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.841+0000 7fcd11e00640 1 -- 172.21.15.6:0/1066017740 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd140af910 con 0x7fcd140cc640 2024-03-22T13:06:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.844+0000 7fcd11e00640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd140cc640 0x7fcd140af3d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400b3e0 tx=0x7fcd0400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.849+0000 7fcd0a000640 1 -- 172.21.15.6:0/1066017740 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04015020 con 0x7fcd140cc640 2024-03-22T13:06:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.850+0000 7fcd0a000640 1 -- 172.21.15.6:0/1066017740 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd04011850 con 0x7fcd140cc640 2024-03-22T13:06:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.850+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd141f0310 con 0x7fcd140cc640 2024-03-22T13:06:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.850+0000 7fcd0a000640 1 -- 172.21.15.6:0/1066017740 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04011c40 con 0x7fcd140cc640 2024-03-22T13:06:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.850+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd141f08d0 con 0x7fcd140cc640 2024-03-22T13:06:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.851+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd14075650 con 0x7fcd140cc640 2024-03-22T13:06:19.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.877+0000 7fcd0a000640 1 -- 172.21.15.6:0/1066017740 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcd04018020 con 0x7fcd140cc640 2024-03-22T13:06:19.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.878+0000 7fcd0a000640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fccf006bc50 0x7fccf006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:19.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.879+0000 7fcd11400640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fccf006bc50 0x7fccf006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:19.880+0000 7fcd11400640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fccf006bc50 0x7fccf006e110 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcd140ce8a0 tx=0x7fccfc007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:20.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.138+0000 7fcd0a000640 1 -- 172.21.15.6:0/1066017740 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fcd04011de0 con 0x7fcd140cc640 2024-03-22T13:06:20.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.150+0000 7fcd0a000640 1 -- 172.21.15.6:0/1066017740 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcd0401f030 con 0x7fcd140cc640 2024-03-22T13:06:20.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.394+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd140af100 con 0x7fcd140cc640 2024-03-22T13:06:20.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.415+0000 7fcd0a000640 1 -- 172.21.15.6:0/1066017740 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fcd040595f0 con 0x7fcd140cc640 2024-03-22T13:06:20.415 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:20.416 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:20.416 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fccf006bc50 msgr2=0x7fccf006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fccf006bc50 0x7fccf006e110 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcd140ce8a0 tx=0x7fccfc007990 comp rx=0 tx=0).stop 2024-03-22T13:06:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd140cc640 msgr2=0x7fcd140af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd140cc640 0x7fcd140af3d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400b3e0 tx=0x7fcd0400b8a0 comp rx=0 tx=0).stop 2024-03-22T13:06:20.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 shutdown_connections 2024-03-22T13:06:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd1415b7a0 0x7fcd14151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fccf006bc50 0x7fccf006e110 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd140cca20 0x7fcd141e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 --2- 172.21.15.6:0/1066017740 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd140cc640 0x7fcd140af3d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 >> 172.21.15.6:0/1066017740 conn(0x7fcd140cf9c0 msgr2=0x7fcd14153c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 shutdown_connections 2024-03-22T13:06:20.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:20.418+0000 7fcd13200640 1 -- 172.21.15.6:0/1066017740 wait complete. 2024-03-22T13:06:21.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:21.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:22.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.229+0000 7f1a25600640 1 Processor -- start 2024-03-22T13:06:22.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.229+0000 7f1a25600640 1 -- start start 2024-03-22T13:06:22.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.230+0000 7f1a25600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2006c230 0x7f1a2015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.230+0000 7f1a25600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a200cee00 con 0x7f1a200cc380 2024-03-22T13:06:22.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.230+0000 7f1a25600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a200cf0c0 con 0x7f1a200ce8a0 2024-03-22T13:06:22.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.230+0000 7f1a25600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2006c660 con 0x7f1a2006c230 2024-03-22T13:06:22.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.233+0000 7f1a17400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1a200ce8a0 0x7f1a20162b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35884/0 (socket says 172.21.15.6:35884) 2024-03-22T13:06:22.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.233+0000 7f1a17400640 1 -- 172.21.15.6:0/1708691534 learned_addr learned my addr 172.21.15.6:0/1708691534 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:22.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.235+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167000476 0 0) 0x7f1a200cf0c0 con 0x7f1a200ce8a0 2024-03-22T13:06:22.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.235+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a04003520 con 0x7f1a200ce8a0 2024-03-22T13:06:22.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.237+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657610330 0 0) 0x7f1a200cee00 con 0x7f1a200cc380 2024-03-22T13:06:22.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.237+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a200cf0c0 con 0x7f1a200cc380 2024-03-22T13:06:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181257218 0 0) 0x7f1a04003520 con 0x7f1a200ce8a0 2024-03-22T13:06:22.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2006c230 msgr2=0x7f1a2015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2006c230 0x7f1a2015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 >> v1:172.21.15.6:6789/0 conn(0x7f1a200cc380 legacy=0x7f1a200cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a200cf3a0 con 0x7f1a200ce8a0 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f1a08001670 con 0x7f1a200ce8a0 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 >> v1:172.21.15.92:6789/0 conn(0x7f1a200ce8a0 legacy=0x7f1a20162b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a04003b40 0x7f1a04003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2006c230 0x7f1a04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a040051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2006c660 con 0x7f1a04004d90 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0400bb80 con 0x7f1a04003b40 2024-03-22T13:06:22.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.240+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0400bd80 con 0x7f1a2006c230 2024-03-22T13:06:22.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.242+0000 7f1a1fe00640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a04003b40 0x7f1a04003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:22.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.243+0000 7f1a24c00640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a040051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:22.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.246+0000 7f1a24c00640 1 -- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2006c230 msgr2=0x7f1a04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:22.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.246+0000 7f1a24c00640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a2006c230 0x7f1a04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.246+0000 7f1a24c00640 1 -- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a04003b40 msgr2=0x7f1a04003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:22.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.246+0000 7f1a24c00640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a04003b40 0x7f1a04003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.246+0000 7f1a24c00640 1 -- 172.21.15.6:0/1708691534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04003520 con 0x7f1a04004d90 2024-03-22T13:06:22.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.249+0000 7f1a24c00640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a040051f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009e40 tx=0x7f1a04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87544d9fc21964f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:22.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.254+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18039070 con 0x7f1a04004d90 2024-03-22T13:06:22.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.254+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a18006e50 con 0x7f1a04004d90 2024-03-22T13:06:22.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.254+0000 7f1a1f400640 1 -- 172.21.15.6:0/1708691534 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18009a10 con 0x7f1a04004d90 2024-03-22T13:06:22.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.254+0000 7f1a25600640 1 -- 172.21.15.6:0/1708691534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 msgr2=0x7f1a040051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:22.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.254+0000 7f1a25600640 1 --2- 172.21.15.6:0/1708691534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a040051f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009e40 tx=0x7f1a04005830 comp rx=0 tx=0).stop 2024-03-22T13:06:22.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.255+0000 7f1a24c00640 1 -- 172.21.15.6:0/1708691534 reap_dead start 2024-03-22T13:06:22.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.255+0000 7f1a25600640 1 -- 172.21.15.6:0/1708691534 shutdown_connections 2024-03-22T13:06:22.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.255+0000 7f1a25600640 1 -- 172.21.15.6:0/1708691534 >> 172.21.15.6:0/1708691534 conn(0x7f1a200b1480 msgr2=0x7f1a200b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:22.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.255+0000 7f1a25600640 1 -- 172.21.15.6:0/1708691534 shutdown_connections 2024-03-22T13:06:22.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.255+0000 7f1a25600640 1 -- 172.21.15.6:0/1708691534 wait complete. 2024-03-22T13:06:22.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.257+0000 7f1a25600640 1 Processor -- start 2024-03-22T13:06:22.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.257+0000 7f1a25600640 1 -- start start 2024-03-22T13:06:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a25600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a201e7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a25600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a20151960 0x7f1a201f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a25600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a201f0c20 0x7f1a200c1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a25600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2015a910 con 0x7f1a04004d90 2024-03-22T13:06:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a25600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2015a630 con 0x7f1a20151960 2024-03-22T13:06:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a25600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a201549d0 con 0x7f1a201f0c20 2024-03-22T13:06:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a17400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a20151960 0x7f1a201f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:22.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.258+0000 7f1a1fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a201e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.259+0000 7f1a17400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a20151960 0x7f1a201f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56400/0 (socket says 172.21.15.6:56400) 2024-03-22T13:06:22.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.259+0000 7f1a17400640 1 -- 172.21.15.6:0/2844106419 learned_addr learned my addr 172.21.15.6:0/2844106419 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:22.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.264+0000 7f1a17400640 1 -- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a201f0c20 msgr2=0x7f1a200c1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.264+0000 7f1a17400640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a201f0c20 0x7f1a200c1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.264+0000 7f1a17400640 1 -- 172.21.15.6:0/2844106419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 msgr2=0x7f1a201e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.264+0000 7f1a17400640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a201e7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.264+0000 7f1a17400640 1 -- 172.21.15.6:0/2844106419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a201f10a0 con 0x7f1a20151960 2024-03-22T13:06:22.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.267+0000 7f1a17400640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a20151960 0x7f1a201f06e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800c780 tx=0x7f1a0800cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:22.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.272+0000 7f1a1cc00640 1 -- 172.21.15.6:0/2844106419 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a08018070 con 0x7f1a20151960 2024-03-22T13:06:22.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.272+0000 7f1a1cc00640 1 -- 172.21.15.6:0/2844106419 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a080072d0 con 0x7f1a20151960 2024-03-22T13:06:22.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.272+0000 7f1a1cc00640 1 -- 172.21.15.6:0/2844106419 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a080047e0 con 0x7f1a20151960 2024-03-22T13:06:22.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.272+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a201f3980 con 0x7f1a20151960 2024-03-22T13:06:22.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.272+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a201f3f20 con 0x7f1a20151960 2024-03-22T13:06:22.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.274+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a20075650 con 0x7f1a20151960 2024-03-22T13:06:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.305+0000 7f1a1cc00640 1 -- 172.21.15.6:0/2844106419 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a0801b070 con 0x7f1a20151960 2024-03-22T13:06:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.307+0000 7f1a1cc00640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f19fc06bc50 0x7f19fc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.307+0000 7f1a1fe00640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f19fc06bc50 0x7f19fc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.308+0000 7f1a1fe00640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f19fc06bc50 0x7f19fc06e110 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a100018d0 tx=0x7f1a10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.310+0000 7f1a1cc00640 1 -- 172.21.15.6:0/2844106419 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1a080569c0 con 0x7f1a20151960 2024-03-22T13:06:22.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.570+0000 7f1a1cc00640 1 -- 172.21.15.6:0/2844106419 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a08022020 con 0x7f1a20151960 2024-03-22T13:06:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.817+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a20158bf0 con 0x7f1a20151960 2024-03-22T13:06:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.829+0000 7f1a1cc00640 1 -- 172.21.15.6:0/2844106419 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f1a201f10a0 con 0x7f1a20151960 2024-03-22T13:06:22.829 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:22.829 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:05:18.182604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:06:22.829 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-03-22T13:06:22.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f19fc06bc50 msgr2=0x7f19fc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f19fc06bc50 0x7f19fc06e110 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a100018d0 tx=0x7f1a10009040 comp rx=0 tx=0).stop 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a20151960 msgr2=0x7f1a201f06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a20151960 0x7f1a201f06e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800c780 tx=0x7f1a0800cb80 comp rx=0 tx=0).stop 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 shutdown_connections 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a201f0c20 0x7f1a200c1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f19fc06bc50 0x7f19fc06e110 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a20151960 0x7f1a201f06e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 --2- 172.21.15.6:0/2844106419 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a04004d90 0x7f1a201e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.832+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 >> 172.21.15.6:0/2844106419 conn(0x7f1a200b1480 msgr2=0x7f1a20149e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.833+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 shutdown_connections 2024-03-22T13:06:22.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:22.833+0000 7f1a25600640 1 -- 172.21.15.6:0/2844106419 wait complete. 2024-03-22T13:06:23.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:05:18.182604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:06:23.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:24.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:24.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.661+0000 7f67dfe00640 1 Processor -- start 2024-03-22T13:06:24.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.661+0000 7f67dfe00640 1 -- start start 2024-03-22T13:06:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.663+0000 7f67dfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f67d80ca590 0x7f67d80ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.663+0000 7f67dfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d815a790 con 0x7f67d80cdfd0 2024-03-22T13:06:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.663+0000 7f67dfe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d815aa70 con 0x7f67d80cc5f0 2024-03-22T13:06:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.663+0000 7f67dfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d80cdb40 con 0x7f67d80ca590 2024-03-22T13:06:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.664+0000 7f67dca00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f67d80cc5f0 0x7f67d80cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43294/0 (socket says 172.21.15.6:43294) 2024-03-22T13:06:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.664+0000 7f67dca00640 1 -- 172.21.15.6:0/3761131802 learned_addr learned my addr 172.21.15.6:0/3761131802 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:24.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.664+0000 7f67dd400640 1 --2- 172.21.15.6:0/3761131802 >> v2:172.21.15.6:3300/0 conn(0x7f67d80ca590 0x7f67d80ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:24.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.667+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961431937 0 0) 0x7f67d815aa70 con 0x7f67d80cc5f0 2024-03-22T13:06:24.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.667+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c0003520 con 0x7f67d80cc5f0 2024-03-22T13:06:24.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 836449897 0 0) 0x7f67c0003520 con 0x7f67d80cc5f0 2024-03-22T13:06:24.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 >> v2:172.21.15.6:3300/0 conn(0x7f67d80ca590 msgr2=0x7f67d80ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:24.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 --2- 172.21.15.6:0/3761131802 >> v2:172.21.15.6:3300/0 conn(0x7f67d80ca590 0x7f67d80ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:24.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 >> v1:172.21.15.6:6789/0 conn(0x7f67d80cdfd0 legacy=0x7f67d80ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:24.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d815ad50 con 0x7f67d80cc5f0 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f67c4001670 con 0x7f67d80cc5f0 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 >> v1:172.21.15.92:6789/0 conn(0x7f67d80cc5f0 legacy=0x7f67d80cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00039b0 0x7f67c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67c0004800 0x7f67c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67c00051c0 0x7f67c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80cdb40 con 0x7f67c0004800 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c000c070 con 0x7f67c00039b0 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.671+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c000c270 con 0x7f67c00051c0 2024-03-22T13:06:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.672+0000 7f67dde00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67c00051c0 0x7f67c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.674+0000 7f67dd400640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00039b0 0x7f67c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.676+0000 7f67dde00640 1 -- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00039b0 msgr2=0x7f67c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.676+0000 7f67dde00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00039b0 0x7f67c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.676+0000 7f67dde00640 1 -- 172.21.15.6:0/3761131802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67c0004800 msgr2=0x7f67c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.676+0000 7f67dde00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67c0004800 0x7f67c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.676+0000 7f67dde00640 1 -- 172.21.15.6:0/3761131802 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c0003520 con 0x7f67c00051c0 2024-03-22T13:06:24.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.677+0000 7f67dde00640 1 -- 172.21.15.6:0/3761131802 reap_dead start 2024-03-22T13:06:24.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.679+0000 7f67dde00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67c00051c0 0x7f67c000b9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f67d0001750 tx=0x7f67c0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aaf3fbd0446d2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:24.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.684+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0001370 con 0x7f67c00051c0 2024-03-22T13:06:24.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.684+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d00052f0 con 0x7f67c00051c0 2024-03-22T13:06:24.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.684+0000 7f67d7e00640 1 -- 172.21.15.6:0/3761131802 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d0003830 con 0x7f67c00051c0 2024-03-22T13:06:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.685+0000 7f67dfe00640 1 -- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67c00051c0 msgr2=0x7f67c000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.685+0000 7f67dfe00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67c00051c0 0x7f67c000b9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f67d0001750 tx=0x7f67c0005830 comp rx=0 tx=0).stop 2024-03-22T13:06:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.685+0000 7f67dfe00640 1 -- 172.21.15.6:0/3761131802 shutdown_connections 2024-03-22T13:06:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.685+0000 7f67dfe00640 1 --2- 172.21.15.6:0/3761131802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67c00051c0 0x7f67c000b9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.685+0000 7f67dfe00640 1 -- 172.21.15.6:0/3761131802 >> 172.21.15.6:0/3761131802 conn(0x7f67d80b1720 msgr2=0x7f67d80b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.685+0000 7f67dfe00640 1 -- 172.21.15.6:0/3761131802 shutdown_connections 2024-03-22T13:06:24.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.685+0000 7f67dfe00640 1 -- 172.21.15.6:0/3761131802 wait complete. 2024-03-22T13:06:24.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.688+0000 7f67dfe00640 1 Processor -- start 2024-03-22T13:06:24.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.688+0000 7f67dfe00640 1 -- start start 2024-03-22T13:06:24.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.689+0000 7f67dfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00051c0 0x7f67d8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.689+0000 7f67dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d80b3c50 0x7f67d80c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.689+0000 7f67dfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67d8156950 0x7f67d8156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.689+0000 7f67dfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d815a480 con 0x7f67d80b3c50 2024-03-22T13:06:24.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.689+0000 7f67dfe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d815b310 con 0x7f67c00051c0 2024-03-22T13:06:24.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.689+0000 7f67dfe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80cd4b0 con 0x7f67d8156950 2024-03-22T13:06:24.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.690+0000 7f67dde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67d8156950 0x7f67d8156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:24.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.690+0000 7f67dd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00051c0 0x7f67d8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:24.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.690+0000 7f67dca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d80b3c50 0x7f67d80c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:24.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.691+0000 7f67dde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67d8156950 0x7f67d8156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35706/0 (socket says 172.21.15.6:35706) 2024-03-22T13:06:24.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.691+0000 7f67dde00640 1 -- 172.21.15.6:0/787957857 learned_addr learned my addr 172.21.15.6:0/787957857 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:24.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.694+0000 7f67dd400640 1 -- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67d8156950 msgr2=0x7f67d8156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:24.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.694+0000 7f67dd400640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67d8156950 0x7f67d8156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:24.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.694+0000 7f67dd400640 1 -- 172.21.15.6:0/787957857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d80b3c50 msgr2=0x7f67d80c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:24.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.694+0000 7f67dd400640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d80b3c50 0x7f67d80c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:24.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.694+0000 7f67dd400640 1 -- 172.21.15.6:0/787957857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80b2910 con 0x7f67c00051c0 2024-03-22T13:06:24.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.697+0000 7f67dd400640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00051c0 0x7f67d8155cd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f67cc011c50 tx=0x7f67cc00f580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.701+0000 7f67d5600640 1 -- 172.21.15.6:0/787957857 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc00fea0 con 0x7f67c00051c0 2024-03-22T13:06:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.701+0000 7f67d5600640 1 -- 172.21.15.6:0/787957857 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc015630 con 0x7f67c00051c0 2024-03-22T13:06:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.701+0000 7f67d5600640 1 -- 172.21.15.6:0/787957857 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc015a20 con 0x7f67c00051c0 2024-03-22T13:06:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.701+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d8157340 con 0x7f67c00051c0 2024-03-22T13:06:24.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.701+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d80b2b80 con 0x7f67c00051c0 2024-03-22T13:06:24.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.702+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d80756c0 con 0x7f67c00051c0 2024-03-22T13:06:24.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.872+0000 7f67d5600640 1 -- 172.21.15.6:0/787957857 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f67cc015bc0 con 0x7f67c00051c0 2024-03-22T13:06:24.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.875+0000 7f67d5600640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f67b806bc20 0x7f67b806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:24.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.875+0000 7f67dca00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f67b806bc20 0x7f67b806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:24.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:24.876+0000 7f67dca00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f67b806bc20 0x7f67b806e0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f67c4001050 tx=0x7f67c4001ae0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:25.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.014+0000 7f67d5600640 1 -- 172.21.15.6:0/787957857 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f67cc0585d0 con 0x7f67c00051c0 2024-03-22T13:06:25.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.046+0000 7f67d5600640 1 -- 172.21.15.6:0/787957857 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f67cc019030 con 0x7f67c00051c0 2024-03-22T13:06:25.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.355+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67d80c9bc0 con 0x7f67c00051c0 2024-03-22T13:06:25.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.369+0000 7f67d5600640 1 -- 172.21.15.6:0/787957857 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f67cc010040 con 0x7f67c00051c0 2024-03-22T13:06:25.369 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:25.369 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:25.369 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:25.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.372+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f67b806bc20 msgr2=0x7f67b806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.372+0000 7f67dfe00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f67b806bc20 0x7f67b806e0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f67c4001050 tx=0x7f67c4001ae0 comp rx=0 tx=0).stop 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.372+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00051c0 msgr2=0x7f67d8155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.372+0000 7f67dfe00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00051c0 0x7f67d8155cd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f67cc011c50 tx=0x7f67cc00f580 comp rx=0 tx=0).stop 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 shutdown_connections 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f67d8156950 0x7f67d8156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f67d80b3c50 0x7f67d80c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f67b806bc20 0x7f67b806e0e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 --2- 172.21.15.6:0/787957857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67c00051c0 0x7f67d8155cd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 >> 172.21.15.6:0/787957857 conn(0x7f67d80b1720 msgr2=0x7f67d81599b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 shutdown_connections 2024-03-22T13:06:25.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:25.373+0000 7f67dfe00640 1 -- 172.21.15.6:0/787957857 wait complete. 2024-03-22T13:06:25.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:25.994 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:26.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:27.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.187+0000 7fb6b3200640 1 Processor -- start 2024-03-22T13:06:27.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.187+0000 7fb6b3200640 1 -- start start 2024-03-22T13:06:27.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.188+0000 7fb6b3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb6ac154240 0x7fb6ac15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:27.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.188+0000 7fb6b3200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb6ac0c7ee0 0x7fb6ac0c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:27.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.188+0000 7fb6b3200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac154620 con 0x7fb6ac0c68c0 2024-03-22T13:06:27.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.188+0000 7fb6b3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ac15e130 con 0x7fb6ac154240 2024-03-22T13:06:27.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.188+0000 7fb6b3200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ac0c6e00 con 0x7fb6ac0c7ee0 2024-03-22T13:06:27.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.189+0000 7fb6abe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb6ac154240 0x7fb6ac15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:27.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.189+0000 7fb6b1200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb6ac0c68c0 0x7fb6ac06b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43298/0 (socket says 172.21.15.6:43298) 2024-03-22T13:06:27.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.189+0000 7fb6b1200640 1 -- 172.21.15.6:0/3186706458 learned_addr learned my addr 172.21.15.6:0/3186706458 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:27.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.190+0000 7fb6ab400640 1 --2- 172.21.15.6:0/3186706458 >> v2:172.21.15.92:3300/0 conn(0x7fb6ac0c7ee0 0x7fb6ac0c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.193+0000 7fb6abe00640 1 -- 172.21.15.6:0/3186706458 >> v2:172.21.15.92:3300/0 conn(0x7fb6ac0c7ee0 msgr2=0x7fb6ac0c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.193+0000 7fb6abe00640 1 --2- 172.21.15.6:0/3186706458 >> v2:172.21.15.92:3300/0 conn(0x7fb6ac0c7ee0 0x7fb6ac0c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.193+0000 7fb6abe00640 1 -- 172.21.15.6:0/3186706458 >> v1:172.21.15.92:6789/0 conn(0x7fb6ac0c68c0 legacy=0x7fb6ac06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.193+0000 7fb6abe00640 1 -- 172.21.15.6:0/3186706458 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac0c70e0 con 0x7fb6ac154240 2024-03-22T13:06:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.196+0000 7fb6abe00640 1 --2- 172.21.15.6:0/3186706458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac15ac90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac06be90 tx=0x7fb69c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1674522783aa4e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:27.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.202+0000 7fb6aaa00640 1 -- 172.21.15.6:0/3186706458 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c034070 con 0x7fb6ac154240 2024-03-22T13:06:27.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.202+0000 7fb6aaa00640 1 -- 172.21.15.6:0/3186706458 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c030610 con 0x7fb6ac154240 2024-03-22T13:06:27.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.202+0000 7fb6aaa00640 1 -- 172.21.15.6:0/3186706458 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0309e0 con 0x7fb6ac154240 2024-03-22T13:06:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.202+0000 7fb6b3200640 1 -- 172.21.15.6:0/3186706458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 msgr2=0x7fb6ac15ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.202+0000 7fb6b3200640 1 --2- 172.21.15.6:0/3186706458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac15ac90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac06be90 tx=0x7fb69c02b960 comp rx=0 tx=0).stop 2024-03-22T13:06:27.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.203+0000 7fb6b3200640 1 -- 172.21.15.6:0/3186706458 shutdown_connections 2024-03-22T13:06:27.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.203+0000 7fb6b3200640 1 --2- 172.21.15.6:0/3186706458 >> v2:172.21.15.92:3300/0 conn(0x7fb6ac0c7ee0 0x7fb6ac0c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.203+0000 7fb6b3200640 1 --2- 172.21.15.6:0/3186706458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac15ac90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.203+0000 7fb6b3200640 1 -- 172.21.15.6:0/3186706458 >> 172.21.15.6:0/3186706458 conn(0x7fb6ac0cb930 msgr2=0x7fb6ac0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:27.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.203+0000 7fb6b3200640 1 -- 172.21.15.6:0/3186706458 shutdown_connections 2024-03-22T13:06:27.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.203+0000 7fb6b3200640 1 -- 172.21.15.6:0/3186706458 wait complete. 2024-03-22T13:06:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.205+0000 7fb6b3200640 1 Processor -- start 2024-03-22T13:06:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.205+0000 7fb6b3200640 1 -- start start 2024-03-22T13:06:27.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c68c0 0x7fb6ac1effd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c7ee0 0x7fb6ac0ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac0719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6b3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac15ddc0 con 0x7fb6ac154240 2024-03-22T13:06:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6b3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac15dae0 con 0x7fb6ac0c68c0 2024-03-22T13:06:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6b3200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac15e5c0 con 0x7fb6ac0c7ee0 2024-03-22T13:06:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6abe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c68c0 0x7fb6ac1effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6b1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac0719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.206+0000 7fb6ab400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c7ee0 0x7fb6ac0ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.207+0000 7fb6abe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c68c0 0x7fb6ac1effd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40800/0 (socket says 172.21.15.6:40800) 2024-03-22T13:06:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.207+0000 7fb6abe00640 1 -- 172.21.15.6:0/820402973 learned_addr learned my addr 172.21.15.6:0/820402973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.210+0000 7fb6b1200640 1 -- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c7ee0 msgr2=0x7fb6ac0ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.210+0000 7fb6b1200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c7ee0 0x7fb6ac0ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.210+0000 7fb6b1200640 1 -- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c68c0 msgr2=0x7fb6ac1effd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.210+0000 7fb6b1200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c68c0 0x7fb6ac1effd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.210+0000 7fb6b1200640 1 -- 172.21.15.6:0/820402973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac0c9d90 con 0x7fb6ac154240 2024-03-22T13:06:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.211+0000 7fb6abe00640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c68c0 0x7fb6ac1effd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:06:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.213+0000 7fb6b1200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac0719b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400cad0 tx=0x7fb6a400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.217+0000 7fb6a3e00640 1 -- 172.21.15.6:0/820402973 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4017070 con 0x7fb6ac154240 2024-03-22T13:06:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.218+0000 7fb6a3e00640 1 -- 172.21.15.6:0/820402973 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a400ac30 con 0x7fb6ac154240 2024-03-22T13:06:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.218+0000 7fb6a3e00640 1 -- 172.21.15.6:0/820402973 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4012400 con 0x7fb6ac154240 2024-03-22T13:06:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.218+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac0cade0 con 0x7fb6ac154240 2024-03-22T13:06:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.218+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac0cb3a0 con 0x7fb6ac154240 2024-03-22T13:06:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.219+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac0756c0 con 0x7fb6ac154240 2024-03-22T13:06:27.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.247+0000 7fb6a3e00640 1 -- 172.21.15.6:0/820402973 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb6a401a070 con 0x7fb6ac154240 2024-03-22T13:06:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.248+0000 7fb6a3e00640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.249+0000 7fb6abe00640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:27.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.249+0000 7fb6abe00640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009560 tx=0x7fb69c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:27.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.564+0000 7fb6a3e00640 1 -- 172.21.15.6:0/820402973 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb6a40570d0 con 0x7fb6ac154240 2024-03-22T13:06:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.565+0000 7fb6a3e00640 1 -- 172.21.15.6:0/820402973 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb6a4059030 con 0x7fb6ac154240 2024-03-22T13:06:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.844+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6ac0c7ab0 con 0x7fb6ac154240 2024-03-22T13:06:27.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.927+0000 7fb6a3e00640 1 -- 172.21.15.6:0/820402973 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fb6a405d0d0 con 0x7fb6ac154240 2024-03-22T13:06:27.929 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:27.929 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:27.929 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 msgr2=0x7fb68c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009560 tx=0x7fb69c03b040 comp rx=0 tx=0).stop 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 msgr2=0x7fb6ac0719b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac0719b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400cad0 tx=0x7fb6a400cf90 comp rx=0 tx=0).stop 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 shutdown_connections 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6ac154240 0x7fb6ac0719b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6ac0c7ee0 0x7fb6ac0ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb68c06bc50 0x7fb68c06e110 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 --2- 172.21.15.6:0/820402973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6ac0c68c0 0x7fb6ac1effd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 >> 172.21.15.6:0/820402973 conn(0x7fb6ac0cb930 msgr2=0x7fb6ac0c5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:27.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 shutdown_connections 2024-03-22T13:06:27.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:27.931+0000 7fb6b3200640 1 -- 172.21.15.6:0/820402973 wait complete. 2024-03-22T13:06:28.555 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:28.557 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:29.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.754+0000 7fc419800640 1 Processor -- start 2024-03-22T13:06:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.755+0000 7fc419800640 1 -- start start 2024-03-22T13:06:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.756+0000 7fc419800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc4141589f0 0x7fc41415f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.756+0000 7fc419800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc414071dc0 0x7fc41406b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.756+0000 7fc419800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc414158dd0 con 0x7fc4140c7640 2024-03-22T13:06:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.756+0000 7fc419800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc414161c10 con 0x7fc4141589f0 2024-03-22T13:06:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.756+0000 7fc419800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140c7b40 con 0x7fc414071dc0 2024-03-22T13:06:29.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.757+0000 7fc412a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc4141589f0 0x7fc41415f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:29.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.757+0000 7fc412a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc4141589f0 0x7fc41415f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40804/0 (socket says 172.21.15.6:40804) 2024-03-22T13:06:29.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.757+0000 7fc412a00640 1 -- 172.21.15.6:0/1641648504 learned_addr learned my addr 172.21.15.6:0/1641648504 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:29.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.758+0000 7fc413400640 1 --2- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc414071dc0 0x7fc41406b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:29.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.760+0000 7fc412a00640 1 -- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc414071dc0 msgr2=0x7fc41406b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:29.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.760+0000 7fc412a00640 1 --2- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc414071dc0 0x7fc41406b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:29.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.760+0000 7fc412a00640 1 -- 172.21.15.6:0/1641648504 >> v1:172.21.15.6:6789/0 conn(0x7fc4140c7640 legacy=0x7fc414162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:29.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.760+0000 7fc412a00640 1 -- 172.21.15.6:0/1641648504 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4140c6b10 con 0x7fc4141589f0 2024-03-22T13:06:29.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.762+0000 7fc412a00640 1 --2- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4141589f0 0x7fc41415f0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc404009870 tx=0x7fc40402ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebf71e8b2a13b95b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.767+0000 7fc411600640 1 -- 172.21.15.6:0/1641648504 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40402f840 con 0x7fc4141589f0 2024-03-22T13:06:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.767+0000 7fc411600640 1 -- 172.21.15.6:0/1641648504 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40402f9e0 con 0x7fc4141589f0 2024-03-22T13:06:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.767+0000 7fc411600640 1 -- 172.21.15.6:0/1641648504 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40402fdb0 con 0x7fc4141589f0 2024-03-22T13:06:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 -- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4141589f0 msgr2=0x7fc41415f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 --2- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4141589f0 0x7fc41415f0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc404009870 tx=0x7fc40402ba20 comp rx=0 tx=0).stop 2024-03-22T13:06:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 -- 172.21.15.6:0/1641648504 shutdown_connections 2024-03-22T13:06:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 --2- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc414071dc0 0x7fc41406b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 --2- 172.21.15.6:0/1641648504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4141589f0 0x7fc41415f0c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 -- 172.21.15.6:0/1641648504 >> 172.21.15.6:0/1641648504 conn(0x7fc4140cb930 msgr2=0x7fc4140cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 -- 172.21.15.6:0/1641648504 shutdown_connections 2024-03-22T13:06:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.768+0000 7fc419800640 1 -- 172.21.15.6:0/1641648504 wait complete. 2024-03-22T13:06:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.770+0000 7fc419800640 1 Processor -- start 2024-03-22T13:06:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.770+0000 7fc419800640 1 -- start start 2024-03-22T13:06:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.771+0000 7fc419800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc414071dc0 0x7fc41415b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.771+0000 7fc419800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4140c7640 0x7fc4140caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.771+0000 7fc419800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4141589f0 0x7fc4140cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.771+0000 7fc419800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc414161e60 con 0x7fc4140c7640 2024-03-22T13:06:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.771+0000 7fc419800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140c70d0 con 0x7fc414071dc0 2024-03-22T13:06:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.771+0000 7fc419800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc414162a90 con 0x7fc4141589f0 2024-03-22T13:06:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.772+0000 7fc412a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc414071dc0 0x7fc41415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.772+0000 7fc413400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4141589f0 0x7fc4140cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.772+0000 7fc412a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc414071dc0 0x7fc41415b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40816/0 (socket says 172.21.15.6:40816) 2024-03-22T13:06:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.772+0000 7fc412a00640 1 -- 172.21.15.6:0/1136230018 learned_addr learned my addr 172.21.15.6:0/1136230018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.775+0000 7fc412000640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4140c7640 0x7fc4140caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.777+0000 7fc412a00640 1 -- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4141589f0 msgr2=0x7fc4140cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.777+0000 7fc412a00640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4141589f0 0x7fc4140cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.777+0000 7fc412a00640 1 -- 172.21.15.6:0/1136230018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4140c7640 msgr2=0x7fc4140caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.777+0000 7fc412a00640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4140c7640 0x7fc4140caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.777+0000 7fc412a00640 1 -- 172.21.15.6:0/1136230018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4141f7f50 con 0x7fc414071dc0 2024-03-22T13:06:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.781+0000 7fc412a00640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc414071dc0 0x7fc41415b470 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc404009990 tx=0x7fc404004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.785+0000 7fc40aa00640 1 -- 172.21.15.6:0/1136230018 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404047070 con 0x7fc414071dc0 2024-03-22T13:06:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.785+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4141f8230 con 0x7fc414071dc0 2024-03-22T13:06:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.785+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4141f87f0 con 0x7fc414071dc0 2024-03-22T13:06:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:29.786+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4140756c0 con 0x7fc414071dc0 2024-03-22T13:06:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.112+0000 7fc40aa00640 1 -- 172.21.15.6:0/1136230018 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40402f940 con 0x7fc414071dc0 2024-03-22T13:06:30.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.112+0000 7fc40aa00640 1 -- 172.21.15.6:0/1136230018 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404051060 con 0x7fc414071dc0 2024-03-22T13:06:30.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.115+0000 7fc40aa00640 1 -- 172.21.15.6:0/1136230018 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc40404a070 con 0x7fc414071dc0 2024-03-22T13:06:30.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.117+0000 7fc40aa00640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc3f406bbd0 0x7fc3f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:30.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.117+0000 7fc412000640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc3f406bbd0 0x7fc3f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:30.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.119+0000 7fc412000640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc3f406bbd0 0x7fc3f406e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc4141f23d0 tx=0x7fc400009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:30.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.119+0000 7fc40aa00640 1 -- 172.21.15.6:0/1136230018 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc404079ae0 con 0x7fc414071dc0 2024-03-22T13:06:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.137+0000 7fc40aa00640 1 -- 172.21.15.6:0/1136230018 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc404033030 con 0x7fc414071dc0 2024-03-22T13:06:30.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.389+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc41406c0d0 con 0x7fc414071dc0 2024-03-22T13:06:30.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.403+0000 7fc40aa00640 1 -- 172.21.15.6:0/1136230018 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fc40407ba40 con 0x7fc414071dc0 2024-03-22T13:06:30.404 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:30.404 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:30.405 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc3f406bbd0 msgr2=0x7fc3f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc3f406bbd0 0x7fc3f406e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc4141f23d0 tx=0x7fc400009040 comp rx=0 tx=0).stop 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc414071dc0 msgr2=0x7fc41415b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc414071dc0 0x7fc41415b470 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc404009990 tx=0x7fc404004730 comp rx=0 tx=0).stop 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 shutdown_connections 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4141589f0 0x7fc4140cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4140c7640 0x7fc4140caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc3f406bbd0 0x7fc3f406e090 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 --2- 172.21.15.6:0/1136230018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc414071dc0 0x7fc41415b470 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 >> 172.21.15.6:0/1136230018 conn(0x7fc4140cb930 msgr2=0x7fc414155b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 shutdown_connections 2024-03-22T13:06:30.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:30.406+0000 7fc419800640 1 -- 172.21.15.6:0/1136230018 wait complete. 2024-03-22T13:06:31.028 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:31.028 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:32.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:32.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:06:32.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:06:32.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.215+0000 7f08ffe00640 1 Processor -- start 2024-03-22T13:06:32.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.216+0000 7f08ffe00640 1 -- start start 2024-03-22T13:06:32.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.217+0000 7f08ffe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f08f806f8c0 0x7f08f80716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:32.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.217+0000 7f08ffe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f81590b0 con 0x7f08f8152400 2024-03-22T13:06:32.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.217+0000 7f08ffe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80721f0 con 0x7f08f8071c10 2024-03-22T13:06:32.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.217+0000 7f08ffe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f80724d0 con 0x7f08f806f8c0 2024-03-22T13:06:32.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.218+0000 7f08fde00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f08f8071c10 0x7f08f80c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43308/0 (socket says 172.21.15.6:43308) 2024-03-22T13:06:32.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.218+0000 7f08fde00640 1 -- 172.21.15.6:0/1250278255 learned_addr learned my addr 172.21.15.6:0/1250278255 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:32.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.218+0000 7f08fca00640 1 --2- 172.21.15.6:0/1250278255 >> v2:172.21.15.6:3300/0 conn(0x7f08f806f8c0 0x7f08f80716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:32.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.221+0000 7f08f7e00640 1 -- 172.21.15.6:0/1250278255 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482894334 0 0) 0x7f08f80721f0 con 0x7f08f8071c10 2024-03-22T13:06:32.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.221+0000 7f08f7e00640 1 -- 172.21.15.6:0/1250278255 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e0003520 con 0x7f08f8071c10 2024-03-22T13:06:32.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.224+0000 7f08fca00640 1 -- 172.21.15.6:0/1250278255 >> v1:172.21.15.92:6789/0 conn(0x7f08f8071c10 legacy=0x7f08f80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:32.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.224+0000 7f08fca00640 1 -- 172.21.15.6:0/1250278255 >> v1:172.21.15.6:6789/0 conn(0x7f08f8152400 legacy=0x7f08f815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:32.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.224+0000 7f08fca00640 1 -- 172.21.15.6:0/1250278255 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f81582f0 con 0x7f08f806f8c0 2024-03-22T13:06:32.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.229+0000 7f08fca00640 1 --2- 172.21.15.6:0/1250278255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 0x7f08f80716d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f08e40097a0 tx=0x7f08e402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6520d328878c732f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.235+0000 7f08f7e00640 1 -- 172.21.15.6:0/1250278255 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e402f810 con 0x7f08f806f8c0 2024-03-22T13:06:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.235+0000 7f08f7e00640 1 -- 172.21.15.6:0/1250278255 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e402f9b0 con 0x7f08f806f8c0 2024-03-22T13:06:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.235+0000 7f08f7e00640 1 -- 172.21.15.6:0/1250278255 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e402fd80 con 0x7f08f806f8c0 2024-03-22T13:06:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.236+0000 7f08ffe00640 1 -- 172.21.15.6:0/1250278255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 msgr2=0x7f08f80716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.236+0000 7f08ffe00640 1 --2- 172.21.15.6:0/1250278255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 0x7f08f80716d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f08e40097a0 tx=0x7f08e402b9f0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.236+0000 7f08ffe00640 1 -- 172.21.15.6:0/1250278255 shutdown_connections 2024-03-22T13:06:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.236+0000 7f08ffe00640 1 --2- 172.21.15.6:0/1250278255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 0x7f08f80716d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.236+0000 7f08ffe00640 1 -- 172.21.15.6:0/1250278255 >> 172.21.15.6:0/1250278255 conn(0x7f08f80cb930 msgr2=0x7f08f80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.236+0000 7f08ffe00640 1 -- 172.21.15.6:0/1250278255 shutdown_connections 2024-03-22T13:06:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.236+0000 7f08ffe00640 1 -- 172.21.15.6:0/1250278255 wait complete. 2024-03-22T13:06:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.238+0000 7f08ffe00640 1 Processor -- start 2024-03-22T13:06:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.238+0000 7f08ffe00640 1 -- start start 2024-03-22T13:06:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.239+0000 7f08ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 0x7f08f81efef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.239+0000 7f08ffe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08f8071c10 0x7f08f81457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.239+0000 7f08ffe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08f8152400 0x7f08f8156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.239+0000 7f08ffe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f8158b90 con 0x7f08f806f8c0 2024-03-22T13:06:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.239+0000 7f08ffe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f81588b0 con 0x7f08f8071c10 2024-03-22T13:06:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.239+0000 7f08ffe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f81527e0 con 0x7f08f8152400 2024-03-22T13:06:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.240+0000 7f08fca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08f8071c10 0x7f08f81457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.240+0000 7f08fd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 0x7f08f81efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.240+0000 7f08fde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08f8152400 0x7f08f8156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.240+0000 7f08fde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08f8152400 0x7f08f8156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35736/0 (socket says 172.21.15.6:35736) 2024-03-22T13:06:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.240+0000 7f08fde00640 1 -- 172.21.15.6:0/3083578111 learned_addr learned my addr 172.21.15.6:0/3083578111 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.245+0000 7f08fca00640 1 -- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08f8152400 msgr2=0x7f08f8156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.245+0000 7f08fca00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08f8152400 0x7f08f8156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.245+0000 7f08fca00640 1 -- 172.21.15.6:0/3083578111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 msgr2=0x7f08f81efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.245+0000 7f08fca00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 0x7f08f81efef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.245+0000 7f08fca00640 1 -- 172.21.15.6:0/3083578111 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f81f3880 con 0x7f08f8071c10 2024-03-22T13:06:32.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.248+0000 7f08fca00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08f8071c10 0x7f08f81457f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f08f80cc9a0 tx=0x7f08e40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:32.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.251+0000 7f08f5600640 1 -- 172.21.15.6:0/3083578111 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e40378d0 con 0x7f08f8071c10 2024-03-22T13:06:32.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.251+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f81f3b40 con 0x7f08f8071c10 2024-03-22T13:06:32.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.251+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f81f4100 con 0x7f08f8071c10 2024-03-22T13:06:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.252+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f81f4340 con 0x7f08f8071c10 2024-03-22T13:06:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.253+0000 7f08f5600640 1 -- 172.21.15.6:0/3083578111 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e402f9b0 con 0x7f08f8071c10 2024-03-22T13:06:32.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.253+0000 7f08f5600640 1 -- 172.21.15.6:0/3083578111 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4037c90 con 0x7f08f8071c10 2024-03-22T13:06:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.526+0000 7f08f5600640 1 -- 172.21.15.6:0/3083578111 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f08e4049060 con 0x7f08f8071c10 2024-03-22T13:06:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.528+0000 7f08f5600640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08d806bbd0 0x7f08d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.528+0000 7f08f5600640 1 -- 172.21.15.6:0/3083578111 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f08e40ad060 con 0x7f08f8071c10 2024-03-22T13:06:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.529+0000 7f08fd400640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08d806bbd0 0x7f08d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.530+0000 7f08fd400640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08d806bbd0 0x7f08d806e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0016e0 tx=0x7f08ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:32.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.539+0000 7f08f5600640 1 -- 172.21.15.6:0/3083578111 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f08f81f4340 con 0x7f08f8071c10 2024-03-22T13:06:32.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.786+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08f80756c0 con 0x7f08f8071c10 2024-03-22T13:06:32.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.809+0000 7f08f5600640 1 -- 172.21.15.6:0/3083578111 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f08e4033030 con 0x7f08f8071c10 2024-03-22T13:06:32.810 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:32.810 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:32.810 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.811+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08d806bbd0 msgr2=0x7f08d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.811+0000 7f08ffe00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08d806bbd0 0x7f08d806e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0016e0 tx=0x7f08ec009040 comp rx=0 tx=0).stop 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.811+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08f8071c10 msgr2=0x7f08f81457f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.811+0000 7f08ffe00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08f8071c10 0x7f08f81457f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f08f80cc9a0 tx=0x7f08e40046c0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 shutdown_connections 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08f8152400 0x7f08f8156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08d806bbd0 0x7f08d806e090 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08f8071c10 0x7f08f81457f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 --2- 172.21.15.6:0/3083578111 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08f806f8c0 0x7f08f81efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 >> 172.21.15.6:0/3083578111 conn(0x7f08f80cb930 msgr2=0x7f08f8146440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 shutdown_connections 2024-03-22T13:06:32.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:32.812+0000 7f08ffe00640 1 -- 172.21.15.6:0/3083578111 wait complete. 2024-03-22T13:06:33.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:33.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:34.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:34.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.635+0000 7f9b2f200640 1 Processor -- start 2024-03-22T13:06:34.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.635+0000 7f9b2f200640 1 -- start start 2024-03-22T13:06:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.637+0000 7f9b2f200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9b28154980 0x7f9b280ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.637+0000 7f9b2f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b280cc380 con 0x7f9b280cd300 2024-03-22T13:06:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.637+0000 7f9b2f200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b280cc660 con 0x7f9b2815b0c0 2024-03-22T13:06:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.637+0000 7f9b2f200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2815a520 con 0x7f9b28154980 2024-03-22T13:06:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.638+0000 7f9b27400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9b280cd300 0x7f9b2815a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59298/0 (socket says 172.21.15.6:59298) 2024-03-22T13:06:34.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.638+0000 7f9b27400640 1 -- 172.21.15.6:0/2493312145 learned_addr learned my addr 172.21.15.6:0/2493312145 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:34.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.638+0000 7f9b27e00640 1 --2- 172.21.15.6:0/2493312145 >> v2:172.21.15.92:3300/0 conn(0x7f9b28154980 0x7f9b280ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:34.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.641+0000 7f9b26a00640 1 -- 172.21.15.6:0/2493312145 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445935903 0 0) 0x7f9b280cc380 con 0x7f9b280cd300 2024-03-22T13:06:34.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.641+0000 7f9b26a00640 1 -- 172.21.15.6:0/2493312145 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b0c003520 con 0x7f9b280cd300 2024-03-22T13:06:34.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.641+0000 7f9b27e00640 1 -- 172.21.15.6:0/2493312145 >> v1:172.21.15.92:6789/0 conn(0x7f9b2815b0c0 legacy=0x7f9b280ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:34.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.641+0000 7f9b27e00640 1 -- 172.21.15.6:0/2493312145 >> v1:172.21.15.6:6789/0 conn(0x7f9b280cd300 legacy=0x7f9b2815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:34.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.641+0000 7f9b27e00640 1 -- 172.21.15.6:0/2493312145 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b280cc940 con 0x7f9b28154980 2024-03-22T13:06:34.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.645+0000 7f9b27e00640 1 --2- 172.21.15.6:0/2493312145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280ce300 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9b180097a0 tx=0x7f9b1802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=466dc6a9086536b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:34.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.650+0000 7f9b26a00640 1 -- 172.21.15.6:0/2493312145 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18035020 con 0x7f9b28154980 2024-03-22T13:06:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.650+0000 7f9b26a00640 1 -- 172.21.15.6:0/2493312145 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b18031830 con 0x7f9b28154980 2024-03-22T13:06:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.650+0000 7f9b26a00640 1 -- 172.21.15.6:0/2493312145 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18031c00 con 0x7f9b28154980 2024-03-22T13:06:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.650+0000 7f9b2f200640 1 -- 172.21.15.6:0/2493312145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 msgr2=0x7f9b280ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.650+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2493312145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280ce300 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9b180097a0 tx=0x7f9b1802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:06:34.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.651+0000 7f9b2f200640 1 -- 172.21.15.6:0/2493312145 shutdown_connections 2024-03-22T13:06:34.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.651+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2493312145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280ce300 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:34.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.651+0000 7f9b2f200640 1 -- 172.21.15.6:0/2493312145 >> 172.21.15.6:0/2493312145 conn(0x7f9b280b16c0 msgr2=0x7f9b280b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:34.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.651+0000 7f9b2f200640 1 -- 172.21.15.6:0/2493312145 shutdown_connections 2024-03-22T13:06:34.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.651+0000 7f9b2f200640 1 -- 172.21.15.6:0/2493312145 wait complete. 2024-03-22T13:06:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.653+0000 7f9b2f200640 1 Processor -- start 2024-03-22T13:06:34.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.654+0000 7f9b2f200640 1 -- start start 2024-03-22T13:06:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.655+0000 7f9b2f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b280cd300 0x7f9b280b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.655+0000 7f9b2f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.655+0000 7f9b2f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9b2815b0c0 0x7f9b280c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.655+0000 7f9b2f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280cf5b0 con 0x7f9b280cd300 2024-03-22T13:06:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.655+0000 7f9b2f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280ccf00 con 0x7f9b28154980 2024-03-22T13:06:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.655+0000 7f9b2f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280cf730 con 0x7f9b2815b0c0 2024-03-22T13:06:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.656+0000 7f9b27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.656+0000 7f9b2d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9b2815b0c0 0x7f9b280c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.656+0000 7f9b27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b280cd300 0x7f9b280b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.656+0000 7f9b27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:59552/0 (socket says 172.21.15.6:59552) 2024-03-22T13:06:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.656+0000 7f9b27400640 1 -- 172.21.15.6:0/2982207200 learned_addr learned my addr 172.21.15.6:0/2982207200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:34.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.660+0000 7f9b27400640 1 -- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9b2815b0c0 msgr2=0x7f9b280c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:34.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.660+0000 7f9b27400640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9b2815b0c0 0x7f9b280c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:34.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.660+0000 7f9b27400640 1 -- 172.21.15.6:0/2982207200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b280cd300 msgr2=0x7f9b280b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:34.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.660+0000 7f9b27400640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b280cd300 0x7f9b280b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:34.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.660+0000 7f9b27400640 1 -- 172.21.15.6:0/2982207200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b280c6610 con 0x7f9b28154980 2024-03-22T13:06:34.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.663+0000 7f9b27400640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280b48e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400c400 tx=0x7f9b1400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.668+0000 7f9b1fe00640 1 -- 172.21.15.6:0/2982207200 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b14017070 con 0x7f9b28154980 2024-03-22T13:06:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.668+0000 7f9b1fe00640 1 -- 172.21.15.6:0/2982207200 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b14005d00 con 0x7f9b28154980 2024-03-22T13:06:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.668+0000 7f9b1fe00640 1 -- 172.21.15.6:0/2982207200 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b14007310 con 0x7f9b28154980 2024-03-22T13:06:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.668+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b280b3a10 con 0x7f9b28154980 2024-03-22T13:06:34.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.668+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b281f0ac0 con 0x7f9b28154980 2024-03-22T13:06:34.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.670+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b280756c0 con 0x7f9b28154980 2024-03-22T13:06:34.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.701+0000 7f9b1fe00640 1 -- 172.21.15.6:0/2982207200 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9b1401a070 con 0x7f9b28154980 2024-03-22T13:06:34.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.703+0000 7f9b1fe00640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b0806bc50 0x7f9b0806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:34.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.704+0000 7f9b27e00640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b0806bc50 0x7f9b0806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:34.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.704+0000 7f9b27e00640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b0806bc50 0x7f9b0806e110 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9b18036000 tx=0x7f9b1803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:34.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.705+0000 7f9b1fe00640 1 -- 172.21.15.6:0/2982207200 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9b140567c0 con 0x7f9b28154980 2024-03-22T13:06:34.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:34.966+0000 7f9b1fe00640 1 -- 172.21.15.6:0/2982207200 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9b14021020 con 0x7f9b28154980 2024-03-22T13:06:35.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.223+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b280b5b30 con 0x7f9b28154980 2024-03-22T13:06:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.237+0000 7f9b1fe00640 1 -- 172.21.15.6:0/2982207200 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f9b14058720 con 0x7f9b28154980 2024-03-22T13:06:35.238 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:35.238 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:35.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.240+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b0806bc50 msgr2=0x7f9b0806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:35.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.240+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b0806bc50 0x7f9b0806e110 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9b18036000 tx=0x7f9b1803c040 comp rx=0 tx=0).stop 2024-03-22T13:06:35.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.240+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 msgr2=0x7f9b280b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:35.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.240+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280b48e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400c400 tx=0x7f9b1400c8c0 comp rx=0 tx=0).stop 2024-03-22T13:06:35.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 shutdown_connections 2024-03-22T13:06:35.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9b2815b0c0 0x7f9b280c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9b0806bc50 0x7f9b0806e110 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b28154980 0x7f9b280b48e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 --2- 172.21.15.6:0/2982207200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b280cd300 0x7f9b280b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 >> 172.21.15.6:0/2982207200 conn(0x7f9b280b16c0 msgr2=0x7f9b281530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 shutdown_connections 2024-03-22T13:06:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:35.241+0000 7f9b2f200640 1 -- 172.21.15.6:0/2982207200 wait complete. 2024-03-22T13:06:35.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:35.863 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:36.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:37.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.076+0000 7f6046200640 1 Processor -- start 2024-03-22T13:06:37.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.077+0000 7f6046200640 1 -- start start 2024-03-22T13:06:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.078+0000 7f6046200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f604006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.078+0000 7f6046200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60400ce8a0 0x7f604015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.078+0000 7f6046200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60400cee00 con 0x7f60400cc760 2024-03-22T13:06:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.078+0000 7f6046200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60400cf0c0 con 0x7f60400ce8a0 2024-03-22T13:06:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.078+0000 7f6046200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60400cf3a0 con 0x7f60400cc380 2024-03-22T13:06:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.078+0000 7f6044e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f604006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.079+0000 7f6044e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f604006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60414/0 (socket says 172.21.15.6:60414) 2024-03-22T13:06:37.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.079+0000 7f6044e00640 1 -- 172.21.15.6:0/3216221118 learned_addr learned my addr 172.21.15.6:0/3216221118 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:37.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.079+0000 7f603fe00640 1 --2- 172.21.15.6:0/3216221118 >> v2:172.21.15.6:3300/0 conn(0x7f60400ce8a0 0x7f604015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.082+0000 7f6044e00640 1 -- 172.21.15.6:0/3216221118 >> v2:172.21.15.6:3300/0 conn(0x7f60400ce8a0 msgr2=0x7f604015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.082+0000 7f6044e00640 1 --2- 172.21.15.6:0/3216221118 >> v2:172.21.15.6:3300/0 conn(0x7f60400ce8a0 0x7f604015ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.082+0000 7f6044e00640 1 -- 172.21.15.6:0/3216221118 >> v1:172.21.15.92:6789/0 conn(0x7f60400cc760 legacy=0x7f60400cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.082+0000 7f6044e00640 1 -- 172.21.15.6:0/3216221118 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400cf680 con 0x7f60400cc380 2024-03-22T13:06:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.084+0000 7f6044e00640 1 --2- 172.21.15.6:0/3216221118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f604006c230 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6034009870 tx=0x7f603402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50444838a4b322c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.088+0000 7f603f400640 1 -- 172.21.15.6:0/3216221118 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034034070 con 0x7f60400cc380 2024-03-22T13:06:37.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.089+0000 7f603f400640 1 -- 172.21.15.6:0/3216221118 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034030840 con 0x7f60400cc380 2024-03-22T13:06:37.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.089+0000 7f603f400640 1 -- 172.21.15.6:0/3216221118 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034030c10 con 0x7f60400cc380 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 -- 172.21.15.6:0/3216221118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 msgr2=0x7f604006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 --2- 172.21.15.6:0/3216221118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f604006c230 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6034009870 tx=0x7f603402b980 comp rx=0 tx=0).stop 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 -- 172.21.15.6:0/3216221118 shutdown_connections 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 --2- 172.21.15.6:0/3216221118 >> v2:172.21.15.6:3300/0 conn(0x7f60400ce8a0 0x7f604015ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 --2- 172.21.15.6:0/3216221118 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f604006c230 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 -- 172.21.15.6:0/3216221118 >> 172.21.15.6:0/3216221118 conn(0x7f60400b1480 msgr2=0x7f60400b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 -- 172.21.15.6:0/3216221118 shutdown_connections 2024-03-22T13:06:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.090+0000 7f6046200640 1 -- 172.21.15.6:0/3216221118 wait complete. 2024-03-22T13:06:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.092+0000 7f6046200640 1 Processor -- start 2024-03-22T13:06:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.092+0000 7f6046200640 1 -- start start 2024-03-22T13:06:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.093+0000 7f6046200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f60401f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.093+0000 7f6046200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60400cc760 0x7f6040151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.093+0000 7f6046200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60400ce8a0 0x7f60400c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.093+0000 7f6046200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60401549d0 con 0x7f60400ce8a0 2024-03-22T13:06:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.093+0000 7f6046200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604015a910 con 0x7f60400cc760 2024-03-22T13:06:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.093+0000 7f6046200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60400ccb90 con 0x7f60400cc380 2024-03-22T13:06:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.093+0000 7f6044e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f60401f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.094+0000 7f603fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60400cc760 0x7f6040151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.094+0000 7f6044e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f60401f0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60416/0 (socket says 172.21.15.6:60416) 2024-03-22T13:06:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.094+0000 7f6044e00640 1 -- 172.21.15.6:0/154656195 learned_addr learned my addr 172.21.15.6:0/154656195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.094+0000 7f6045800640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60400ce8a0 0x7f60400c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.097+0000 7f6044e00640 1 -- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60400cc760 msgr2=0x7f6040151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.097+0000 7f6044e00640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60400cc760 0x7f6040151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.097+0000 7f6044e00640 1 -- 172.21.15.6:0/154656195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60400ce8a0 msgr2=0x7f60400c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.097+0000 7f6044e00640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60400ce8a0 0x7f60400c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.097+0000 7f6044e00640 1 -- 172.21.15.6:0/154656195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604015b760 con 0x7f60400cc380 2024-03-22T13:06:37.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.100+0000 7f6044e00640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f60401f0e40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f603402b9c0 tx=0x7f60340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.105+0000 7f603cc00640 1 -- 172.21.15.6:0/154656195 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034034040 con 0x7f60400cc380 2024-03-22T13:06:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.105+0000 7f603cc00640 1 -- 172.21.15.6:0/154656195 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034039d90 con 0x7f60400cc380 2024-03-22T13:06:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.105+0000 7f603cc00640 1 -- 172.21.15.6:0/154656195 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034043630 con 0x7f60400cc380 2024-03-22T13:06:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.105+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604015ba40 con 0x7f60400cc380 2024-03-22T13:06:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.105+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60400c6df0 con 0x7f60400cc380 2024-03-22T13:06:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.106+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6040075650 con 0x7f60400cc380 2024-03-22T13:06:37.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.391+0000 7f603cc00640 1 -- 172.21.15.6:0/154656195 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f603404a060 con 0x7f60400cc380 2024-03-22T13:06:37.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.393+0000 7f603cc00640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f602006bc50 0x7f602006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:37.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.393+0000 7f603fe00640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f602006bc50 0x7f602006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:37.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.394+0000 7f603fe00640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f602006bc50 0x7f602006e110 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f60401edff0 tx=0x7f602c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:37.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.395+0000 7f603cc00640 1 -- 172.21.15.6:0/154656195 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6034079430 con 0x7f60400cc380 2024-03-22T13:06:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.414+0000 7f603cc00640 1 -- 172.21.15.6:0/154656195 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f603403e070 con 0x7f60400cc380 2024-03-22T13:06:37.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.668+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60400c9990 con 0x7f60400cc380 2024-03-22T13:06:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.683+0000 7f603cc00640 1 -- 172.21.15.6:0/154656195 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f603407b390 con 0x7f60400cc380 2024-03-22T13:06:37.683 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:37.684 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f602006bc50 msgr2=0x7f602006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f602006bc50 0x7f602006e110 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f60401edff0 tx=0x7f602c009190 comp rx=0 tx=0).stop 2024-03-22T13:06:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 msgr2=0x7f60401f0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f60401f0e40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f603402b9c0 tx=0x7f60340046c0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 shutdown_connections 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60400ce8a0 0x7f60400c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f602006bc50 0x7f602006e110 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60400cc760 0x7f6040151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 --2- 172.21.15.6:0/154656195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60400cc380 0x7f60401f0e40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 >> 172.21.15.6:0/154656195 conn(0x7f60400b1480 msgr2=0x7f60401592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 shutdown_connections 2024-03-22T13:06:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:37.685+0000 7f6046200640 1 -- 172.21.15.6:0/154656195 wait complete. 2024-03-22T13:06:38.316 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:38.316 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:39.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.521+0000 7fa902800640 1 Processor -- start 2024-03-22T13:06:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.521+0000 7fa902800640 1 -- start start 2024-03-22T13:06:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.522+0000 7fa902800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c5210 0x7fa8fc158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.522+0000 7fa902800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa8fc0c55f0 0x7fa8fc0c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.523+0000 7fa902800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8fc06e0f0 con 0x7fa8fc0c67d0 2024-03-22T13:06:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.523+0000 7fa902800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8fc0c6f90 con 0x7fa8fc0c55f0 2024-03-22T13:06:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.523+0000 7fa902800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc06d350 con 0x7fa8fc0c5210 2024-03-22T13:06:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.523+0000 7fa901e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8fc0c67d0 0x7fa8fc06ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59300/0 (socket says 172.21.15.6:59300) 2024-03-22T13:06:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.523+0000 7fa901e00640 1 -- 172.21.15.6:0/157036795 learned_addr learned my addr 172.21.15.6:0/157036795 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.523+0000 7fa901400640 1 --2- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c5210 0x7fa8fc158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:39.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.523+0000 7fa900a00640 1 --2- 172.21.15.6:0/157036795 >> v2:172.21.15.92:3300/0 conn(0x7fa8fc0c55f0 0x7fa8fc0c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.526+0000 7fa900a00640 1 -- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c5210 msgr2=0x7fa8fc158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.526+0000 7fa900a00640 1 --2- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c5210 0x7fa8fc158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.526+0000 7fa900a00640 1 -- 172.21.15.6:0/157036795 >> v1:172.21.15.6:6789/0 conn(0x7fa8fc0c67d0 legacy=0x7fa8fc06ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.526+0000 7fa900a00640 1 -- 172.21.15.6:0/157036795 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0c6cb0 con 0x7fa8fc0c55f0 2024-03-22T13:06:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.528+0000 7fa900a00640 1 --2- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 0x7fa8fc0c6290 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80097a0 tx=0x7fa8e802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58ecb45ad4b99f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:39.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.532+0000 7fa8fbe00640 1 -- 172.21.15.6:0/157036795 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8035020 con 0x7fa8fc0c55f0 2024-03-22T13:06:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.532+0000 7fa8fbe00640 1 -- 172.21.15.6:0/157036795 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8031830 con 0x7fa8fc0c55f0 2024-03-22T13:06:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.532+0000 7fa8fbe00640 1 -- 172.21.15.6:0/157036795 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8031c00 con 0x7fa8fc0c55f0 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.532+0000 7fa902800640 1 -- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 msgr2=0x7fa8fc0c6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.532+0000 7fa902800640 1 --2- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 0x7fa8fc0c6290 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80097a0 tx=0x7fa8e802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.533+0000 7fa902800640 1 -- 172.21.15.6:0/157036795 shutdown_connections 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.533+0000 7fa902800640 1 --2- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 0x7fa8fc0c6290 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.533+0000 7fa902800640 1 --2- 172.21.15.6:0/157036795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c5210 0x7fa8fc158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.533+0000 7fa902800640 1 -- 172.21.15.6:0/157036795 >> 172.21.15.6:0/157036795 conn(0x7fa8fc0cb890 msgr2=0x7fa8fc0cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.533+0000 7fa902800640 1 -- 172.21.15.6:0/157036795 shutdown_connections 2024-03-22T13:06:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.533+0000 7fa902800640 1 -- 172.21.15.6:0/157036795 wait complete. 2024-03-22T13:06:39.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.534+0000 7fa902800640 1 Processor -- start 2024-03-22T13:06:39.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.535+0000 7fa902800640 1 -- start start 2024-03-22T13:06:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.536+0000 7fa902800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8fc0c5210 0x7fa8fc1426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.536+0000 7fa902800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 0x7fa8fc14cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.536+0000 7fa902800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c67d0 0x7fa8fc1ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.536+0000 7fa902800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc06dbd0 con 0x7fa8fc0c5210 2024-03-22T13:06:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.536+0000 7fa902800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc06d8f0 con 0x7fa8fc0c55f0 2024-03-22T13:06:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.536+0000 7fa902800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc06f4f0 con 0x7fa8fc0c67d0 2024-03-22T13:06:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.537+0000 7fa900a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 0x7fa8fc14cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:39.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.537+0000 7fa901400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8fc0c5210 0x7fa8fc1426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:39.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.538+0000 7fa901400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8fc0c5210 0x7fa8fc1426f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51370/0 (socket says 172.21.15.6:51370) 2024-03-22T13:06:39.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.538+0000 7fa901400640 1 -- 172.21.15.6:0/1082279957 learned_addr learned my addr 172.21.15.6:0/1082279957 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.539+0000 7fa901e00640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c67d0 0x7fa8fc1ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:39.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.542+0000 7fa901400640 1 -- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c67d0 msgr2=0x7fa8fc1ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:39.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.542+0000 7fa901400640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c67d0 0x7fa8fc1ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:39.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.542+0000 7fa901400640 1 -- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 msgr2=0x7fa8fc14cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:39.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.542+0000 7fa901400640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 0x7fa8fc14cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:39.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.542+0000 7fa901400640 1 -- 172.21.15.6:0/1082279957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc142ed0 con 0x7fa8fc0c5210 2024-03-22T13:06:39.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.545+0000 7fa901400640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8fc0c5210 0x7fa8fc1426f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000e9b0 tx=0x7fa8f000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:39.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.551+0000 7fa8f9600640 1 -- 172.21.15.6:0/1082279957 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0017070 con 0x7fa8fc0c5210 2024-03-22T13:06:39.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.551+0000 7fa8f9600640 1 -- 172.21.15.6:0/1082279957 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f000c9a0 con 0x7fa8fc0c5210 2024-03-22T13:06:39.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.551+0000 7fa8f9600640 1 -- 172.21.15.6:0/1082279957 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f000cd90 con 0x7fa8fc0c5210 2024-03-22T13:06:39.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.551+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc14dae0 con 0x7fa8fc0c5210 2024-03-22T13:06:39.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.551+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8fc143150 con 0x7fa8fc0c5210 2024-03-22T13:06:39.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.552+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc0c79f0 con 0x7fa8fc0c5210 2024-03-22T13:06:39.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.577+0000 7fa8f9600640 1 -- 172.21.15.6:0/1082279957 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa8f001a070 con 0x7fa8fc0c5210 2024-03-22T13:06:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.577+0000 7fa8f9600640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa8dc06bbd0 0x7fa8dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:39.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.578+0000 7fa900a00640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa8dc06bbd0 0x7fa8dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.579+0000 7fa900a00640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa8dc06bbd0 0x7fa8dc06e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8036000 tx=0x7fa8e8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:39.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.879+0000 7fa8f9600640 1 -- 172.21.15.6:0/1082279957 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa8f00577a0 con 0x7fa8fc0c5210 2024-03-22T13:06:39.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:39.879+0000 7fa8f9600640 1 -- 172.21.15.6:0/1082279957 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa8fc0c79f0 con 0x7fa8fc0c5210 2024-03-22T13:06:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.176+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8fc075650 con 0x7fa8fc0c5210 2024-03-22T13:06:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.199+0000 7fa8f9600640 1 -- 172.21.15.6:0/1082279957 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fa8fc142ed0 con 0x7fa8fc0c5210 2024-03-22T13:06:40.199 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:40.199 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa8dc06bbd0 msgr2=0x7fa8dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa8dc06bbd0 0x7fa8dc06e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8036000 tx=0x7fa8e8008030 comp rx=0 tx=0).stop 2024-03-22T13:06:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8fc0c5210 msgr2=0x7fa8fc1426f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8fc0c5210 0x7fa8fc1426f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000e9b0 tx=0x7fa8f000ee70 comp rx=0 tx=0).stop 2024-03-22T13:06:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 shutdown_connections 2024-03-22T13:06:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8fc0c67d0 0x7fa8fc1ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa8dc06bbd0 0x7fa8dc06e090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8fc0c55f0 0x7fa8fc14cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 --2- 172.21.15.6:0/1082279957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8fc0c5210 0x7fa8fc1426f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 >> 172.21.15.6:0/1082279957 conn(0x7fa8fc0cb890 msgr2=0x7fa8fc0c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 shutdown_connections 2024-03-22T13:06:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:40.202+0000 7fa902800640 1 -- 172.21.15.6:0/1082279957 wait complete. 2024-03-22T13:06:40.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:40.823 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:41.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:42.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.023+0000 7f7c42600640 1 Processor -- start 2024-03-22T13:06:42.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.023+0000 7f7c42600640 1 -- start start 2024-03-22T13:06:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.025+0000 7f7c42600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7c3c152400 0x7f7c3c15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.025+0000 7f7c42600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c3c15bc80 0x7f7c3c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.025+0000 7f7c42600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c06f8c0 0x7f7c3c071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.025+0000 7f7c42600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3c0724d0 con 0x7f7c3c152400 2024-03-22T13:06:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.025+0000 7f7c42600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3c1590b0 con 0x7f7c3c15bc80 2024-03-22T13:06:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.025+0000 7f7c42600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c0721f0 con 0x7f7c3c06f8c0 2024-03-22T13:06:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.026+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c06f8c0 0x7f7c3c071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.026+0000 7f7c3b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7c3c152400 0x7f7c3c15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.026+0000 7f7c32a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c3c15bc80 0x7f7c3c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.026+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c06f8c0 0x7f7c3c071c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60464/0 (socket says 172.21.15.6:60464) 2024-03-22T13:06:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.026+0000 7f7c3be00640 1 -- 172.21.15.6:0/311341482 learned_addr learned my addr 172.21.15.6:0/311341482 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.030+0000 7f7c32a00640 1 -- 172.21.15.6:0/311341482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c06f8c0 msgr2=0x7f7c3c071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.030+0000 7f7c32a00640 1 --2- 172.21.15.6:0/311341482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c06f8c0 0x7f7c3c071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.030+0000 7f7c32a00640 1 -- 172.21.15.6:0/311341482 >> v2:172.21.15.6:3300/0 conn(0x7f7c3c152400 msgr2=0x7f7c3c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:42.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.030+0000 7f7c32a00640 1 --2- 172.21.15.6:0/311341482 >> v2:172.21.15.6:3300/0 conn(0x7f7c3c152400 0x7f7c3c15b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.030+0000 7f7c32a00640 1 -- 172.21.15.6:0/311341482 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c1582f0 con 0x7f7c3c15bc80 2024-03-22T13:06:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.032+0000 7f7c32a00640 1 --2- 172.21.15.6:0/311341482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c15bc80 0x7f7c3c0716d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c15c060 tx=0x7f7c2402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a6f87adb6aca5a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:42.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.035+0000 7f7c3aa00640 1 -- 172.21.15.6:0/311341482 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24035020 con 0x7f7c3c15bc80 2024-03-22T13:06:42.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.037+0000 7f7c3aa00640 1 -- 172.21.15.6:0/311341482 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c24031830 con 0x7f7c3c15bc80 2024-03-22T13:06:42.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.037+0000 7f7c3aa00640 1 -- 172.21.15.6:0/311341482 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24031c00 con 0x7f7c3c15bc80 2024-03-22T13:06:42.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 -- 172.21.15.6:0/311341482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c15bc80 msgr2=0x7f7c3c0716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:42.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 --2- 172.21.15.6:0/311341482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c15bc80 0x7f7c3c0716d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c15c060 tx=0x7f7c2402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 -- 172.21.15.6:0/311341482 shutdown_connections 2024-03-22T13:06:42.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 --2- 172.21.15.6:0/311341482 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c06f8c0 0x7f7c3c071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 --2- 172.21.15.6:0/311341482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c15bc80 0x7f7c3c0716d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 --2- 172.21.15.6:0/311341482 >> v2:172.21.15.6:3300/0 conn(0x7f7c3c152400 0x7f7c3c15b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 -- 172.21.15.6:0/311341482 >> 172.21.15.6:0/311341482 conn(0x7f7c3c0cb930 msgr2=0x7f7c3c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 -- 172.21.15.6:0/311341482 shutdown_connections 2024-03-22T13:06:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.038+0000 7f7c42600640 1 -- 172.21.15.6:0/311341482 wait complete. 2024-03-22T13:06:42.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.040+0000 7f7c42600640 1 Processor -- start 2024-03-22T13:06:42.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.041+0000 7f7c42600640 1 -- start start 2024-03-22T13:06:42.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.041+0000 7f7c42600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c06f8c0 0x7f7c3c14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:42.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c42600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c3c152400 0x7f7c3c157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:42.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c42600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c15bc80 0x7f7c3c1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:42.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c42600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c158b90 con 0x7f7c3c152400 2024-03-22T13:06:42.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c42600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1588b0 con 0x7f7c3c06f8c0 2024-03-22T13:06:42.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c42600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1527e0 con 0x7f7c3c15bc80 2024-03-22T13:06:42.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c3b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c06f8c0 0x7f7c3c14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:42.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c3be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c15bc80 0x7f7c3c1577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.042+0000 7f7c32a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c3c152400 0x7f7c3c157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.043+0000 7f7c3b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c06f8c0 0x7f7c3c14d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:59580/0 (socket says 172.21.15.6:59580) 2024-03-22T13:06:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.043+0000 7f7c3b400640 1 -- 172.21.15.6:0/517145483 learned_addr learned my addr 172.21.15.6:0/517145483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.047+0000 7f7c3be00640 1 -- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c06f8c0 msgr2=0x7f7c3c14d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.047+0000 7f7c3be00640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c06f8c0 0x7f7c3c14d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.047+0000 7f7c3be00640 1 -- 172.21.15.6:0/517145483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c3c152400 msgr2=0x7f7c3c157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.047+0000 7f7c3be00640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c3c152400 0x7f7c3c157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.047+0000 7f7c3be00640 1 -- 172.21.15.6:0/517145483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c1598d0 con 0x7f7c3c15bc80 2024-03-22T13:06:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.049+0000 7f7c3be00640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c15bc80 0x7f7c3c1577c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400e640 tx=0x7f7c3400eb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.053+0000 7f7c33e00640 1 -- 172.21.15.6:0/517145483 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34017070 con 0x7f7c3c15bc80 2024-03-22T13:06:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.053+0000 7f7c33e00640 1 -- 172.21.15.6:0/517145483 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c34013680 con 0x7f7c3c15bc80 2024-03-22T13:06:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.053+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c1f3790 con 0x7f7c3c15bc80 2024-03-22T13:06:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.054+0000 7f7c33e00640 1 -- 172.21.15.6:0/517145483 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34013a90 con 0x7f7c3c15bc80 2024-03-22T13:06:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.054+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c14cbd0 con 0x7f7c3c15bc80 2024-03-22T13:06:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.055+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c0756c0 con 0x7f7c3c15bc80 2024-03-22T13:06:42.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.075+0000 7f7c33e00640 1 -- 172.21.15.6:0/517145483 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c3401a070 con 0x7f7c3c15bc80 2024-03-22T13:06:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.076+0000 7f7c33e00640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c1806bc20 0x7f7c1806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.077+0000 7f7c3b400640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c1806bc20 0x7f7c1806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:42.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.077+0000 7f7c3b400640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c1806bc20 0x7f7c1806e0e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c009a40 tx=0x7f7c2c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.321+0000 7f7c33e00640 1 -- 172.21.15.6:0/517145483 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f7c34057400 con 0x7f7c3c15bc80 2024-03-22T13:06:42.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.340+0000 7f7c33e00640 1 -- 172.21.15.6:0/517145483 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c34059360 con 0x7f7c3c15bc80 2024-03-22T13:06:42.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.617+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c3c0c6400 con 0x7f7c3c15bc80 2024-03-22T13:06:42.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.641+0000 7f7c33e00640 1 -- 172.21.15.6:0/517145483 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f7c3405d400 con 0x7f7c3c15bc80 2024-03-22T13:06:42.641 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:42.641 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:42.642 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c1806bc20 msgr2=0x7f7c1806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c1806bc20 0x7f7c1806e0e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c009a40 tx=0x7f7c2c00b040 comp rx=0 tx=0).stop 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c15bc80 msgr2=0x7f7c3c1577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c15bc80 0x7f7c3c1577c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400e640 tx=0x7f7c3400eb00 comp rx=0 tx=0).stop 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 shutdown_connections 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c3c15bc80 0x7f7c3c1577c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7c3c152400 0x7f7c3c157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7c1806bc20 0x7f7c1806e0e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 --2- 172.21.15.6:0/517145483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c06f8c0 0x7f7c3c14d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 >> 172.21.15.6:0/517145483 conn(0x7f7c3c0cb930 msgr2=0x7f7c3c0715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 shutdown_connections 2024-03-22T13:06:42.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:42.644+0000 7f7c42600640 1 -- 172.21.15.6:0/517145483 wait complete. 2024-03-22T13:06:43.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:43.265 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:44.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:44.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.455+0000 7f5dab200640 1 Processor -- start 2024-03-22T13:06:44.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.455+0000 7f5dab200640 1 -- start start 2024-03-22T13:06:44.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.457+0000 7f5dab200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5da4152400 0x7f5da415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:44.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.457+0000 7f5dab200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5da415bc80 0x7f5da40716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:44.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.457+0000 7f5dab200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da41590b0 con 0x7f5da406f8c0 2024-03-22T13:06:44.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.457+0000 7f5dab200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da40724d0 con 0x7f5da4152400 2024-03-22T13:06:44.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.457+0000 7f5dab200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da40721f0 con 0x7f5da415bc80 2024-03-22T13:06:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.485+0000 7f5d9b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5da415bc80 0x7f5da40716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.486+0000 7f5da9200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5da406f8c0 0x7f5da40c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:57848/0 (socket says 172.21.15.6:57848) 2024-03-22T13:06:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.486+0000 7f5da9200640 1 -- 172.21.15.6:0/2074883616 learned_addr learned my addr 172.21.15.6:0/2074883616 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.489+0000 7f5d9b400640 1 -- 172.21.15.6:0/2074883616 >> v2:172.21.15.6:3300/0 conn(0x7f5da4152400 msgr2=0x7f5da415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.489+0000 7f5d9b400640 1 --2- 172.21.15.6:0/2074883616 >> v2:172.21.15.6:3300/0 conn(0x7f5da4152400 0x7f5da415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.489+0000 7f5d9b400640 1 -- 172.21.15.6:0/2074883616 >> v1:172.21.15.92:6789/0 conn(0x7f5da406f8c0 legacy=0x7f5da40c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:44.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.489+0000 7f5d9b400640 1 -- 172.21.15.6:0/2074883616 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da41582f0 con 0x7f5da415bc80 2024-03-22T13:06:44.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.498+0000 7f5d9b400640 1 --2- 172.21.15.6:0/2074883616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da40716d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5da415c060 tx=0x7f5d8c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d4b6b666fc3d503 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:44.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.502+0000 7f5da3400640 1 -- 172.21.15.6:0/2074883616 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c034070 con 0x7f5da415bc80 2024-03-22T13:06:44.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.502+0000 7f5da3400640 1 -- 172.21.15.6:0/2074883616 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c030610 con 0x7f5da415bc80 2024-03-22T13:06:44.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.502+0000 7f5da3400640 1 -- 172.21.15.6:0/2074883616 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0309e0 con 0x7f5da415bc80 2024-03-22T13:06:44.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.504+0000 7f5dab200640 1 -- 172.21.15.6:0/2074883616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 msgr2=0x7f5da40716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:44.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.504+0000 7f5dab200640 1 --2- 172.21.15.6:0/2074883616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da40716d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5da415c060 tx=0x7f5d8c02b960 comp rx=0 tx=0).stop 2024-03-22T13:06:44.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.504+0000 7f5dab200640 1 -- 172.21.15.6:0/2074883616 shutdown_connections 2024-03-22T13:06:44.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.504+0000 7f5dab200640 1 --2- 172.21.15.6:0/2074883616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da40716d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:44.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.504+0000 7f5dab200640 1 --2- 172.21.15.6:0/2074883616 >> v2:172.21.15.6:3300/0 conn(0x7f5da4152400 0x7f5da415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:44.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.504+0000 7f5dab200640 1 -- 172.21.15.6:0/2074883616 >> 172.21.15.6:0/2074883616 conn(0x7f5da40cb930 msgr2=0x7f5da40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:44.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.505+0000 7f5dab200640 1 -- 172.21.15.6:0/2074883616 shutdown_connections 2024-03-22T13:06:44.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.505+0000 7f5dab200640 1 -- 172.21.15.6:0/2074883616 wait complete. 2024-03-22T13:06:44.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.506+0000 7f5dab200640 1 Processor -- start 2024-03-22T13:06:44.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.507+0000 7f5dab200640 1 -- start start 2024-03-22T13:06:44.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.507+0000 7f5dab200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da406f8c0 0x7f5da40c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.508+0000 7f5dab200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5da4152400 0x7f5da40c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.508+0000 7f5dab200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da41f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.508+0000 7f5dab200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4158b90 con 0x7f5da406f8c0 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.508+0000 7f5dab200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41588b0 con 0x7f5da415bc80 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.508+0000 7f5dab200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41527e0 con 0x7f5da4152400 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.508+0000 7f5da3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da406f8c0 0x7f5da40c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.508+0000 7f5da9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da41f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:44.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.509+0000 7f5da3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da406f8c0 0x7f5da40c4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45836/0 (socket says 172.21.15.6:45836) 2024-03-22T13:06:44.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.509+0000 7f5da3e00640 1 -- 172.21.15.6:0/2924886509 learned_addr learned my addr 172.21.15.6:0/2924886509 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.512+0000 7f5da9200640 1 -- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5da4152400 msgr2=0x7f5da40c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.512+0000 7f5da9200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5da4152400 0x7f5da40c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.512+0000 7f5da9200640 1 -- 172.21.15.6:0/2924886509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da406f8c0 msgr2=0x7f5da40c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.512+0000 7f5da9200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da406f8c0 0x7f5da40c4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:44.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.512+0000 7f5da9200640 1 -- 172.21.15.6:0/2924886509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40c4910 con 0x7f5da415bc80 2024-03-22T13:06:44.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.514+0000 7f5da9200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da41f0a10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00c3c0 tx=0x7f5d9c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:44.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.519+0000 7f5da0c00640 1 -- 172.21.15.6:0/2924886509 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c017070 con 0x7f5da415bc80 2024-03-22T13:06:44.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.519+0000 7f5da0c00640 1 -- 172.21.15.6:0/2924886509 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c005ce0 con 0x7f5da415bc80 2024-03-22T13:06:44.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.519+0000 7f5da0c00640 1 -- 172.21.15.6:0/2924886509 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c0072d0 con 0x7f5da415bc80 2024-03-22T13:06:44.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.519+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da41f4140 con 0x7f5da415bc80 2024-03-22T13:06:44.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.519+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da41f4700 con 0x7f5da415bc80 2024-03-22T13:06:44.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.520+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da4072750 con 0x7f5da415bc80 2024-03-22T13:06:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.812+0000 7f5da0c00640 1 -- 172.21.15.6:0/2924886509 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5d9c01a070 con 0x7f5da415bc80 2024-03-22T13:06:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.814+0000 7f5da0c00640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5d8006bc20 0x7f5d8006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.815+0000 7f5da3e00640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5d8006bc20 0x7f5d8006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.816+0000 7f5da0c00640 1 -- 172.21.15.6:0/2924886509 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5d9c056680 con 0x7f5da415bc80 2024-03-22T13:06:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.816+0000 7f5da3e00640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5d8006bc20 0x7f5d8006e0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5da40c9310 tx=0x7f5d9400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:44.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:44.835+0000 7f5da0c00640 1 -- 172.21.15.6:0/2924886509 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5da4072750 con 0x7f5da415bc80 2024-03-22T13:06:45.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.090+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5da414ee40 con 0x7f5da415bc80 2024-03-22T13:06:45.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.104+0000 7f5da0c00640 1 -- 172.21.15.6:0/2924886509 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f5da414ee40 con 0x7f5da415bc80 2024-03-22T13:06:45.104 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:45.105 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:45.105 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:45.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5d8006bc20 msgr2=0x7f5d8006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5d8006bc20 0x7f5d8006e0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5da40c9310 tx=0x7f5d9400b040 comp rx=0 tx=0).stop 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 msgr2=0x7f5da41f0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da41f0a10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00c3c0 tx=0x7f5d9c00c880 comp rx=0 tx=0).stop 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 shutdown_connections 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5d8006bc20 0x7f5d8006e0e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da415bc80 0x7f5da41f0a10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5da4152400 0x7f5da40c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 --2- 172.21.15.6:0/2924886509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5da406f8c0 0x7f5da40c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.107+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 >> 172.21.15.6:0/2924886509 conn(0x7f5da40cb930 msgr2=0x7f5da40c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.108+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 shutdown_connections 2024-03-22T13:06:45.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:45.108+0000 7f5dab200640 1 -- 172.21.15.6:0/2924886509 wait complete. 2024-03-22T13:06:45.729 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:45.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:46.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:46.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.931+0000 7fec2c600640 1 Processor -- start 2024-03-22T13:06:46.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.932+0000 7fec2c600640 1 -- start start 2024-03-22T13:06:46.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.933+0000 7fec2c600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:46.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.933+0000 7fec2c600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fec24071c10 0x7fec240c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:46.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.933+0000 7fec2c600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec241590b0 con 0x7fec24152400 2024-03-22T13:06:46.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.933+0000 7fec2c600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec240724d0 con 0x7fec24071c10 2024-03-22T13:06:46.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.933+0000 7fec2c600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec240721f0 con 0x7fec2406f8c0 2024-03-22T13:06:46.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.934+0000 7fec29200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:46.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.934+0000 7fec29c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fec24152400 0x7fec2415b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:57864/0 (socket says 172.21.15.6:57864) 2024-03-22T13:06:46.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.934+0000 7fec29c00640 1 -- 172.21.15.6:0/63874334 learned_addr learned my addr 172.21.15.6:0/63874334 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:46.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.935+0000 7fec2a600640 1 --2- 172.21.15.6:0/63874334 >> v2:172.21.15.6:3300/0 conn(0x7fec24071c10 0x7fec240c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:46.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.937+0000 7fec23e00640 1 -- 172.21.15.6:0/63874334 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144471151 0 0) 0x7fec241590b0 con 0x7fec24152400 2024-03-22T13:06:46.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.937+0000 7fec23e00640 1 -- 172.21.15.6:0/63874334 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec0c003520 con 0x7fec24152400 2024-03-22T13:06:46.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.938+0000 7fec2a600640 1 -- 172.21.15.6:0/63874334 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 msgr2=0x7fec240716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:46.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.938+0000 7fec2a600640 1 --2- 172.21.15.6:0/63874334 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:46.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.938+0000 7fec2a600640 1 -- 172.21.15.6:0/63874334 >> v1:172.21.15.92:6789/0 conn(0x7fec24152400 legacy=0x7fec2415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:46.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.938+0000 7fec2a600640 1 -- 172.21.15.6:0/63874334 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec241582f0 con 0x7fec24071c10 2024-03-22T13:06:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.940+0000 7fec2a600640 1 --2- 172.21.15.6:0/63874334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24071c10 0x7fec240c5040 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009d90 tx=0x7fec1c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2774adab9c99a1d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:46.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.946+0000 7fec23e00640 1 -- 172.21.15.6:0/63874334 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c039070 con 0x7fec24071c10 2024-03-22T13:06:46.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.946+0000 7fec23e00640 1 -- 172.21.15.6:0/63874334 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c02cce0 con 0x7fec24071c10 2024-03-22T13:06:46.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.946+0000 7fec23e00640 1 -- 172.21.15.6:0/63874334 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c034660 con 0x7fec24071c10 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.946+0000 7fec2c600640 1 -- 172.21.15.6:0/63874334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24071c10 msgr2=0x7fec240c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.946+0000 7fec2c600640 1 --2- 172.21.15.6:0/63874334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24071c10 0x7fec240c5040 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009d90 tx=0x7fec1c02bf30 comp rx=0 tx=0).stop 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.947+0000 7fec2c600640 1 -- 172.21.15.6:0/63874334 shutdown_connections 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.947+0000 7fec2c600640 1 --2- 172.21.15.6:0/63874334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24071c10 0x7fec240c5040 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.947+0000 7fec2c600640 1 --2- 172.21.15.6:0/63874334 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.947+0000 7fec2c600640 1 -- 172.21.15.6:0/63874334 >> 172.21.15.6:0/63874334 conn(0x7fec240cb930 msgr2=0x7fec240cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.947+0000 7fec2c600640 1 -- 172.21.15.6:0/63874334 shutdown_connections 2024-03-22T13:06:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.947+0000 7fec2c600640 1 -- 172.21.15.6:0/63874334 wait complete. 2024-03-22T13:06:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.948+0000 7fec2c600640 1 Processor -- start 2024-03-22T13:06:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.949+0000 7fec2c600640 1 -- start start 2024-03-22T13:06:46.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.950+0000 7fec2c600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240c3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:46.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.950+0000 7fec2c600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec24071c10 0x7fec24157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:46.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.950+0000 7fec2c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24152400 0x7fec241577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:46.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.950+0000 7fec2c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec24158b90 con 0x7fec24152400 2024-03-22T13:06:46.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.950+0000 7fec2c600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec241588b0 con 0x7fec24071c10 2024-03-22T13:06:46.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.950+0000 7fec2c600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec241527e0 con 0x7fec2406f8c0 2024-03-22T13:06:46.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.951+0000 7fec29c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:46.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.951+0000 7fec2a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24152400 0x7fec241577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:46.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.951+0000 7fec29c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36956/0 (socket says 172.21.15.6:36956) 2024-03-22T13:06:46.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.951+0000 7fec29c00640 1 -- 172.21.15.6:0/2027303064 learned_addr learned my addr 172.21.15.6:0/2027303064 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:46.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.955+0000 7fec2a600640 1 -- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 msgr2=0x7fec240c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:46.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.955+0000 7fec2a600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:46.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.955+0000 7fec2a600640 1 -- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec24071c10 msgr2=0x7fec24157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:46.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.955+0000 7fec2a600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec24071c10 0x7fec24157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:46.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.955+0000 7fec2a600640 1 -- 172.21.15.6:0/2027303064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240c47e0 con 0x7fec24152400 2024-03-22T13:06:46.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.958+0000 7fec2a600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24152400 0x7fec241577d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec1c033000 tx=0x7fec1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.964+0000 7fec21600640 1 -- 172.21.15.6:0/2027303064 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c044020 con 0x7fec24152400 2024-03-22T13:06:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.964+0000 7fec21600640 1 -- 172.21.15.6:0/2027303064 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c03f470 con 0x7fec24152400 2024-03-22T13:06:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.964+0000 7fec21600640 1 -- 172.21.15.6:0/2027303064 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c039040 con 0x7fec24152400 2024-03-22T13:06:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.964+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec240c4ac0 con 0x7fec24152400 2024-03-22T13:06:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.964+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec24145340 con 0x7fec24152400 2024-03-22T13:06:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.965+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec240756c0 con 0x7fec24152400 2024-03-22T13:06:46.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.988+0000 7fec21600640 1 -- 172.21.15.6:0/2027303064 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fec1c047020 con 0x7fec24152400 2024-03-22T13:06:46.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.988+0000 7fec21600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec0406bc50 0x7fec0406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:46.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.989+0000 7fec29c00640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec0406bc50 0x7fec0406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:46.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.990+0000 7fec29c00640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec0406bc50 0x7fec0406e110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec18005880 tx=0x7fec18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:46.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:46.992+0000 7fec21600640 1 -- 172.21.15.6:0/2027303064 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fec1c079b50 con 0x7fec24152400 2024-03-22T13:06:47.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.343+0000 7fec21600640 1 -- 172.21.15.6:0/2027303064 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fec1c04e030 con 0x7fec24152400 2024-03-22T13:06:47.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.619+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec24147ff0 con 0x7fec24152400 2024-03-22T13:06:47.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.644+0000 7fec21600640 1 -- 172.21.15.6:0/2027303064 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fec1c07c1b0 con 0x7fec24152400 2024-03-22T13:06:47.644 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:47.644 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.646+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec0406bc50 msgr2=0x7fec0406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.646+0000 7fec2c600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec0406bc50 0x7fec0406e110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec18005880 tx=0x7fec18009040 comp rx=0 tx=0).stop 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.646+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24152400 msgr2=0x7fec241577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.646+0000 7fec2c600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24152400 0x7fec241577d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec1c033000 tx=0x7fec1c004a40 comp rx=0 tx=0).stop 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 shutdown_connections 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec24152400 0x7fec241577d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec0406bc50 0x7fec0406e110 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec24071c10 0x7fec24157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 --2- 172.21.15.6:0/2027303064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2406f8c0 0x7fec240c3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 >> 172.21.15.6:0/2027303064 conn(0x7fec240cb930 msgr2=0x7fec240c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:47.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 shutdown_connections 2024-03-22T13:06:47.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:47.647+0000 7fec2c600640 1 -- 172.21.15.6:0/2027303064 wait complete. 2024-03-22T13:06:48.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:48.267 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:49.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.465+0000 7fc551c00640 1 Processor -- start 2024-03-22T13:06:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.465+0000 7fc551c00640 1 -- start start 2024-03-22T13:06:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.466+0000 7fc551c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc54c0cc790 0x7fc54c0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.466+0000 7fc551c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.467+0000 7fc551c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c15b510 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.467+0000 7fc551c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54c0cd0f0 con 0x7fc54c0cc790 2024-03-22T13:06:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.467+0000 7fc551c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c0ce230 con 0x7fc54c15ab40 2024-03-22T13:06:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.467+0000 7fc54b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.467+0000 7fc54a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc54c0cc790 0x7fc54c0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:49.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.468+0000 7fc54b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36970/0 (socket says 172.21.15.6:36970) 2024-03-22T13:06:49.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.468+0000 7fc54b400640 1 -- 172.21.15.6:0/1624876620 learned_addr learned my addr 172.21.15.6:0/1624876620 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.471+0000 7fc54b400640 1 -- 172.21.15.6:0/1624876620 >> v2:172.21.15.6:3300/0 conn(0x7fc54c0cc790 msgr2=0x7fc54c0ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.471+0000 7fc54b400640 1 --2- 172.21.15.6:0/1624876620 >> v2:172.21.15.6:3300/0 conn(0x7fc54c0cc790 0x7fc54c0ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.471+0000 7fc54b400640 1 -- 172.21.15.6:0/1624876620 >> v1:172.21.15.6:6789/0 conn(0x7fc54c0cc3b0 legacy=0x7fc54c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.471+0000 7fc54b400640 1 -- 172.21.15.6:0/1624876620 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0ce510 con 0x7fc54c15ab40 2024-03-22T13:06:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.474+0000 7fc54b400640 1 --2- 172.21.15.6:0/1624876620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15afd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc544009870 tx=0x7fc54402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5321c5084f97a9d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.480+0000 7fc549600640 1 -- 172.21.15.6:0/1624876620 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544035020 con 0x7fc54c15ab40 2024-03-22T13:06:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.480+0000 7fc549600640 1 -- 172.21.15.6:0/1624876620 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc544031640 con 0x7fc54c15ab40 2024-03-22T13:06:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.480+0000 7fc549600640 1 -- 172.21.15.6:0/1624876620 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54402d040 con 0x7fc54c15ab40 2024-03-22T13:06:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.480+0000 7fc551c00640 1 -- 172.21.15.6:0/1624876620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 msgr2=0x7fc54c15afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.480+0000 7fc551c00640 1 --2- 172.21.15.6:0/1624876620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15afd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc544009870 tx=0x7fc54402b8f0 comp rx=0 tx=0).stop 2024-03-22T13:06:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.481+0000 7fc551c00640 1 -- 172.21.15.6:0/1624876620 shutdown_connections 2024-03-22T13:06:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.481+0000 7fc551c00640 1 --2- 172.21.15.6:0/1624876620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15afd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.481+0000 7fc551c00640 1 --2- 172.21.15.6:0/1624876620 >> v2:172.21.15.6:3300/0 conn(0x7fc54c0cc790 0x7fc54c0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.481+0000 7fc551c00640 1 -- 172.21.15.6:0/1624876620 >> 172.21.15.6:0/1624876620 conn(0x7fc54c0b16d0 msgr2=0x7fc54c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.481+0000 7fc551c00640 1 -- 172.21.15.6:0/1624876620 shutdown_connections 2024-03-22T13:06:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.481+0000 7fc551c00640 1 -- 172.21.15.6:0/1624876620 wait complete. 2024-03-22T13:06:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.482+0000 7fc551c00640 1 Processor -- start 2024-03-22T13:06:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.483+0000 7fc551c00640 1 -- start start 2024-03-22T13:06:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.483+0000 7fc551c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc54c0cc3b0 0x7fc54c0c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.483+0000 7fc551c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc54c0cc790 0x7fc54c1f0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.483+0000 7fc551c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.483+0000 7fc551c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c0cedb0 con 0x7fc54c0cc790 2024-03-22T13:06:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.483+0000 7fc551c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c0cead0 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.483+0000 7fc551c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c06dc10 con 0x7fc54c15ab40 2024-03-22T13:06:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.484+0000 7fc54b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.484+0000 7fc54aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc54c0cc3b0 0x7fc54c0c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.485+0000 7fc54a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc54c0cc790 0x7fc54c1f0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:49.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.485+0000 7fc54b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36984/0 (socket says 172.21.15.6:36984) 2024-03-22T13:06:49.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.485+0000 7fc54aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc54c0cc3b0 0x7fc54c0c5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56090/0 (socket says 172.21.15.6:56090) 2024-03-22T13:06:49.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.485+0000 7fc54b400640 1 -- 172.21.15.6:0/2489808406 learned_addr learned my addr 172.21.15.6:0/2489808406 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:49.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.489+0000 7fc54aa00640 1 -- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 msgr2=0x7fc54c15d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:49.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.489+0000 7fc54aa00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:49.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.489+0000 7fc54aa00640 1 -- 172.21.15.6:0/2489808406 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc54c0cc790 msgr2=0x7fc54c1f0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:49.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.489+0000 7fc54aa00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc54c0cc790 0x7fc54c1f0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:49.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.489+0000 7fc54aa00640 1 -- 172.21.15.6:0/2489808406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c1f1300 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.492+0000 7fc54aa00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc54c0cc3b0 0x7fc54c0c5870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c760 tx=0x7fc53c00cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.496+0000 7fc542a00640 1 -- 172.21.15.6:0/2489808406 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c016020 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.496+0000 7fc542a00640 1 -- 172.21.15.6:0/2489808406 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c006d50 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.496+0000 7fc542a00640 1 -- 172.21.15.6:0/2489808406 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c003bb0 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.497+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c0c5f40 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.497+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c1f3b80 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.498+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54c0756c0 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.816+0000 7fc542a00640 1 -- 172.21.15.6:0/2489808406 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc53c00a7f0 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.818+0000 7fc542a00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc52806bc20 0x7fc52806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:49.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.819+0000 7fc54a000640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc52806bc20 0x7fc52806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:49.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.819+0000 7fc54a000640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc52806bc20 0x7fc52806e0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc54c1594e0 tx=0x7fc534009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:49.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.821+0000 7fc542a00640 1 -- 172.21.15.6:0/2489808406 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc53c0578f0 con 0x7fc54c0cc3b0 2024-03-22T13:06:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:49.836+0000 7fc542a00640 1 -- 172.21.15.6:0/2489808406 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc53c020020 con 0x7fc54c0cc3b0 2024-03-22T13:06:50.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.087+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc54c15e970 con 0x7fc54c0cc3b0 2024-03-22T13:06:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.108+0000 7fc542a00640 1 -- 172.21.15.6:0/2489808406 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fc54c1f1300 con 0x7fc54c0cc3b0 2024-03-22T13:06:50.108 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:50.108 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:50.108 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.110+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc52806bc20 msgr2=0x7fc52806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.110+0000 7fc551c00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc52806bc20 0x7fc52806e0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc54c1594e0 tx=0x7fc534009190 comp rx=0 tx=0).stop 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.110+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc54c0cc3b0 msgr2=0x7fc54c0c5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.110+0000 7fc551c00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc54c0cc3b0 0x7fc54c0c5870 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c760 tx=0x7fc53c00cc20 comp rx=0 tx=0).stop 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 shutdown_connections 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc54c15ab40 0x7fc54c15d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc54c0cc790 0x7fc54c1f0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc52806bc20 0x7fc52806e0e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 --2- 172.21.15.6:0/2489808406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc54c0cc3b0 0x7fc54c0c5870 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 >> 172.21.15.6:0/2489808406 conn(0x7fc54c0b16d0 msgr2=0x7fc54c0cab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:50.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 shutdown_connections 2024-03-22T13:06:50.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:50.111+0000 7fc551c00640 1 -- 172.21.15.6:0/2489808406 wait complete. 2024-03-22T13:06:50.731 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:50.731 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:51.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:51.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.938+0000 7f08ee200640 1 Processor -- start 2024-03-22T13:06:51.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.938+0000 7f08ee200640 1 -- start start 2024-03-22T13:06:51.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.939+0000 7f08ee200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f08e806f8c0 0x7f08e80716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:51.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.939+0000 7f08ee200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e81590b0 con 0x7f08e8071c10 2024-03-22T13:06:51.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.939+0000 7f08ee200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e80721f0 con 0x7f08e8152400 2024-03-22T13:06:51.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.939+0000 7f08ee200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e80724d0 con 0x7f08e806f8c0 2024-03-22T13:06:51.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.940+0000 7f08e6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f08e806f8c0 0x7f08e80716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:51.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.940+0000 7f08e7400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f08e8152400 0x7f08e815b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:57878/0 (socket says 172.21.15.6:57878) 2024-03-22T13:06:51.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.940+0000 7f08e7400640 1 -- 172.21.15.6:0/2801340886 learned_addr learned my addr 172.21.15.6:0/2801340886 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:51.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.943+0000 7f08e6000640 1 -- 172.21.15.6:0/2801340886 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852822992 0 0) 0x7f08e80721f0 con 0x7f08e8152400 2024-03-22T13:06:51.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.943+0000 7f08e6000640 1 -- 172.21.15.6:0/2801340886 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08cc003520 con 0x7f08e8152400 2024-03-22T13:06:51.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.944+0000 7f08e6a00640 1 -- 172.21.15.6:0/2801340886 >> v1:172.21.15.92:6789/0 conn(0x7f08e8152400 legacy=0x7f08e815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:51.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.944+0000 7f08e6a00640 1 -- 172.21.15.6:0/2801340886 >> v1:172.21.15.6:6789/0 conn(0x7f08e8071c10 legacy=0x7f08e80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:51.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.944+0000 7f08e6a00640 1 -- 172.21.15.6:0/2801340886 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e81582f0 con 0x7f08e806f8c0 2024-03-22T13:06:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.949+0000 7f08e6a00640 1 --2- 172.21.15.6:0/2801340886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e806f8c0 0x7f08e80716d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f08d40097a0 tx=0x7f08d402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bbff32e9f1b0b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:51.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.955+0000 7f08e6000640 1 -- 172.21.15.6:0/2801340886 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d402f810 con 0x7f08e806f8c0 2024-03-22T13:06:51.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.955+0000 7f08e6000640 1 -- 172.21.15.6:0/2801340886 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d402f9b0 con 0x7f08e806f8c0 2024-03-22T13:06:51.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.955+0000 7f08e6000640 1 -- 172.21.15.6:0/2801340886 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d402fd80 con 0x7f08e806f8c0 2024-03-22T13:06:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.956+0000 7f08ee200640 1 -- 172.21.15.6:0/2801340886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e806f8c0 msgr2=0x7f08e80716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.956+0000 7f08ee200640 1 --2- 172.21.15.6:0/2801340886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e806f8c0 0x7f08e80716d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f08d40097a0 tx=0x7f08d402b9f0 comp rx=0 tx=0).stop 2024-03-22T13:06:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.957+0000 7f08ee200640 1 -- 172.21.15.6:0/2801340886 shutdown_connections 2024-03-22T13:06:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.957+0000 7f08ee200640 1 --2- 172.21.15.6:0/2801340886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e806f8c0 0x7f08e80716d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.957+0000 7f08ee200640 1 -- 172.21.15.6:0/2801340886 >> 172.21.15.6:0/2801340886 conn(0x7f08e80cb930 msgr2=0x7f08e80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.957+0000 7f08ee200640 1 -- 172.21.15.6:0/2801340886 shutdown_connections 2024-03-22T13:06:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.957+0000 7f08ee200640 1 -- 172.21.15.6:0/2801340886 wait complete. 2024-03-22T13:06:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.959+0000 7f08ee200640 1 Processor -- start 2024-03-22T13:06:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.959+0000 7f08ee200640 1 -- start start 2024-03-22T13:06:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.960+0000 7f08ee200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08e806f8c0 0x7f08e8140770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.960+0000 7f08ee200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08e8071c10 0x7f08e80c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.960+0000 7f08ee200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e8152400 0x7f08e80c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.960+0000 7f08ee200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8158b90 con 0x7f08e8152400 2024-03-22T13:06:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.960+0000 7f08ee200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e81588b0 con 0x7f08e806f8c0 2024-03-22T13:06:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.960+0000 7f08ee200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e81527e0 con 0x7f08e8071c10 2024-03-22T13:06:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.961+0000 7f08e7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08e806f8c0 0x7f08e8140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.961+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e8152400 0x7f08e80c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.961+0000 7f08e6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08e8071c10 0x7f08e80c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.961+0000 7f08e7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08e806f8c0 0x7f08e8140770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56100/0 (socket says 172.21.15.6:56100) 2024-03-22T13:06:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.961+0000 7f08e7400640 1 -- 172.21.15.6:0/2647716778 learned_addr learned my addr 172.21.15.6:0/2647716778 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.965+0000 7f08e7400640 1 -- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08e8071c10 msgr2=0x7f08e80c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.965+0000 7f08e7400640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08e8071c10 0x7f08e80c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.965+0000 7f08e7400640 1 -- 172.21.15.6:0/2647716778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e8152400 msgr2=0x7f08e80c34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.965+0000 7f08e7400640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e8152400 0x7f08e80c34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.965+0000 7f08e7400640 1 -- 172.21.15.6:0/2647716778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e81ebd60 con 0x7f08e806f8c0 2024-03-22T13:06:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.968+0000 7f08e7400640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08e806f8c0 0x7f08e8140770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f08d800c410 tx=0x7f08d800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.973+0000 7f08df400640 1 -- 172.21.15.6:0/2647716778 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d8017070 con 0x7f08e806f8c0 2024-03-22T13:06:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.973+0000 7f08df400640 1 -- 172.21.15.6:0/2647716778 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d8005ce0 con 0x7f08e806f8c0 2024-03-22T13:06:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.973+0000 7f08df400640 1 -- 172.21.15.6:0/2647716778 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80072f0 con 0x7f08e806f8c0 2024-03-22T13:06:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.973+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e8140cb0 con 0x7f08e806f8c0 2024-03-22T13:06:51.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.973+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e81ebfe0 con 0x7f08e806f8c0 2024-03-22T13:06:51.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:51.974+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e80721f0 con 0x7f08e806f8c0 2024-03-22T13:06:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.236+0000 7f08df400640 1 -- 172.21.15.6:0/2647716778 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f08d801a070 con 0x7f08e806f8c0 2024-03-22T13:06:52.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.238+0000 7f08df400640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08c806bc20 0x7f08c806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:52.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.238+0000 7f08df400640 1 -- 172.21.15.6:0/2647716778 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f08d808a6d0 con 0x7f08e806f8c0 2024-03-22T13:06:52.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.239+0000 7f08e6a00640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08c806bc20 0x7f08c806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.240+0000 7f08e6a00640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08c806bc20 0x7f08c806e0e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f08d4036000 tx=0x7f08d4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:52.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.248+0000 7f08df400640 1 -- 172.21.15.6:0/2647716778 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f08e80721f0 con 0x7f08e806f8c0 2024-03-22T13:06:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.505+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e8072750 con 0x7f08e806f8c0 2024-03-22T13:06:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.518+0000 7f08df400640 1 -- 172.21.15.6:0/2647716778 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f08e8072750 con 0x7f08e806f8c0 2024-03-22T13:06:52.518 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:52.518 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08c806bc20 msgr2=0x7f08c806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08c806bc20 0x7f08c806e0e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f08d4036000 tx=0x7f08d4008030 comp rx=0 tx=0).stop 2024-03-22T13:06:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08e806f8c0 msgr2=0x7f08e8140770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08e806f8c0 0x7f08e8140770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f08d800c410 tx=0x7f08d800c8d0 comp rx=0 tx=0).stop 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 shutdown_connections 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08e8152400 0x7f08e80c34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08e8071c10 0x7f08e80c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f08c806bc20 0x7f08c806e0e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 --2- 172.21.15.6:0/2647716778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08e806f8c0 0x7f08e8140770 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 >> 172.21.15.6:0/2647716778 conn(0x7f08e80cb930 msgr2=0x7f08e80c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 shutdown_connections 2024-03-22T13:06:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:52.521+0000 7f08ee200640 1 -- 172.21.15.6:0/2647716778 wait complete. 2024-03-22T13:06:53.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:53.142 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:54.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.446+0000 7f809c600640 1 Processor -- start 2024-03-22T13:06:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.446+0000 7f809c600640 1 -- start start 2024-03-22T13:06:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.448+0000 7f809c600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f809406f8c0 0x7f80940716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.448+0000 7f809c600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8094071c10 0x7f80940c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.448+0000 7f809c600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80941590b0 con 0x7f8094152400 2024-03-22T13:06:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.448+0000 7f809c600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940724d0 con 0x7f809406f8c0 2024-03-22T13:06:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.448+0000 7f809c600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940721f0 con 0x7f8094071c10 2024-03-22T13:06:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.449+0000 7f8099200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f809406f8c0 0x7f80940716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.449+0000 7f809a600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8094071c10 0x7f80940c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.449+0000 7f8099c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8094152400 0x7f809415b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:55666/0 (socket says 172.21.15.6:55666) 2024-03-22T13:06:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.449+0000 7f8099c00640 1 -- 172.21.15.6:0/1391529367 learned_addr learned my addr 172.21.15.6:0/1391529367 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.452+0000 7f8099200640 1 -- 172.21.15.6:0/1391529367 >> v2:172.21.15.92:3300/0 conn(0x7f8094071c10 msgr2=0x7f80940c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.452+0000 7f8099200640 1 --2- 172.21.15.6:0/1391529367 >> v2:172.21.15.92:3300/0 conn(0x7f8094071c10 0x7f80940c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.452+0000 7f8099200640 1 -- 172.21.15.6:0/1391529367 >> v1:172.21.15.92:6789/0 conn(0x7f8094152400 legacy=0x7f809415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.452+0000 7f8099200640 1 -- 172.21.15.6:0/1391529367 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80941582f0 con 0x7f809406f8c0 2024-03-22T13:06:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.454+0000 7f8099200640 1 --2- 172.21.15.6:0/1391529367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f80940716d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f809415c830 tx=0x7f808002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9a3a0630477ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.459+0000 7f8093e00640 1 -- 172.21.15.6:0/1391529367 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080034070 con 0x7f809406f8c0 2024-03-22T13:06:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.459+0000 7f8093e00640 1 -- 172.21.15.6:0/1391529367 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080030610 con 0x7f809406f8c0 2024-03-22T13:06:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.459+0000 7f8093e00640 1 -- 172.21.15.6:0/1391529367 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80800309e0 con 0x7f809406f8c0 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 -- 172.21.15.6:0/1391529367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 msgr2=0x7f80940716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 --2- 172.21.15.6:0/1391529367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f80940716d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f809415c830 tx=0x7f808002b960 comp rx=0 tx=0).stop 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 -- 172.21.15.6:0/1391529367 shutdown_connections 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 --2- 172.21.15.6:0/1391529367 >> v2:172.21.15.92:3300/0 conn(0x7f8094071c10 0x7f80940c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 --2- 172.21.15.6:0/1391529367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f80940716d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 -- 172.21.15.6:0/1391529367 >> 172.21.15.6:0/1391529367 conn(0x7f80940cb930 msgr2=0x7f80940cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 -- 172.21.15.6:0/1391529367 shutdown_connections 2024-03-22T13:06:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.460+0000 7f809c600640 1 -- 172.21.15.6:0/1391529367 wait complete. 2024-03-22T13:06:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.462+0000 7f809c600640 1 Processor -- start 2024-03-22T13:06:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.463+0000 7f809c600640 1 -- start start 2024-03-22T13:06:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f809c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f8094147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f809c600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8094071c10 0x7f80941f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f809c600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8094152400 0x7f80941f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f809c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8094158b90 con 0x7f809406f8c0 2024-03-22T13:06:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f809c600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80941588b0 con 0x7f8094152400 2024-03-22T13:06:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f809c600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80941527e0 con 0x7f8094071c10 2024-03-22T13:06:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f809a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8094152400 0x7f80941f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.464+0000 7f8099c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f8094147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.465+0000 7f809a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8094152400 0x7f80941f49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54764/0 (socket says 172.21.15.6:54764) 2024-03-22T13:06:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.465+0000 7f8099c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f8094147720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41386/0 (socket says 172.21.15.6:41386) 2024-03-22T13:06:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.465+0000 7f809a600640 1 -- 172.21.15.6:0/4070421557 learned_addr learned my addr 172.21.15.6:0/4070421557 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.469+0000 7f809a600640 1 -- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8094071c10 msgr2=0x7f80941f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.469+0000 7f809a600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8094071c10 0x7f80941f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.469+0000 7f809a600640 1 -- 172.21.15.6:0/4070421557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 msgr2=0x7f8094147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.469+0000 7f809a600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f8094147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.469+0000 7f809a600640 1 -- 172.21.15.6:0/4070421557 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940ca350 con 0x7f8094152400 2024-03-22T13:06:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.471+0000 7f809a600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8094152400 0x7f80941f49b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f808c00efc0 tx=0x7f808c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.476+0000 7f8091600640 1 -- 172.21.15.6:0/4070421557 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c018020 con 0x7f8094152400 2024-03-22T13:06:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.476+0000 7f8091600640 1 -- 172.21.15.6:0/4070421557 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c014480 con 0x7f8094152400 2024-03-22T13:06:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.476+0000 7f8091600640 1 -- 172.21.15.6:0/4070421557 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c014870 con 0x7f8094152400 2024-03-22T13:06:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.476+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80940ca630 con 0x7f8094152400 2024-03-22T13:06:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.476+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8094149480 con 0x7f8094152400 2024-03-22T13:06:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.478+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8094072d10 con 0x7f8094152400 2024-03-22T13:06:54.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.496+0000 7f8091600640 1 -- 172.21.15.6:0/4070421557 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f808c005d10 con 0x7f8094152400 2024-03-22T13:06:54.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.497+0000 7f8091600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807406bc50 0x7f807406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:54.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.497+0000 7f8099c00640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807406bc50 0x7f807406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.498+0000 7f8099c00640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807406bc50 0x7f807406e110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8088005880 tx=0x7f8088009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:54.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.739+0000 7f8091600640 1 -- 172.21.15.6:0/4070421557 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f808c058150 con 0x7f8094152400 2024-03-22T13:06:54.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:54.750+0000 7f8091600640 1 -- 172.21.15.6:0/4070421557 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8094072d10 con 0x7f8094152400 2024-03-22T13:06:55.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.012+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80940756c0 con 0x7f8094152400 2024-03-22T13:06:55.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.044+0000 7f8091600640 1 -- 172.21.15.6:0/4070421557 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f808c022070 con 0x7f8094152400 2024-03-22T13:06:55.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:55.044 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:55.044 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:55.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807406bc50 msgr2=0x7f807406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:55.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807406bc50 0x7f807406e110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8088005880 tx=0x7f8088009040 comp rx=0 tx=0).stop 2024-03-22T13:06:55.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8094152400 msgr2=0x7f80941f49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:55.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8094152400 0x7f80941f49b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f808c00efc0 tx=0x7f808c00c490 comp rx=0 tx=0).stop 2024-03-22T13:06:55.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 shutdown_connections 2024-03-22T13:06:55.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f807406bc50 0x7f807406e110 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:55.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8094152400 0x7f80941f49b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8094071c10 0x7f80941f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 --2- 172.21.15.6:0/4070421557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f809406f8c0 0x7f8094147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 >> 172.21.15.6:0/4070421557 conn(0x7f80940cb930 msgr2=0x7f8094150cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 shutdown_connections 2024-03-22T13:06:55.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:55.047+0000 7f809c600640 1 -- 172.21.15.6:0/4070421557 wait complete. 2024-03-22T13:06:55.675 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:55.675 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:56.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.873+0000 7fcea1200640 1 Processor -- start 2024-03-22T13:06:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.874+0000 7fcea1200640 1 -- start start 2024-03-22T13:06:56.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.875+0000 7fcea1200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce9c0cc3b0 0x7fce9c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:56.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.875+0000 7fcea1200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9c0ce230 con 0x7fce9c0cf770 2024-03-22T13:06:56.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.875+0000 7fcea1200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9c0ce510 con 0x7fce9c0ccea0 2024-03-22T13:06:56.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.875+0000 7fcea1200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce9c0ce7f0 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.876+0000 7fce99e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce9c0cc3b0 0x7fce9c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:56.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.876+0000 7fce99400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fce9c0cf770 0x7fce9c0cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55022/0 (socket says 172.21.15.6:55022) 2024-03-22T13:06:56.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.876+0000 7fce99400640 1 -- 172.21.15.6:0/340027066 learned_addr learned my addr 172.21.15.6:0/340027066 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:06:56.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.880+0000 7fce98a00640 1 -- 172.21.15.6:0/340027066 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866554075 0 0) 0x7fce9c0ce230 con 0x7fce9c0cf770 2024-03-22T13:06:56.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.880+0000 7fce98a00640 1 -- 172.21.15.6:0/340027066 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce80003520 con 0x7fce9c0cf770 2024-03-22T13:06:56.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.881+0000 7fce99e00640 1 -- 172.21.15.6:0/340027066 >> v1:172.21.15.92:6789/0 conn(0x7fce9c0ccea0 legacy=0x7fce9c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:56.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.881+0000 7fce99e00640 1 -- 172.21.15.6:0/340027066 >> v1:172.21.15.6:6789/0 conn(0x7fce9c0cf770 legacy=0x7fce9c0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:56.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.881+0000 7fce99e00640 1 -- 172.21.15.6:0/340027066 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c0cead0 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.886+0000 7fce99e00640 1 --2- 172.21.15.6:0/340027066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cc3b0 0x7fce9c0cf230 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0097a0 tx=0x7fce8c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65075031f2287454 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:56.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.891+0000 7fce98a00640 1 -- 172.21.15.6:0/340027066 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c035020 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.891+0000 7fce98a00640 1 -- 172.21.15.6:0/340027066 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c031830 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.891+0000 7fce98a00640 1 -- 172.21.15.6:0/340027066 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c031c00 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.891+0000 7fcea1200640 1 -- 172.21.15.6:0/340027066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cc3b0 msgr2=0x7fce9c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:56.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.891+0000 7fcea1200640 1 --2- 172.21.15.6:0/340027066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cc3b0 0x7fce9c0cf230 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0097a0 tx=0x7fce8c02b8d0 comp rx=0 tx=0).stop 2024-03-22T13:06:56.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.892+0000 7fcea1200640 1 -- 172.21.15.6:0/340027066 shutdown_connections 2024-03-22T13:06:56.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.892+0000 7fcea1200640 1 --2- 172.21.15.6:0/340027066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cc3b0 0x7fce9c0cf230 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:56.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.892+0000 7fcea1200640 1 -- 172.21.15.6:0/340027066 >> 172.21.15.6:0/340027066 conn(0x7fce9c0b16d0 msgr2=0x7fce9c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:56.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.892+0000 7fcea1200640 1 -- 172.21.15.6:0/340027066 shutdown_connections 2024-03-22T13:06:56.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.892+0000 7fcea1200640 1 -- 172.21.15.6:0/340027066 wait complete. 2024-03-22T13:06:56.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.894+0000 7fcea1200640 1 Processor -- start 2024-03-22T13:06:56.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.895+0000 7fcea1200640 1 -- start start 2024-03-22T13:06:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.896+0000 7fcea1200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce9c0cc3b0 0x7fce9c0c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.896+0000 7fcea1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce9c0ccea0 0x7fce9c0b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.896+0000 7fcea1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cf770 0x7fce9c157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.896+0000 7fcea1200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c15a5d0 con 0x7fce9c0ccea0 2024-03-22T13:06:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.896+0000 7fcea1200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c06dc10 con 0x7fce9c0cf770 2024-03-22T13:06:56.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.896+0000 7fcea1200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c154bb0 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.897+0000 7fce99e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce9c0cc3b0 0x7fce9c0c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.898+0000 7fce9a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cf770 0x7fce9c157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.898+0000 7fce99400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce9c0ccea0 0x7fce9c0b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.898+0000 7fce99e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce9c0cc3b0 0x7fce9c0c8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36234/0 (socket says 172.21.15.6:36234) 2024-03-22T13:06:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.898+0000 7fce99e00640 1 -- 172.21.15.6:0/2537224708 learned_addr learned my addr 172.21.15.6:0/2537224708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:56.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.901+0000 7fce99e00640 1 -- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cf770 msgr2=0x7fce9c157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:56.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.901+0000 7fce99e00640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cf770 0x7fce9c157190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:56.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.901+0000 7fce99e00640 1 -- 172.21.15.6:0/2537224708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce9c0ccea0 msgr2=0x7fce9c0b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:56.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.901+0000 7fce99e00640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce9c0ccea0 0x7fce9c0b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:56.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.901+0000 7fce99e00640 1 -- 172.21.15.6:0/2537224708 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c0c7970 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.903+0000 7fce99e00640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce9c0cc3b0 0x7fce9c0c8e50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fce8c02bee0 tx=0x7fce8c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:56.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.908+0000 7fce92000640 1 -- 172.21.15.6:0/2537224708 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c035040 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.908+0000 7fce92000640 1 -- 172.21.15.6:0/2537224708 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c03f070 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.908+0000 7fce92000640 1 -- 172.21.15.6:0/2537224708 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c03bb60 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.908+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9c0c7c10 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.908+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9c0c81d0 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.910+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c0cb030 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.944+0000 7fce92000640 1 -- 172.21.15.6:0/2537224708 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fce8c038020 con 0x7fce9c0cc3b0 2024-03-22T13:06:56.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.946+0000 7fce92000640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce7c06bc50 0x7fce7c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:56.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.946+0000 7fce99400640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce7c06bc50 0x7fce7c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:56.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:56.948+0000 7fce99400640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce7c06bc50 0x7fce7c06e110 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0c5d30 tx=0x7fce88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:57.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.200+0000 7fce92000640 1 -- 172.21.15.6:0/2537224708 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fce8c078e90 con 0x7fce9c0cc3b0 2024-03-22T13:06:57.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.220+0000 7fce92000640 1 -- 172.21.15.6:0/2537224708 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fce9c0cb030 con 0x7fce9c0cc3b0 2024-03-22T13:06:57.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.496+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce9c0b1fe0 con 0x7fce9c0cc3b0 2024-03-22T13:06:57.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.511+0000 7fce92000640 1 -- 172.21.15.6:0/2537224708 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fce9c0b1fe0 con 0x7fce9c0cc3b0 2024-03-22T13:06:57.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:06:57.511 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:06:57.511 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce7c06bc50 msgr2=0x7fce7c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce7c06bc50 0x7fce7c06e110 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fce9c0c5d30 tx=0x7fce88009040 comp rx=0 tx=0).stop 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce9c0cc3b0 msgr2=0x7fce9c0c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce9c0cc3b0 0x7fce9c0c8e50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fce8c02bee0 tx=0x7fce8c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 shutdown_connections 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce7c06bc50 0x7fce7c06e110 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce9c0cf770 0x7fce9c157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce9c0ccea0 0x7fce9c0b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 --2- 172.21.15.6:0/2537224708 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce9c0cc3b0 0x7fce9c0c8e50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 >> 172.21.15.6:0/2537224708 conn(0x7fce9c0b16d0 msgr2=0x7fce9c06d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 shutdown_connections 2024-03-22T13:06:57.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:57.514+0000 7fcea1200640 1 -- 172.21.15.6:0/2537224708 wait complete. 2024-03-22T13:06:58.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:06:58.135 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:06:59.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:06:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.335+0000 7f35ea400640 1 Processor -- start 2024-03-22T13:06:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.335+0000 7f35ea400640 1 -- start start 2024-03-22T13:06:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.336+0000 7f35ea400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f35e40ca370 0x7f35e415e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.336+0000 7f35ea400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35e406d340 0x7f35e406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.336+0000 7f35ea400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e4154990 con 0x7f35e40ca750 2024-03-22T13:06:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.336+0000 7f35ea400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e406ceb0 con 0x7f35e406d340 2024-03-22T13:06:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.336+0000 7f35ea400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e406dcf0 con 0x7f35e40ca370 2024-03-22T13:06:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.337+0000 7f35e9000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f35e40ca370 0x7f35e415e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.337+0000 7f35e9000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f35e40ca370 0x7f35e415e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54804/0 (socket says 172.21.15.6:54804) 2024-03-22T13:06:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.337+0000 7f35e9000640 1 -- 172.21.15.6:0/1891255475 learned_addr learned my addr 172.21.15.6:0/1891255475 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.338+0000 7f35e9a00640 1 --2- 172.21.15.6:0/1891255475 >> v2:172.21.15.6:3300/0 conn(0x7f35e406d340 0x7f35e406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.339+0000 7f35e9000640 1 -- 172.21.15.6:0/1891255475 >> v2:172.21.15.6:3300/0 conn(0x7f35e406d340 msgr2=0x7f35e406d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:06:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.339+0000 7f35e9000640 1 --2- 172.21.15.6:0/1891255475 >> v2:172.21.15.6:3300/0 conn(0x7f35e406d340 0x7f35e406d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.339+0000 7f35e9000640 1 -- 172.21.15.6:0/1891255475 >> v1:172.21.15.6:6789/0 conn(0x7f35e40ca750 legacy=0x7f35e40cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.339+0000 7f35e9000640 1 -- 172.21.15.6:0/1891255475 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e406c2c0 con 0x7f35e40ca370 2024-03-22T13:06:59.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.342+0000 7f35e9000640 1 --2- 172.21.15.6:0/1891255475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e415e290 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35e408cbc0 tx=0x7f35dc02bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e139ae119ec2feb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:59.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.345+0000 7f35e3400640 1 -- 172.21.15.6:0/1891255475 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc02cb20 con 0x7f35e40ca370 2024-03-22T13:06:59.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.347+0000 7f35e3400640 1 -- 172.21.15.6:0/1891255475 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc02ccc0 con 0x7f35e40ca370 2024-03-22T13:06:59.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.347+0000 7f35e3400640 1 -- 172.21.15.6:0/1891255475 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc034610 con 0x7f35e40ca370 2024-03-22T13:06:59.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 -- 172.21.15.6:0/1891255475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 msgr2=0x7f35e415e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:59.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 --2- 172.21.15.6:0/1891255475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e415e290 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35e408cbc0 tx=0x7f35dc02bf80 comp rx=0 tx=0).stop 2024-03-22T13:06:59.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 -- 172.21.15.6:0/1891255475 shutdown_connections 2024-03-22T13:06:59.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 --2- 172.21.15.6:0/1891255475 >> v2:172.21.15.6:3300/0 conn(0x7f35e406d340 0x7f35e406d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:59.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 --2- 172.21.15.6:0/1891255475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e415e290 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 -- 172.21.15.6:0/1891255475 >> 172.21.15.6:0/1891255475 conn(0x7f35e40cf9f0 msgr2=0x7f35e40cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:06:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 -- 172.21.15.6:0/1891255475 shutdown_connections 2024-03-22T13:06:59.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.348+0000 7f35ea400640 1 -- 172.21.15.6:0/1891255475 wait complete. 2024-03-22T13:06:59.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.351+0000 7f35ea400640 1 Processor -- start 2024-03-22T13:06:59.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.351+0000 7f35ea400640 1 -- start start 2024-03-22T13:06:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.352+0000 7f35ea400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35e406d340 0x7f35e41f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.352+0000 7f35ea400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e414b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.352+0000 7f35ea400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35e40ca750 0x7f35e4158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.352+0000 7f35ea400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e406c860 con 0x7f35e40ca750 2024-03-22T13:06:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.352+0000 7f35ea400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e406c580 con 0x7f35e40ca370 2024-03-22T13:06:59.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.353+0000 7f35ea400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e406cb40 con 0x7f35e406d340 2024-03-22T13:06:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.353+0000 7f35e3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e414b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.353+0000 7f35e9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35e40ca750 0x7f35e4158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.353+0000 7f35e9000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35e406d340 0x7f35e41f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.354+0000 7f35e3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e414b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54818/0 (socket says 172.21.15.6:54818) 2024-03-22T13:06:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.354+0000 7f35e9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35e40ca750 0x7f35e4158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41412/0 (socket says 172.21.15.6:41412) 2024-03-22T13:06:59.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.354+0000 7f35e3e00640 1 -- 172.21.15.6:0/3856773826 learned_addr learned my addr 172.21.15.6:0/3856773826 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:06:59.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.357+0000 7f35e9a00640 1 -- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35e406d340 msgr2=0x7f35e41f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:59.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.357+0000 7f35e9a00640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35e406d340 0x7f35e41f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:59.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.357+0000 7f35e9a00640 1 -- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 msgr2=0x7f35e414b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:06:59.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.357+0000 7f35e9a00640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e414b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:06:59.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.357+0000 7f35e9a00640 1 -- 172.21.15.6:0/3856773826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e415a430 con 0x7f35e40ca750 2024-03-22T13:06:59.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.357+0000 7f35e9000640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35e406d340 0x7f35e41f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:06:59.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.360+0000 7f35e9a00640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35e40ca750 0x7f35e4158cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35d400d4c0 tx=0x7f35d400d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:59.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.365+0000 7f35e0c00640 1 -- 172.21.15.6:0/3856773826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4004020 con 0x7f35e40ca750 2024-03-22T13:06:59.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.365+0000 7f35e0c00640 1 -- 172.21.15.6:0/3856773826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d40041e0 con 0x7f35e40ca750 2024-03-22T13:06:59.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.365+0000 7f35e0c00640 1 -- 172.21.15.6:0/3856773826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d40043d0 con 0x7f35e40ca750 2024-03-22T13:06:59.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.365+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e415a710 con 0x7f35e40ca750 2024-03-22T13:06:59.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.365+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e41584c0 con 0x7f35e40ca750 2024-03-22T13:06:59.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.366+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e40b1290 con 0x7f35e40ca750 2024-03-22T13:06:59.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.392+0000 7f35e0c00640 1 -- 172.21.15.6:0/3856773826 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35d401b020 con 0x7f35e40ca750 2024-03-22T13:06:59.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.393+0000 7f35e0c00640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35c006bc20 0x7f35c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:06:59.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.394+0000 7f35e9000640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35c006bc20 0x7f35c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:06:59.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.395+0000 7f35e9000640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35c006bc20 0x7f35c006e0e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f35dc035000 tx=0x7f35dc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:06:59.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.395+0000 7f35e0c00640 1 -- 172.21.15.6:0/3856773826 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f35d40575d0 con 0x7f35e40ca750 2024-03-22T13:06:59.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.716+0000 7f35e0c00640 1 -- 172.21.15.6:0/3856773826 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35e40b1290 con 0x7f35e40ca750 2024-03-22T13:06:59.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:06:59.998+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35e4075650 con 0x7f35e40ca750 2024-03-22T13:07:00.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.021+0000 7f35e0c00640 1 -- 172.21.15.6:0/3856773826 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f35d4059530 con 0x7f35e40ca750 2024-03-22T13:07:00.021 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:00.021 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:00.021 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.023+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35c006bc20 msgr2=0x7f35c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.023+0000 7f35ea400640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35c006bc20 0x7f35c006e0e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f35dc035000 tx=0x7f35dc030000 comp rx=0 tx=0).stop 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.023+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35e40ca750 msgr2=0x7f35e4158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.023+0000 7f35ea400640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35e40ca750 0x7f35e4158cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35d400d4c0 tx=0x7f35d400d980 comp rx=0 tx=0).stop 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 shutdown_connections 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35e40ca750 0x7f35e4158cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f35c006bc20 0x7f35c006e0e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35e40ca370 0x7f35e414b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 --2- 172.21.15.6:0/3856773826 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35e406d340 0x7f35e41f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 >> 172.21.15.6:0/3856773826 conn(0x7f35e40cf9f0 msgr2=0x7f35e415d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 shutdown_connections 2024-03-22T13:07:00.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:00.024+0000 7f35ea400640 1 -- 172.21.15.6:0/3856773826 wait complete. 2024-03-22T13:07:00.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:00.645 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:01.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:01.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.849+0000 7fad06000640 1 Processor -- start 2024-03-22T13:07:01.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.849+0000 7fad06000640 1 -- start start 2024-03-22T13:07:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.850+0000 7fad06000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.850+0000 7fad06000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fad0015b7a0 0x7fad000ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.850+0000 7fad06000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0006c440 con 0x7fad000cc640 2024-03-22T13:07:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.850+0000 7fad06000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0006d330 con 0x7fad0015b7a0 2024-03-22T13:07:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.850+0000 7fad06000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0006c720 con 0x7fad000cca20 2024-03-22T13:07:01.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.851+0000 7facffe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.851+0000 7facffe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36258/0 (socket says 172.21.15.6:36258) 2024-03-22T13:07:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.851+0000 7facffe00640 1 -- 172.21.15.6:0/3319595189 learned_addr learned my addr 172.21.15.6:0/3319595189 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:01.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.854+0000 7facffe00640 1 -- 172.21.15.6:0/3319595189 >> v2:172.21.15.92:3300/0 conn(0x7fad0015b7a0 msgr2=0x7fad000ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:01.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.854+0000 7facffe00640 1 --2- 172.21.15.6:0/3319595189 >> v2:172.21.15.92:3300/0 conn(0x7fad0015b7a0 0x7fad000ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:01.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.854+0000 7facffe00640 1 -- 172.21.15.6:0/3319595189 >> v1:172.21.15.6:6789/0 conn(0x7fad000cc640 legacy=0x7fad0006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:01.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.854+0000 7facffe00640 1 -- 172.21.15.6:0/3319595189 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0006ca00 con 0x7fad000cca20 2024-03-22T13:07:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.856+0000 7facffe00640 1 --2- 172.21.15.6:0/3319595189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015e260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a61d959b2823ebb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.860+0000 7facff400640 1 -- 172.21.15.6:0/3319595189 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec034070 con 0x7fad000cca20 2024-03-22T13:07:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.860+0000 7facff400640 1 -- 172.21.15.6:0/3319595189 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec030840 con 0x7fad000cca20 2024-03-22T13:07:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.860+0000 7facff400640 1 -- 172.21.15.6:0/3319595189 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec030c10 con 0x7fad000cca20 2024-03-22T13:07:01.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 -- 172.21.15.6:0/3319595189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 msgr2=0x7fad0015e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:01.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 --2- 172.21.15.6:0/3319595189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015e260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec02b980 comp rx=0 tx=0).stop 2024-03-22T13:07:01.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 -- 172.21.15.6:0/3319595189 shutdown_connections 2024-03-22T13:07:01.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 --2- 172.21.15.6:0/3319595189 >> v2:172.21.15.92:3300/0 conn(0x7fad0015b7a0 0x7fad000ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:01.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 --2- 172.21.15.6:0/3319595189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015e260 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:01.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 -- 172.21.15.6:0/3319595189 >> 172.21.15.6:0/3319595189 conn(0x7fad000cf9c0 msgr2=0x7fad000cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 -- 172.21.15.6:0/3319595189 shutdown_connections 2024-03-22T13:07:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.861+0000 7fad06000640 1 -- 172.21.15.6:0/3319595189 wait complete. 2024-03-22T13:07:01.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.864+0000 7fad06000640 1 Processor -- start 2024-03-22T13:07:01.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.864+0000 7fad06000640 1 -- start start 2024-03-22T13:07:01.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.865+0000 7fad06000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad000cc640 0x7fad000c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:01.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.866+0000 7fad06000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:01.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.866+0000 7fad06000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad0015b7a0 0x7fad001f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:01.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.866+0000 7fad06000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0015b000 con 0x7fad0015b7a0 2024-03-22T13:07:01.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.866+0000 7fad06000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0006cfc0 con 0x7fad000cc640 2024-03-22T13:07:01.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.866+0000 7fad06000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0015b180 con 0x7fad000cca20 2024-03-22T13:07:01.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.867+0000 7facffe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:01.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.867+0000 7facffe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36262/0 (socket says 172.21.15.6:36262) 2024-03-22T13:07:01.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.867+0000 7facffe00640 1 -- 172.21.15.6:0/2163262053 learned_addr learned my addr 172.21.15.6:0/2163262053 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:01.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.870+0000 7facffe00640 1 -- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad000cc640 msgr2=0x7fad000c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:01.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.870+0000 7facffe00640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad000cc640 0x7fad000c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:01.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.870+0000 7facffe00640 1 -- 172.21.15.6:0/2163262053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad0015b7a0 msgr2=0x7fad001f3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.870+0000 7facffe00640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad0015b7a0 0x7fad001f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.870+0000 7facffe00640 1 -- 172.21.15.6:0/2163262053 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad001e57d0 con 0x7fad000cca20 2024-03-22T13:07:01.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.873+0000 7facffe00640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015a280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.878+0000 7facfcc00640 1 -- 172.21.15.6:0/2163262053 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec048070 con 0x7fad000cca20 2024-03-22T13:07:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.878+0000 7facfcc00640 1 -- 172.21.15.6:0/2163262053 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec030640 con 0x7fad000cca20 2024-03-22T13:07:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.878+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad001f03a0 con 0x7fad000cca20 2024-03-22T13:07:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.878+0000 7facfcc00640 1 -- 172.21.15.6:0/2163262053 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec034040 con 0x7fad000cca20 2024-03-22T13:07:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.878+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad001f0960 con 0x7fad000cca20 2024-03-22T13:07:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.880+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad00075650 con 0x7fad000cca20 2024-03-22T13:07:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.901+0000 7facfcc00640 1 -- 172.21.15.6:0/2163262053 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7facec04b070 con 0x7fad000cca20 2024-03-22T13:07:01.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.903+0000 7facfcc00640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7face006bc50 0x7face006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:01.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.903+0000 7fad04c00640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7face006bc50 0x7face006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.904+0000 7fad04c00640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7face006bc50 0x7face006e110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7facf4001560 tx=0x7facf4007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:01.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:01.905+0000 7facfcc00640 1 -- 172.21.15.6:0/2163262053 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7facec0307f0 con 0x7fad000cca20 2024-03-22T13:07:02.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.177+0000 7facfcc00640 1 -- 172.21.15.6:0/2163262053 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7facec052020 con 0x7fad000cca20 2024-03-22T13:07:02.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:07:02.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:07:02.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.430+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad000af100 con 0x7fad000cca20 2024-03-22T13:07:02.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.443+0000 7facfcc00640 1 -- 172.21.15.6:0/2163262053 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7facec07bcf0 con 0x7fad000cca20 2024-03-22T13:07:02.444 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:02.444 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:02.444 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:02.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7face006bc50 msgr2=0x7face006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:02.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7face006bc50 0x7face006e110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7facf4001560 tx=0x7facf4007990 comp rx=0 tx=0).stop 2024-03-22T13:07:02.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 msgr2=0x7fad0015a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:02.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015a280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec004730 comp rx=0 tx=0).stop 2024-03-22T13:07:02.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 shutdown_connections 2024-03-22T13:07:02.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad0015b7a0 0x7fad001f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:02.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad000cca20 0x7fad0015a280 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:02.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7face006bc50 0x7face006e110 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:02.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 --2- 172.21.15.6:0/2163262053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad000cc640 0x7fad000c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:02.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 >> 172.21.15.6:0/2163262053 conn(0x7fad000cf9c0 msgr2=0x7fad0015d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:02.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 shutdown_connections 2024-03-22T13:07:02.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:02.446+0000 7fad06000640 1 -- 172.21.15.6:0/2163262053 wait complete. 2024-03-22T13:07:03.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:03.066 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:04.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.315+0000 7f7fde000640 1 Processor -- start 2024-03-22T13:07:04.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.315+0000 7f7fde000640 1 -- start start 2024-03-22T13:07:04.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.317+0000 7f7fde000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:04.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.317+0000 7f7fde000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd80ce230 con 0x7f7fd80cc3b0 2024-03-22T13:07:04.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.317+0000 7f7fde000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd80ce510 con 0x7f7fd80cc790 2024-03-22T13:07:04.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.317+0000 7f7fde000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80cd0f0 con 0x7f7fd815b250 2024-03-22T13:07:04.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.317+0000 7f7fd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:04.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.318+0000 7f7fd6a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7fd80cc790 0x7f7fd815ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:33148/0 (socket says 172.21.15.6:33148) 2024-03-22T13:07:04.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.318+0000 7f7fd6a00640 1 -- 172.21.15.6:0/3503195467 learned_addr learned my addr 172.21.15.6:0/3503195467 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:04.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.318+0000 7f7fd7400640 1 --1- 172.21.15.6:0/3503195467 >> v1:172.21.15.6:6789/0 conn(0x7f7fd80cc3b0 0x7f7fd80cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54028/0 (socket says 172.21.15.6:54028) 2024-03-22T13:07:04.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.321+0000 7f7fd6000640 1 -- 172.21.15.6:0/3503195467 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105974794 0 0) 0x7f7fd80ce510 con 0x7f7fd80cc790 2024-03-22T13:07:04.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.321+0000 7f7fd6000640 1 -- 172.21.15.6:0/3503195467 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fbc003520 con 0x7f7fd80cc790 2024-03-22T13:07:04.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.321+0000 7f7fd6000640 1 -- 172.21.15.6:0/3503195467 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502968327 0 0) 0x7f7fd80ce230 con 0x7f7fd80cc3b0 2024-03-22T13:07:04.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.321+0000 7f7fd6000640 1 -- 172.21.15.6:0/3503195467 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd80ce510 con 0x7f7fd80cc3b0 2024-03-22T13:07:04.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.321+0000 7f7fd7e00640 1 -- 172.21.15.6:0/3503195467 >> v1:172.21.15.92:6789/0 conn(0x7f7fd80cc790 legacy=0x7f7fd815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:04.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.321+0000 7f7fd7e00640 1 -- 172.21.15.6:0/3503195467 >> v1:172.21.15.6:6789/0 conn(0x7f7fd80cc3b0 legacy=0x7f7fd80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:04.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.321+0000 7f7fd7e00640 1 -- 172.21.15.6:0/3503195467 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd80ce7f0 con 0x7f7fd815b250 2024-03-22T13:07:04.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.324+0000 7f7fd7e00640 1 --2- 172.21.15.6:0/3503195467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80ccbb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009dc0 tx=0x7f7fd002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f871c642095b27c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.329+0000 7f7fd6000640 1 -- 172.21.15.6:0/3503195467 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0039070 con 0x7f7fd815b250 2024-03-22T13:07:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.329+0000 7f7fd6000640 1 -- 172.21.15.6:0/3503195467 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd002cc40 con 0x7f7fd815b250 2024-03-22T13:07:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.329+0000 7f7fd6000640 1 -- 172.21.15.6:0/3503195467 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0034440 con 0x7f7fd815b250 2024-03-22T13:07:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.330+0000 7f7fde000640 1 -- 172.21.15.6:0/3503195467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 msgr2=0x7f7fd80ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.330+0000 7f7fde000640 1 --2- 172.21.15.6:0/3503195467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80ccbb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009dc0 tx=0x7f7fd002be90 comp rx=0 tx=0).stop 2024-03-22T13:07:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.330+0000 7f7fde000640 1 -- 172.21.15.6:0/3503195467 shutdown_connections 2024-03-22T13:07:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.330+0000 7f7fde000640 1 --2- 172.21.15.6:0/3503195467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80ccbb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.330+0000 7f7fde000640 1 -- 172.21.15.6:0/3503195467 >> 172.21.15.6:0/3503195467 conn(0x7f7fd80b16d0 msgr2=0x7f7fd80b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.330+0000 7f7fde000640 1 -- 172.21.15.6:0/3503195467 shutdown_connections 2024-03-22T13:07:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.330+0000 7f7fde000640 1 -- 172.21.15.6:0/3503195467 wait complete. 2024-03-22T13:07:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.332+0000 7f7fde000640 1 Processor -- start 2024-03-22T13:07:04.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.332+0000 7f7fde000640 1 -- start start 2024-03-22T13:07:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.333+0000 7f7fde000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd80cc3b0 0x7f7fd81f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.333+0000 7f7fde000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fd80cc790 0x7f7fd80c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.333+0000 7f7fde000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.333+0000 7f7fde000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd806dc10 con 0x7f7fd80cc3b0 2024-03-22T13:07:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.333+0000 7f7fde000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80cedb0 con 0x7f7fd80cc790 2024-03-22T13:07:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.333+0000 7f7fde000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80cf940 con 0x7f7fd815b250 2024-03-22T13:07:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.334+0000 7f7fd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.334+0000 7f7fd6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fd80cc790 0x7f7fd80c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.334+0000 7f7fd7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd80cc3b0 0x7f7fd81f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.334+0000 7f7fd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50912/0 (socket says 172.21.15.6:50912) 2024-03-22T13:07:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.334+0000 7f7fd7e00640 1 -- 172.21.15.6:0/2669443181 learned_addr learned my addr 172.21.15.6:0/2669443181 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.339+0000 7f7fd7e00640 1 -- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fd80cc790 msgr2=0x7f7fd80c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.339+0000 7f7fd7e00640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fd80cc790 0x7f7fd80c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.339+0000 7f7fd7e00640 1 -- 172.21.15.6:0/2669443181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd80cc3b0 msgr2=0x7f7fd81f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.339+0000 7f7fd7e00640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd80cc3b0 0x7f7fd81f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.339+0000 7f7fd7e00640 1 -- 172.21.15.6:0/2669443181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd81f0200 con 0x7f7fd815b250 2024-03-22T13:07:04.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.341+0000 7f7fd7e00640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80c8ed0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009ee0 tx=0x7f7fd0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:04.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.346+0000 7f7fcf400640 1 -- 172.21.15.6:0/2669443181 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0044020 con 0x7f7fd815b250 2024-03-22T13:07:04.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.346+0000 7f7fcf400640 1 -- 172.21.15.6:0/2669443181 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0035bc0 con 0x7f7fd815b250 2024-03-22T13:07:04.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.346+0000 7f7fcf400640 1 -- 172.21.15.6:0/2669443181 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0039040 con 0x7f7fd815b250 2024-03-22T13:07:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.346+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd81f04e0 con 0x7f7fd815b250 2024-03-22T13:07:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.346+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd81f0aa0 con 0x7f7fd815b250 2024-03-22T13:07:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.347+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd81f0dd0 con 0x7f7fd815b250 2024-03-22T13:07:04.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.623+0000 7f7fcf400640 1 -- 172.21.15.6:0/2669443181 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7fd0047020 con 0x7f7fd815b250 2024-03-22T13:07:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.626+0000 7f7fcf400640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7fb806bc20 0x7f7fb806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:04.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.626+0000 7f7fd7400640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7fb806bc20 0x7f7fb806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:04.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.627+0000 7f7fd7400640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7fb806bc20 0x7f7fb806e0e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8000f60 tx=0x7f7fc8007cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:04.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.629+0000 7f7fcf400640 1 -- 172.21.15.6:0/2669443181 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f7fd0079fe0 con 0x7f7fd815b250 2024-03-22T13:07:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.645+0000 7f7fcf400640 1 -- 172.21.15.6:0/2669443181 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7fd004e020 con 0x7f7fd815b250 2024-03-22T13:07:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.901+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fd80756c0 con 0x7f7fd815b250 2024-03-22T13:07:04.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.924+0000 7f7fcf400640 1 -- 172.21.15.6:0/2669443181 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f7fd007bf40 con 0x7f7fd815b250 2024-03-22T13:07:04.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:04.925 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:04.925 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7fb806bc20 msgr2=0x7f7fb806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7fb806bc20 0x7f7fb806e0e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8000f60 tx=0x7f7fc8007cf0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 msgr2=0x7f7fd80c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80c8ed0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009ee0 tx=0x7f7fd0004a40 comp rx=0 tx=0).stop 2024-03-22T13:07:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 shutdown_connections 2024-03-22T13:07:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fd815b250 0x7f7fd80c8ed0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7fb806bc20 0x7f7fb806e0e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fd80cc790 0x7f7fd80c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 --2- 172.21.15.6:0/2669443181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fd80cc3b0 0x7f7fd81f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:04.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 >> 172.21.15.6:0/2669443181 conn(0x7f7fd80b16d0 msgr2=0x7f7fd815ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:04.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 shutdown_connections 2024-03-22T13:07:04.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:04.928+0000 7f7fde000640 1 -- 172.21.15.6:0/2669443181 wait complete. 2024-03-22T13:07:05.548 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:05.548 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:06.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:06.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.754+0000 7f12d0400640 1 Processor -- start 2024-03-22T13:07:06.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.755+0000 7f12d0400640 1 -- start start 2024-03-22T13:07:06.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.756+0000 7f12d0400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:06.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.756+0000 7f12d0400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c80c4740 con 0x7f12c80c68c0 2024-03-22T13:07:06.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.756+0000 7f12d0400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c8154620 con 0x7f12c8154240 2024-03-22T13:07:06.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.756+0000 7f12d0400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c815e130 con 0x7f12c80c7ee0 2024-03-22T13:07:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.757+0000 7f12ce400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12c80c68c0 0x7f12c806b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54030/0 (socket says 172.21.15.6:54030) 2024-03-22T13:07:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.757+0000 7f12ce400640 1 -- 172.21.15.6:0/2885714501 learned_addr learned my addr 172.21.15.6:0/2885714501 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:06.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.757+0000 7f12cd000640 1 --2- 172.21.15.6:0/2885714501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:06.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.760+0000 7f12c7e00640 1 -- 172.21.15.6:0/2885714501 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950288910 0 0) 0x7f12c80c4740 con 0x7f12c80c68c0 2024-03-22T13:07:06.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.760+0000 7f12c7e00640 1 -- 172.21.15.6:0/2885714501 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b0003520 con 0x7f12c80c68c0 2024-03-22T13:07:06.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.760+0000 7f12c7e00640 1 -- 172.21.15.6:0/2885714501 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891059953 0 0) 0x7f12c8154620 con 0x7f12c8154240 2024-03-22T13:07:06.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.760+0000 7f12c7e00640 1 -- 172.21.15.6:0/2885714501 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c80c4740 con 0x7f12c8154240 2024-03-22T13:07:06.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.761+0000 7f12cd000640 1 -- 172.21.15.6:0/2885714501 >> v1:172.21.15.92:6789/0 conn(0x7f12c8154240 legacy=0x7f12c80c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:06.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.761+0000 7f12cd000640 1 -- 172.21.15.6:0/2885714501 >> v1:172.21.15.6:6789/0 conn(0x7f12c80c68c0 legacy=0x7f12c806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.761+0000 7f12cd000640 1 -- 172.21.15.6:0/2885714501 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80c6e00 con 0x7f12c80c7ee0 2024-03-22T13:07:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.764+0000 7f12cd000640 1 --2- 172.21.15.6:0/2885714501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c815ac90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f12b40097d0 tx=0x7f12b402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57476d02c76f6bfe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:06.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.766+0000 7f12c7e00640 1 -- 172.21.15.6:0/2885714501 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4034070 con 0x7f12c80c7ee0 2024-03-22T13:07:06.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.769+0000 7f12c7e00640 1 -- 172.21.15.6:0/2885714501 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4030880 con 0x7f12c80c7ee0 2024-03-22T13:07:06.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.769+0000 7f12c7e00640 1 -- 172.21.15.6:0/2885714501 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4030c30 con 0x7f12c80c7ee0 2024-03-22T13:07:06.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.769+0000 7f12d0400640 1 -- 172.21.15.6:0/2885714501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 msgr2=0x7f12c815ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:06.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.769+0000 7f12d0400640 1 --2- 172.21.15.6:0/2885714501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c815ac90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f12b40097d0 tx=0x7f12b402b9c0 comp rx=0 tx=0).stop 2024-03-22T13:07:06.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.770+0000 7f12d0400640 1 -- 172.21.15.6:0/2885714501 shutdown_connections 2024-03-22T13:07:06.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.770+0000 7f12d0400640 1 --2- 172.21.15.6:0/2885714501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c815ac90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:06.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.770+0000 7f12d0400640 1 -- 172.21.15.6:0/2885714501 >> 172.21.15.6:0/2885714501 conn(0x7f12c80cb930 msgr2=0x7f12c80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:06.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.770+0000 7f12d0400640 1 -- 172.21.15.6:0/2885714501 shutdown_connections 2024-03-22T13:07:06.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.770+0000 7f12d0400640 1 -- 172.21.15.6:0/2885714501 wait complete. 2024-03-22T13:07:06.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.771+0000 7f12d0400640 1 Processor -- start 2024-03-22T13:07:06.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.772+0000 7f12d0400640 1 -- start start 2024-03-22T13:07:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.772+0000 7f12d0400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12c80c68c0 0x7f12c81f0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.772+0000 7f12d0400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c81f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.772+0000 7f12d0400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c8154240 0x7f12c8148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.772+0000 7f12d0400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c815dae0 con 0x7f12c80c68c0 2024-03-22T13:07:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.772+0000 7f12d0400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c815d800 con 0x7f12c8154240 2024-03-22T13:07:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.772+0000 7f12d0400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c815ddc0 con 0x7f12c80c7ee0 2024-03-22T13:07:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.773+0000 7f12cd000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c81f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.773+0000 7f12cda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12c80c68c0 0x7f12c81f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.773+0000 7f12ce400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c8154240 0x7f12c8148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.773+0000 7f12cd000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c81f36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50934/0 (socket says 172.21.15.6:50934) 2024-03-22T13:07:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.773+0000 7f12cd000640 1 -- 172.21.15.6:0/4202420851 learned_addr learned my addr 172.21.15.6:0/4202420851 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.777+0000 7f12cd000640 1 -- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c8154240 msgr2=0x7f12c8148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.777+0000 7f12cd000640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c8154240 0x7f12c8148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.777+0000 7f12cd000640 1 -- 172.21.15.6:0/4202420851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12c80c68c0 msgr2=0x7f12c81f0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.777+0000 7f12cd000640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12c80c68c0 0x7f12c81f0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.777+0000 7f12cd000640 1 -- 172.21.15.6:0/4202420851 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c81f3d40 con 0x7f12c80c7ee0 2024-03-22T13:07:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.780+0000 7f12cd000640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c81f36b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f12b402ba00 tx=0x7f12b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.782+0000 7f12c5600640 1 -- 172.21.15.6:0/4202420851 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4034050 con 0x7f12c80c7ee0 2024-03-22T13:07:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.782+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c81f4000 con 0x7f12c80c7ee0 2024-03-22T13:07:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.783+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c81f45c0 con 0x7f12c80c7ee0 2024-03-22T13:07:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.784+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c815e5c0 con 0x7f12c80c7ee0 2024-03-22T13:07:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.784+0000 7f12c5600640 1 -- 172.21.15.6:0/4202420851 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4030680 con 0x7f12c80c7ee0 2024-03-22T13:07:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:06.784+0000 7f12c5600640 1 -- 172.21.15.6:0/4202420851 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4005990 con 0x7f12c80c7ee0 2024-03-22T13:07:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.088+0000 7f12c5600640 1 -- 172.21.15.6:0/4202420851 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f12b4049060 con 0x7f12c80c7ee0 2024-03-22T13:07:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.090+0000 7f12c5600640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12a806bbd0 0x7f12a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.090+0000 7f12cda00640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12a806bbd0 0x7f12a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:07.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.091+0000 7f12cda00640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12a806bbd0 0x7f12a806e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0011e0 tx=0x7f12bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.094+0000 7f12c5600640 1 -- 172.21.15.6:0/4202420851 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f12b4078c50 con 0x7f12c80c7ee0 2024-03-22T13:07:07.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.111+0000 7f12c5600640 1 -- 172.21.15.6:0/4202420851 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f12c815e5c0 con 0x7f12c80c7ee0 2024-03-22T13:07:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.375+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12c80756c0 con 0x7f12c80c7ee0 2024-03-22T13:07:07.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.390+0000 7f12c5600640 1 -- 172.21.15.6:0/4202420851 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f12b407abb0 con 0x7f12c80c7ee0 2024-03-22T13:07:07.390 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:07.390 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:07.390 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.393+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12a806bbd0 msgr2=0x7f12a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.393+0000 7f12d0400640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12a806bbd0 0x7f12a806e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0011e0 tx=0x7f12bc009040 comp rx=0 tx=0).stop 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.393+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 msgr2=0x7f12c81f36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.393+0000 7f12d0400640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c81f36b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f12b402ba00 tx=0x7f12b40046c0 comp rx=0 tx=0).stop 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 shutdown_connections 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12a806bbd0 0x7f12a806e090 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c8154240 0x7f12c8148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c80c7ee0 0x7f12c81f36b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 --2- 172.21.15.6:0/4202420851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12c80c68c0 0x7f12c81f0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 >> 172.21.15.6:0/4202420851 conn(0x7f12c80cb930 msgr2=0x7f12c8153130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 shutdown_connections 2024-03-22T13:07:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:07.394+0000 7f12d0400640 1 -- 172.21.15.6:0/4202420851 wait complete. 2024-03-22T13:07:08.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:08.023 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:09.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.213+0000 7f0a11c00640 1 Processor -- start 2024-03-22T13:07:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.213+0000 7f0a11c00640 1 -- start start 2024-03-22T13:07:09.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.215+0000 7f0a11c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0ce8a0 0x7f0a0c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:09.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.215+0000 7f0a11c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a0c15ae30 0x7f0a0c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:09.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.215+0000 7f0a11c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c15b260 con 0x7f0a0c0cc380 2024-03-22T13:07:09.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.215+0000 7f0a11c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0c0cee00 con 0x7f0a0c15ae30 2024-03-22T13:07:09.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.215+0000 7f0a11c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0cf0c0 con 0x7f0a0c0ce8a0 2024-03-22T13:07:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.216+0000 7f0a03400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0ce8a0 0x7f0a0c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.216+0000 7f0a0be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0a0c0cc380 0x7f0a0c0cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:33168/0 (socket says 172.21.15.6:33168) 2024-03-22T13:07:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.216+0000 7f0a0be00640 1 -- 172.21.15.6:0/1023694788 learned_addr learned my addr 172.21.15.6:0/1023694788 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.216+0000 7f0a11200640 1 --2- 172.21.15.6:0/1023694788 >> v2:172.21.15.6:3300/0 conn(0x7f0a0c15ae30 0x7f0a0c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.219+0000 7f0a11200640 1 -- 172.21.15.6:0/1023694788 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0ce8a0 msgr2=0x7f0a0c06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.219+0000 7f0a11200640 1 --2- 172.21.15.6:0/1023694788 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0ce8a0 0x7f0a0c06c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.219+0000 7f0a11200640 1 -- 172.21.15.6:0/1023694788 >> v1:172.21.15.92:6789/0 conn(0x7f0a0c0cc380 legacy=0x7f0a0c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.219+0000 7f0a11200640 1 -- 172.21.15.6:0/1023694788 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c0cf3a0 con 0x7f0a0c15ae30 2024-03-22T13:07:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.221+0000 7f0a11200640 1 --2- 172.21.15.6:0/1023694788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c0cc760 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009d90 tx=0x7f0a0402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b20f4dd14e9f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.226+0000 7f0a0b400640 1 -- 172.21.15.6:0/1023694788 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04039070 con 0x7f0a0c15ae30 2024-03-22T13:07:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.226+0000 7f0a0b400640 1 -- 172.21.15.6:0/1023694788 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0402cc50 con 0x7f0a0c15ae30 2024-03-22T13:07:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.226+0000 7f0a0b400640 1 -- 172.21.15.6:0/1023694788 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04034440 con 0x7f0a0c15ae30 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 -- 172.21.15.6:0/1023694788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 msgr2=0x7f0a0c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1023694788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c0cc760 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009d90 tx=0x7f0a0402bea0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 -- 172.21.15.6:0/1023694788 shutdown_connections 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1023694788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c0cc760 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1023694788 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0ce8a0 0x7f0a0c06c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 -- 172.21.15.6:0/1023694788 >> 172.21.15.6:0/1023694788 conn(0x7f0a0c0b1480 msgr2=0x7f0a0c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 -- 172.21.15.6:0/1023694788 shutdown_connections 2024-03-22T13:07:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.227+0000 7f0a11c00640 1 -- 172.21.15.6:0/1023694788 wait complete. 2024-03-22T13:07:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.229+0000 7f0a11c00640 1 Processor -- start 2024-03-22T13:07:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.229+0000 7f0a11c00640 1 -- start start 2024-03-22T13:07:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.230+0000 7f0a11c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0cc380 0x7f0a0c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.230+0000 7f0a11c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a0c0ce8a0 0x7f0a0c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.230+0000 7f0a11c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.230+0000 7f0a11c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c15a910 con 0x7f0a0c15ae30 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.230+0000 7f0a11c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c15a630 con 0x7f0a0c0ce8a0 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.230+0000 7f0a11c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c1549d0 con 0x7f0a0c0cc380 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.231+0000 7f0a03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a0c0ce8a0 0x7f0a0c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.231+0000 7f0a0be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0cc380 0x7f0a0c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.231+0000 7f0a11200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.231+0000 7f0a11200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c15d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44598/0 (socket says 172.21.15.6:44598) 2024-03-22T13:07:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.231+0000 7f0a11200640 1 -- 172.21.15.6:0/1775709094 learned_addr learned my addr 172.21.15.6:0/1775709094 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.235+0000 7f0a11200640 1 -- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0cc380 msgr2=0x7f0a0c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.235+0000 7f0a11200640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0cc380 0x7f0a0c0b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.235+0000 7f0a11200640 1 -- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a0c0ce8a0 msgr2=0x7f0a0c0b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.235+0000 7f0a11200640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a0c0ce8a0 0x7f0a0c0b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.235+0000 7f0a11200640 1 -- 172.21.15.6:0/1775709094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c15be70 con 0x7f0a0c15ae30 2024-03-22T13:07:09.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.238+0000 7f0a11200640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c15d000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009d90 tx=0x7f0a04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.242+0000 7f0a08c00640 1 -- 172.21.15.6:0/1775709094 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04039050 con 0x7f0a0c15ae30 2024-03-22T13:07:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.242+0000 7f0a08c00640 1 -- 172.21.15.6:0/1775709094 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a04045070 con 0x7f0a0c15ae30 2024-03-22T13:07:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.242+0000 7f0a08c00640 1 -- 172.21.15.6:0/1775709094 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04009450 con 0x7f0a0c15ae30 2024-03-22T13:07:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.242+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0c14fdb0 con 0x7f0a0c15ae30 2024-03-22T13:07:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.242+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0c150370 con 0x7f0a0c15ae30 2024-03-22T13:07:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.244+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0c075650 con 0x7f0a0c15ae30 2024-03-22T13:07:09.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.559+0000 7f0a08c00640 1 -- 172.21.15.6:0/1775709094 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0a0403e610 con 0x7f0a0c15ae30 2024-03-22T13:07:09.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.560+0000 7f0a08c00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f09e806bbd0 0x7f09e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:09.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.561+0000 7f0a0be00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f09e806bbd0 0x7f09e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:09.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.562+0000 7f0a0be00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f09e806bbd0 0x7f09e806e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f09fc005880 tx=0x7f09fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:09.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.609+0000 7f0a08c00640 1 -- 172.21.15.6:0/1775709094 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0a0407a3b0 con 0x7f0a0c15ae30 2024-03-22T13:07:09.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.610+0000 7f0a08c00640 1 -- 172.21.15.6:0/1775709094 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0a0407c310 con 0x7f0a0c15ae30 2024-03-22T13:07:09.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.882+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a0c0c9d30 con 0x7f0a0c15ae30 2024-03-22T13:07:09.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.898+0000 7f0a08c00640 1 -- 172.21.15.6:0/1775709094 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f0a040803b0 con 0x7f0a0c15ae30 2024-03-22T13:07:09.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:09.898 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:09.898 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f09e806bbd0 msgr2=0x7f09e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f09e806bbd0 0x7f09e806e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f09fc005880 tx=0x7f09fc009040 comp rx=0 tx=0).stop 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 msgr2=0x7f0a0c15d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c15d000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009d90 tx=0x7f0a04004a40 comp rx=0 tx=0).stop 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 shutdown_connections 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a0c15ae30 0x7f0a0c15d000 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f09e806bbd0 0x7f09e806e090 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a0c0ce8a0 0x7f0a0c0b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 --2- 172.21.15.6:0/1775709094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a0c0cc380 0x7f0a0c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:09.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 >> 172.21.15.6:0/1775709094 conn(0x7f0a0c0b1480 msgr2=0x7f0a0c0cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:09.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 shutdown_connections 2024-03-22T13:07:09.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:09.901+0000 7f0a11c00640 1 -- 172.21.15.6:0/1775709094 wait complete. 2024-03-22T13:07:10.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:10.522 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:11.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:11.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.712+0000 7f6e8f200640 1 Processor -- start 2024-03-22T13:07:11.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.712+0000 7f6e8f200640 1 -- start start 2024-03-22T13:07:11.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.714+0000 7f6e8f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e900c55f0 0x7f6e90158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:11.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.714+0000 7f6e8f200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6e9006ede0 0x7f6e9006f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:11.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.714+0000 7f6e8f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9006e0f0 con 0x7f6e900c5210 2024-03-22T13:07:11.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.714+0000 7f6e8f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e900c6f90 con 0x7f6e900c55f0 2024-03-22T13:07:11.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.714+0000 7f6e8f200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9006d350 con 0x7f6e9006ede0 2024-03-22T13:07:11.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.715+0000 7f6e8e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6e9006ede0 0x7f6e9006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.715+0000 7f6e8e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6e9006ede0 0x7f6e9006f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40960/0 (socket says 172.21.15.6:40960) 2024-03-22T13:07:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.715+0000 7f6e8e800640 1 -- 172.21.15.6:0/4031204566 learned_addr learned my addr 172.21.15.6:0/4031204566 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.715+0000 7f6e8d400640 1 --2- 172.21.15.6:0/4031204566 >> v2:172.21.15.6:3300/0 conn(0x7f6e900c55f0 0x7f6e90158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.719+0000 7f6e8e800640 1 -- 172.21.15.6:0/4031204566 >> v2:172.21.15.6:3300/0 conn(0x7f6e900c55f0 msgr2=0x7f6e90158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.719+0000 7f6e8e800640 1 --2- 172.21.15.6:0/4031204566 >> v2:172.21.15.6:3300/0 conn(0x7f6e900c55f0 0x7f6e90158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.719+0000 7f6e8e800640 1 -- 172.21.15.6:0/4031204566 >> v1:172.21.15.6:6789/0 conn(0x7f6e900c5210 legacy=0x7f6e900c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.719+0000 7f6e8e800640 1 -- 172.21.15.6:0/4031204566 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900c6cb0 con 0x7f6e9006ede0 2024-03-22T13:07:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.722+0000 7f6e8e800640 1 --2- 172.21.15.6:0/4031204566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e9006f1c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6e901525b0 tx=0x7f6e8802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c49574434ddb328 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:11.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.725+0000 7f6e8ca00640 1 -- 172.21.15.6:0/4031204566 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8802ca80 con 0x7f6e9006ede0 2024-03-22T13:07:11.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.728+0000 7f6e8ca00640 1 -- 172.21.15.6:0/4031204566 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8802cc20 con 0x7f6e9006ede0 2024-03-22T13:07:11.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.728+0000 7f6e8ca00640 1 -- 172.21.15.6:0/4031204566 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880343f0 con 0x7f6e9006ede0 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.728+0000 7f6e8f200640 1 -- 172.21.15.6:0/4031204566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 msgr2=0x7f6e9006f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.728+0000 7f6e8f200640 1 --2- 172.21.15.6:0/4031204566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e9006f1c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6e901525b0 tx=0x7f6e8802bee0 comp rx=0 tx=0).stop 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.729+0000 7f6e8f200640 1 -- 172.21.15.6:0/4031204566 shutdown_connections 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.729+0000 7f6e8f200640 1 --2- 172.21.15.6:0/4031204566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e9006f1c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.729+0000 7f6e8f200640 1 --2- 172.21.15.6:0/4031204566 >> v2:172.21.15.6:3300/0 conn(0x7f6e900c55f0 0x7f6e90158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.729+0000 7f6e8f200640 1 -- 172.21.15.6:0/4031204566 >> 172.21.15.6:0/4031204566 conn(0x7f6e900cb890 msgr2=0x7f6e900cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.729+0000 7f6e8f200640 1 -- 172.21.15.6:0/4031204566 shutdown_connections 2024-03-22T13:07:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.729+0000 7f6e8f200640 1 -- 172.21.15.6:0/4031204566 wait complete. 2024-03-22T13:07:11.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.731+0000 7f6e8f200640 1 Processor -- start 2024-03-22T13:07:11.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.731+0000 7f6e8f200640 1 -- start start 2024-03-22T13:07:11.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.732+0000 7f6e8f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e90156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:11.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.732+0000 7f6e8f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e900c5210 0x7f6e9015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:11.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.733+0000 7f6e8f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e900c55f0 0x7f6e901491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:11.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.733+0000 7f6e8f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9006dbd0 con 0x7f6e900c55f0 2024-03-22T13:07:11.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.733+0000 7f6e8f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9006d8f0 con 0x7f6e9006ede0 2024-03-22T13:07:11.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.733+0000 7f6e8f200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900c69a0 con 0x7f6e900c5210 2024-03-22T13:07:11.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.734+0000 7f6e8de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e90156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:11.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.734+0000 7f6e8d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e900c5210 0x7f6e9015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:11.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.734+0000 7f6e8e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e900c55f0 0x7f6e901491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:11.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.734+0000 7f6e8de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e90156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:40962/0 (socket says 172.21.15.6:40962) 2024-03-22T13:07:11.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.734+0000 7f6e8de00640 1 -- 172.21.15.6:0/1368552316 learned_addr learned my addr 172.21.15.6:0/1368552316 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:11.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.737+0000 7f6e8de00640 1 -- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e900c5210 msgr2=0x7f6e9015b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:11.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.737+0000 7f6e8de00640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e900c5210 0x7f6e9015b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:11.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.737+0000 7f6e8de00640 1 -- 172.21.15.6:0/1368552316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e900c55f0 msgr2=0x7f6e901491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:11.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.737+0000 7f6e8de00640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e900c55f0 0x7f6e901491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:11.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.737+0000 7f6e8de00640 1 -- 172.21.15.6:0/1368552316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900cb540 con 0x7f6e9006ede0 2024-03-22T13:07:11.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.740+0000 7f6e8de00640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e90156030 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000c400 tx=0x7f6e8000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:11.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.745+0000 7f6e86000640 1 -- 172.21.15.6:0/1368552316 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e800052d0 con 0x7f6e9006ede0 2024-03-22T13:07:11.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.745+0000 7f6e86000640 1 -- 172.21.15.6:0/1368552316 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e80005490 con 0x7f6e9006ede0 2024-03-22T13:07:11.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.745+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9015a5d0 con 0x7f6e9006ede0 2024-03-22T13:07:11.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.745+0000 7f6e86000640 1 -- 172.21.15.6:0/1368552316 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80007260 con 0x7f6e9006ede0 2024-03-22T13:07:11.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.745+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9015ab90 con 0x7f6e9006ede0 2024-03-22T13:07:11.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:11.746+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e900c7710 con 0x7f6e9006ede0 2024-03-22T13:07:12.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.032+0000 7f6e86000640 1 -- 172.21.15.6:0/1368552316 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e80007400 con 0x7f6e9006ede0 2024-03-22T13:07:12.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.034+0000 7f6e86000640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e7006bc50 0x7f6e7006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:12.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.035+0000 7f6e8d400640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e7006bc50 0x7f6e7006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:12.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.036+0000 7f6e8d400640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e7006bc50 0x7f6e7006e110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6e90155820 tx=0x7f6e7c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:12.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.036+0000 7f6e86000640 1 -- 172.21.15.6:0/1368552316 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6e80056cf0 con 0x7f6e9006ede0 2024-03-22T13:07:12.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.055+0000 7f6e86000640 1 -- 172.21.15.6:0/1368552316 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e900c7710 con 0x7f6e9006ede0 2024-03-22T13:07:12.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.318+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e90075650 con 0x7f6e9006ede0 2024-03-22T13:07:12.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.371+0000 7f6e86000640 1 -- 172.21.15.6:0/1368552316 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f6e900cb540 con 0x7f6e9006ede0 2024-03-22T13:07:12.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:12.371 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:12.371 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.373+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e7006bc50 msgr2=0x7f6e7006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.373+0000 7f6e8f200640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e7006bc50 0x7f6e7006e110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6e90155820 tx=0x7f6e7c009190 comp rx=0 tx=0).stop 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.373+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 msgr2=0x7f6e90156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.373+0000 7f6e8f200640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e90156030 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000c400 tx=0x7f6e8000c8c0 comp rx=0 tx=0).stop 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 shutdown_connections 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e900c55f0 0x7f6e901491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e900c5210 0x7f6e9015b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6e7006bc50 0x7f6e7006e110 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 --2- 172.21.15.6:0/1368552316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9006ede0 0x7f6e90156030 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 >> 172.21.15.6:0/1368552316 conn(0x7f6e900cb890 msgr2=0x7f6e9014bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 shutdown_connections 2024-03-22T13:07:12.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:12.374+0000 7f6e8f200640 1 -- 172.21.15.6:0/1368552316 wait complete. 2024-03-22T13:07:13.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:13.003 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:14.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:14.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.202+0000 7f6d6da00640 1 Processor -- start 2024-03-22T13:07:14.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.202+0000 7f6d6da00640 1 -- start start 2024-03-22T13:07:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.203+0000 7f6d6da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d68152400 0x7f6d6815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.203+0000 7f6d6da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d681590b0 con 0x7f6d6806f8c0 2024-03-22T13:07:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.204+0000 7f6d6da00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680721f0 con 0x7f6d6815bc80 2024-03-22T13:07:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.204+0000 7f6d6da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d680724d0 con 0x7f6d68152400 2024-03-22T13:07:14.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.204+0000 7f6d66a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d68152400 0x7f6d6815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:14.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.204+0000 7f6d66000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6d6815bc80 0x7f6d680716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60412/0 (socket says 172.21.15.6:60412) 2024-03-22T13:07:14.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.204+0000 7f6d66000640 1 -- 172.21.15.6:0/1022221621 learned_addr learned my addr 172.21.15.6:0/1022221621 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:14.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.207+0000 7f6d65600640 1 -- 172.21.15.6:0/1022221621 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139338891 0 0) 0x7f6d680721f0 con 0x7f6d6815bc80 2024-03-22T13:07:14.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.207+0000 7f6d65600640 1 -- 172.21.15.6:0/1022221621 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d4c003520 con 0x7f6d6815bc80 2024-03-22T13:07:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.209+0000 7f6d66a00640 1 -- 172.21.15.6:0/1022221621 >> v1:172.21.15.92:6789/0 conn(0x7f6d6815bc80 legacy=0x7f6d680716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.209+0000 7f6d66a00640 1 -- 172.21.15.6:0/1022221621 >> v1:172.21.15.6:6789/0 conn(0x7f6d6806f8c0 legacy=0x7f6d680c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.209+0000 7f6d66a00640 1 -- 172.21.15.6:0/1022221621 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d681582f0 con 0x7f6d68152400 2024-03-22T13:07:14.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.215+0000 7f6d66a00640 1 --2- 172.21.15.6:0/1022221621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d68152400 0x7f6d6815b740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009d90 tx=0x7f6d5802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4179e54acd92b0cb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.221+0000 7f6d65600640 1 -- 172.21.15.6:0/1022221621 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d58039070 con 0x7f6d68152400 2024-03-22T13:07:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.221+0000 7f6d65600640 1 -- 172.21.15.6:0/1022221621 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5802cd80 con 0x7f6d68152400 2024-03-22T13:07:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.221+0000 7f6d65600640 1 -- 172.21.15.6:0/1022221621 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d58034660 con 0x7f6d68152400 2024-03-22T13:07:14.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.222+0000 7f6d6da00640 1 -- 172.21.15.6:0/1022221621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d68152400 msgr2=0x7f6d6815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:14.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.222+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1022221621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d68152400 0x7f6d6815b740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009d90 tx=0x7f6d5802bfd0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.222+0000 7f6d6da00640 1 -- 172.21.15.6:0/1022221621 shutdown_connections 2024-03-22T13:07:14.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.222+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1022221621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d68152400 0x7f6d6815b740 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.222+0000 7f6d6da00640 1 -- 172.21.15.6:0/1022221621 >> 172.21.15.6:0/1022221621 conn(0x7f6d680cb930 msgr2=0x7f6d680cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:14.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.222+0000 7f6d6da00640 1 -- 172.21.15.6:0/1022221621 shutdown_connections 2024-03-22T13:07:14.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.222+0000 7f6d6da00640 1 -- 172.21.15.6:0/1022221621 wait complete. 2024-03-22T13:07:14.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.224+0000 7f6d6da00640 1 Processor -- start 2024-03-22T13:07:14.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.224+0000 7f6d6da00640 1 -- start start 2024-03-22T13:07:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.225+0000 7f6d6da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d6806f8c0 0x7f6d681ec830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.225+0000 7f6d6da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d68152400 0x7f6d680ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.225+0000 7f6d6da00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d6815bc80 0x7f6d68148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.225+0000 7f6d6da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d68158b90 con 0x7f6d6806f8c0 2024-03-22T13:07:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.225+0000 7f6d6da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d681588b0 con 0x7f6d68152400 2024-03-22T13:07:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.225+0000 7f6d6da00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d681527e0 con 0x7f6d6815bc80 2024-03-22T13:07:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.226+0000 7f6d66a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d6806f8c0 0x7f6d681ec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.226+0000 7f6d66000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d68152400 0x7f6d680ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.226+0000 7f6d67400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d6815bc80 0x7f6d68148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.226+0000 7f6d66a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d6806f8c0 0x7f6d681ec830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37268/0 (socket says 172.21.15.6:37268) 2024-03-22T13:07:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.226+0000 7f6d66a00640 1 -- 172.21.15.6:0/1577071031 learned_addr learned my addr 172.21.15.6:0/1577071031 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:14.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.230+0000 7f6d66a00640 1 -- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d6815bc80 msgr2=0x7f6d68148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:14.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.230+0000 7f6d66a00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d6815bc80 0x7f6d68148180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.230+0000 7f6d66a00640 1 -- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d68152400 msgr2=0x7f6d680ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:14.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.230+0000 7f6d66a00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d68152400 0x7f6d680ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.230+0000 7f6d66a00640 1 -- 172.21.15.6:0/1577071031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d68159da0 con 0x7f6d6806f8c0 2024-03-22T13:07:14.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.233+0000 7f6d66a00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d6806f8c0 0x7f6d681ec830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009d90 tx=0x7f6d58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.239+0000 7f6d5ea00640 1 -- 172.21.15.6:0/1577071031 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d58044020 con 0x7f6d6806f8c0 2024-03-22T13:07:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.239+0000 7f6d5ea00640 1 -- 172.21.15.6:0/1577071031 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5803f470 con 0x7f6d6806f8c0 2024-03-22T13:07:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.239+0000 7f6d5ea00640 1 -- 172.21.15.6:0/1577071031 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d58039040 con 0x7f6d6806f8c0 2024-03-22T13:07:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.239+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6815a040 con 0x7f6d6806f8c0 2024-03-22T13:07:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.239+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d681ec030 con 0x7f6d6806f8c0 2024-03-22T13:07:14.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.240+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d68072d10 con 0x7f6d6806f8c0 2024-03-22T13:07:14.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.574+0000 7f6d5ea00640 1 -- 172.21.15.6:0/1577071031 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6d58047020 con 0x7f6d6806f8c0 2024-03-22T13:07:14.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.578+0000 7f6d5ea00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d4406bc20 0x7f6d4406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:14.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.578+0000 7f6d66000640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d4406bc20 0x7f6d4406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:14.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.579+0000 7f6d66000640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d4406bc20 0x7f6d4406e0e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6d681f04c0 tx=0x7f6d50009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:14.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.617+0000 7f6d5ea00640 1 -- 172.21.15.6:0/1577071031 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6d58079d80 con 0x7f6d6806f8c0 2024-03-22T13:07:14.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.618+0000 7f6d5ea00640 1 -- 172.21.15.6:0/1577071031 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6d68072d10 con 0x7f6d6806f8c0 2024-03-22T13:07:14.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.888+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d680756c0 con 0x7f6d6806f8c0 2024-03-22T13:07:14.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.912+0000 7f6d5ea00640 1 -- 172.21.15.6:0/1577071031 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f6d5804e030 con 0x7f6d6806f8c0 2024-03-22T13:07:14.913 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:14.913 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d4406bc20 msgr2=0x7f6d4406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d4406bc20 0x7f6d4406e0e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6d681f04c0 tx=0x7f6d50009040 comp rx=0 tx=0).stop 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d6806f8c0 msgr2=0x7f6d681ec830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d6806f8c0 0x7f6d681ec830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009d90 tx=0x7f6d58004a40 comp rx=0 tx=0).stop 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 shutdown_connections 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d6815bc80 0x7f6d68148180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d4406bc20 0x7f6d4406e0e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d68152400 0x7f6d680ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 --2- 172.21.15.6:0/1577071031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d6806f8c0 0x7f6d681ec830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.916+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 >> 172.21.15.6:0/1577071031 conn(0x7f6d680cb930 msgr2=0x7f6d680744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.917+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 shutdown_connections 2024-03-22T13:07:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:14.917+0000 7f6d6da00640 1 -- 172.21.15.6:0/1577071031 wait complete. 2024-03-22T13:07:15.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:15.545 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:16.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:16.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.758+0000 7f0834800640 1 Processor -- start 2024-03-22T13:07:16.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.758+0000 7f0834800640 1 -- start start 2024-03-22T13:07:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.759+0000 7f0834800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c152400 0x7f082c15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.759+0000 7f0834800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f082c06f8c0 0x7f082c0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.759+0000 7f0834800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c1590b0 con 0x7f082c15bc80 2024-03-22T13:07:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.759+0000 7f0834800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082c0724d0 con 0x7f082c06f8c0 2024-03-22T13:07:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.759+0000 7f0834800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0721f0 con 0x7f082c152400 2024-03-22T13:07:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.760+0000 7f0832800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f082c06f8c0 0x7f082c0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.761+0000 7f0832800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f082c06f8c0 0x7f082c0c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:55452/0 (socket says 172.21.15.6:55452) 2024-03-22T13:07:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.761+0000 7f0832800640 1 -- 172.21.15.6:0/367321002 learned_addr learned my addr 172.21.15.6:0/367321002 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.765+0000 7f0832800640 1 -- 172.21.15.6:0/367321002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c152400 msgr2=0x7f082c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.765+0000 7f0832800640 1 --2- 172.21.15.6:0/367321002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c152400 0x7f082c15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.765+0000 7f0832800640 1 -- 172.21.15.6:0/367321002 >> v1:172.21.15.92:6789/0 conn(0x7f082c15bc80 legacy=0x7f082c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.765+0000 7f0832800640 1 -- 172.21.15.6:0/367321002 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c1582f0 con 0x7f082c06f8c0 2024-03-22T13:07:16.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.771+0000 7f0832800640 1 --2- 172.21.15.6:0/367321002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c06f8c0 0x7f082c0c5040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0828009e50 tx=0x7f082802bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ec709daff95b497 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:16.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.775+0000 7f0830a00640 1 -- 172.21.15.6:0/367321002 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082802cb90 con 0x7f082c06f8c0 2024-03-22T13:07:16.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.775+0000 7f0830a00640 1 -- 172.21.15.6:0/367321002 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082802cd30 con 0x7f082c06f8c0 2024-03-22T13:07:16.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.775+0000 7f0830a00640 1 -- 172.21.15.6:0/367321002 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828034610 con 0x7f082c06f8c0 2024-03-22T13:07:16.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.776+0000 7f0834800640 1 -- 172.21.15.6:0/367321002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c06f8c0 msgr2=0x7f082c0c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:16.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.776+0000 7f0834800640 1 --2- 172.21.15.6:0/367321002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c06f8c0 0x7f082c0c5040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0828009e50 tx=0x7f082802bff0 comp rx=0 tx=0).stop 2024-03-22T13:07:16.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.776+0000 7f0834800640 1 -- 172.21.15.6:0/367321002 shutdown_connections 2024-03-22T13:07:16.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.776+0000 7f0834800640 1 --2- 172.21.15.6:0/367321002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c06f8c0 0x7f082c0c5040 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:16.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.776+0000 7f0834800640 1 --2- 172.21.15.6:0/367321002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c152400 0x7f082c15b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:16.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.776+0000 7f0834800640 1 -- 172.21.15.6:0/367321002 >> 172.21.15.6:0/367321002 conn(0x7f082c0cb930 msgr2=0x7f082c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:16.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.777+0000 7f0834800640 1 -- 172.21.15.6:0/367321002 shutdown_connections 2024-03-22T13:07:16.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.777+0000 7f0834800640 1 -- 172.21.15.6:0/367321002 wait complete. 2024-03-22T13:07:16.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.778+0000 7f0834800640 1 Processor -- start 2024-03-22T13:07:16.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.779+0000 7f0834800640 1 -- start start 2024-03-22T13:07:16.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.780+0000 7f0834800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c06f8c0 0x7f082c1f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:16.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.780+0000 7f0834800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c152400 0x7f082c1e7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:16.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.780+0000 7f0834800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f082c15bc80 0x7f082c1f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:16.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.780+0000 7f0834800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c158b90 con 0x7f082c15bc80 2024-03-22T13:07:16.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.780+0000 7f0834800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c1588b0 con 0x7f082c152400 2024-03-22T13:07:16.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.780+0000 7f0834800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c1527e0 con 0x7f082c06f8c0 2024-03-22T13:07:16.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.780+0000 7f0831400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c152400 0x7f082c1e7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:16.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.781+0000 7f0832800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f082c15bc80 0x7f082c1f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:16.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.781+0000 7f0831400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c152400 0x7f082c1e7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:55456/0 (socket says 172.21.15.6:55456) 2024-03-22T13:07:16.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.781+0000 7f0831400640 1 -- 172.21.15.6:0/3289044105 learned_addr learned my addr 172.21.15.6:0/3289044105 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:16.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.784+0000 7f0831400640 1 -- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c06f8c0 msgr2=0x7f082c1f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:16.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.784+0000 7f0831400640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c06f8c0 0x7f082c1f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:16.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.784+0000 7f0831400640 1 -- 172.21.15.6:0/3289044105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f082c15bc80 msgr2=0x7f082c1f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:16.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.784+0000 7f0831400640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f082c15bc80 0x7f082c1f0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:16.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.784+0000 7f0831400640 1 -- 172.21.15.6:0/3289044105 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c1f48a0 con 0x7f082c152400 2024-03-22T13:07:16.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.787+0000 7f0831400640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c152400 0x7f082c1e7150 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f081c00c470 tx=0x7f081c00c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:16.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.794+0000 7f0826000640 1 -- 172.21.15.6:0/3289044105 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c0052d0 con 0x7f082c152400 2024-03-22T13:07:16.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.794+0000 7f0826000640 1 -- 172.21.15.6:0/3289044105 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081c005490 con 0x7f082c152400 2024-03-22T13:07:16.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.794+0000 7f0826000640 1 -- 172.21.15.6:0/3289044105 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c007260 con 0x7f082c152400 2024-03-22T13:07:16.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.794+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c15a180 con 0x7f082c152400 2024-03-22T13:07:16.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.795+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c15a740 con 0x7f082c152400 2024-03-22T13:07:16.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:16.796+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082c151f60 con 0x7f082c152400 2024-03-22T13:07:17.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.072+0000 7f0826000640 1 -- 172.21.15.6:0/3289044105 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f081c007400 con 0x7f082c152400 2024-03-22T13:07:17.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.074+0000 7f0826000640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f081006bc50 0x7f081006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:17.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.075+0000 7f0831e00640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f081006bc50 0x7f081006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:17.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.076+0000 7f0831e00640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f081006bc50 0x7f081006e110 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0820009710 tx=0x7f0820006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:17.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.076+0000 7f0826000640 1 -- 172.21.15.6:0/3289044105 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f081c056860 con 0x7f082c152400 2024-03-22T13:07:17.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.088+0000 7f0826000640 1 -- 172.21.15.6:0/3289044105 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f082c151f60 con 0x7f082c152400 2024-03-22T13:07:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.332+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f082c1f0410 con 0x7f082c152400 2024-03-22T13:07:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.353+0000 7f0826000640 1 -- 172.21.15.6:0/3289044105 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f081c0587c0 con 0x7f082c152400 2024-03-22T13:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:17.354 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f081006bc50 msgr2=0x7f081006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f081006bc50 0x7f081006e110 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0820009710 tx=0x7f0820006c90 comp rx=0 tx=0).stop 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c152400 msgr2=0x7f082c1e7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c152400 0x7f082c1e7150 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f081c00c470 tx=0x7f081c00c930 comp rx=0 tx=0).stop 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 shutdown_connections 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f082c15bc80 0x7f082c1f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f081006bc50 0x7f081006e110 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f082c152400 0x7f082c1e7150 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 --2- 172.21.15.6:0/3289044105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f082c06f8c0 0x7f082c1f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.356+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 >> 172.21.15.6:0/3289044105 conn(0x7f082c0cb930 msgr2=0x7f082c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.357+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 shutdown_connections 2024-03-22T13:07:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:17.357+0000 7f0834800640 1 -- 172.21.15.6:0/3289044105 wait complete. 2024-03-22T13:07:17.979 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:17.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:18.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:19.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.175+0000 7f2325400640 1 Processor -- start 2024-03-22T13:07:19.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.176+0000 7f2325400640 1 -- start start 2024-03-22T13:07:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.177+0000 7f2325400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.177+0000 7f2325400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f232015bc80 0x7f23200716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.177+0000 7f2325400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23201590b0 con 0x7f232006f8c0 2024-03-22T13:07:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.177+0000 7f2325400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23200724d0 con 0x7f232015bc80 2024-03-22T13:07:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.177+0000 7f2325400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23200721f0 con 0x7f2320152400 2024-03-22T13:07:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.178+0000 7f231e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.179+0000 7f231e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36078/0 (socket says 172.21.15.6:36078) 2024-03-22T13:07:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.179+0000 7f231e800640 1 -- 172.21.15.6:0/556080298 learned_addr learned my addr 172.21.15.6:0/556080298 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.180+0000 7f231de00640 1 --2- 172.21.15.6:0/556080298 >> v2:172.21.15.6:3300/0 conn(0x7f232015bc80 0x7f23200716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:19.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.182+0000 7f231e800640 1 -- 172.21.15.6:0/556080298 >> v2:172.21.15.6:3300/0 conn(0x7f232015bc80 msgr2=0x7f23200716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:19.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.182+0000 7f231e800640 1 --2- 172.21.15.6:0/556080298 >> v2:172.21.15.6:3300/0 conn(0x7f232015bc80 0x7f23200716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.182+0000 7f231e800640 1 -- 172.21.15.6:0/556080298 >> v1:172.21.15.6:6789/0 conn(0x7f232006f8c0 legacy=0x7f23200c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:19.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.182+0000 7f231e800640 1 -- 172.21.15.6:0/556080298 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23201582f0 con 0x7f2320152400 2024-03-22T13:07:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.184+0000 7f231e800640 1 --2- 172.21.15.6:0/556080298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015b740 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2310009870 tx=0x7f231002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2214a537ea83cecc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.189+0000 7f231d400640 1 -- 172.21.15.6:0/556080298 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231002f830 con 0x7f2320152400 2024-03-22T13:07:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.189+0000 7f231d400640 1 -- 172.21.15.6:0/556080298 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231002f9d0 con 0x7f2320152400 2024-03-22T13:07:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f231d400640 1 -- 172.21.15.6:0/556080298 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231002fda0 con 0x7f2320152400 2024-03-22T13:07:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f2325400640 1 -- 172.21.15.6:0/556080298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 msgr2=0x7f232015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f2325400640 1 --2- 172.21.15.6:0/556080298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015b740 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2310009870 tx=0x7f231002ba10 comp rx=0 tx=0).stop 2024-03-22T13:07:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f2325400640 1 -- 172.21.15.6:0/556080298 shutdown_connections 2024-03-22T13:07:19.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f2325400640 1 --2- 172.21.15.6:0/556080298 >> v2:172.21.15.6:3300/0 conn(0x7f232015bc80 0x7f23200716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f2325400640 1 --2- 172.21.15.6:0/556080298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015b740 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f2325400640 1 -- 172.21.15.6:0/556080298 >> 172.21.15.6:0/556080298 conn(0x7f23200cb930 msgr2=0x7f23200cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:19.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.190+0000 7f2325400640 1 -- 172.21.15.6:0/556080298 shutdown_connections 2024-03-22T13:07:19.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.191+0000 7f2325400640 1 -- 172.21.15.6:0/556080298 wait complete. 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.192+0000 7f2325400640 1 Processor -- start 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.192+0000 7f2325400640 1 -- start start 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.193+0000 7f2325400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232006f8c0 0x7f2320147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.193+0000 7f2325400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.193+0000 7f2325400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f232015bc80 0x7f2320145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.193+0000 7f2325400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2320158b90 con 0x7f232015bc80 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.193+0000 7f2325400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23201588b0 con 0x7f232006f8c0 2024-03-22T13:07:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.193+0000 7f2325400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23201527e0 con 0x7f2320152400 2024-03-22T13:07:19.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.194+0000 7f231de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:19.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.194+0000 7f231e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232006f8c0 0x7f2320147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:19.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.194+0000 7f231f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f232015bc80 0x7f2320145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:19.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.194+0000 7f231e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232006f8c0 0x7f2320147aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:55470/0 (socket says 172.21.15.6:55470) 2024-03-22T13:07:19.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.194+0000 7f231e800640 1 -- 172.21.15.6:0/1008853732 learned_addr learned my addr 172.21.15.6:0/1008853732 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:19.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.198+0000 7f231e800640 1 -- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 msgr2=0x7f232015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:19.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.198+0000 7f231e800640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.198+0000 7f231e800640 1 -- 172.21.15.6:0/1008853732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f232015bc80 msgr2=0x7f2320145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:19.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.198+0000 7f231e800640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f232015bc80 0x7f2320145000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.198+0000 7f231e800640 1 -- 172.21.15.6:0/1008853732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200c30a0 con 0x7f232006f8c0 2024-03-22T13:07:19.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.201+0000 7f231e800640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232006f8c0 0x7f2320147aa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2310000c00 tx=0x7f23100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:19.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.204+0000 7f2316a00640 1 -- 172.21.15.6:0/1008853732 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310037d00 con 0x7f232006f8c0 2024-03-22T13:07:19.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.204+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23200c3380 con 0x7f232006f8c0 2024-03-22T13:07:19.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.204+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232015a1d0 con 0x7f232006f8c0 2024-03-22T13:07:19.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.206+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23200756c0 con 0x7f232006f8c0 2024-03-22T13:07:19.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.409+0000 7f2316a00640 1 -- 172.21.15.6:0/1008853732 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2310047070 con 0x7f232006f8c0 2024-03-22T13:07:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.522+0000 7f2316a00640 1 -- 172.21.15.6:0/1008853732 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310042480 con 0x7f232006f8c0 2024-03-22T13:07:19.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.525+0000 7f2316a00640 1 -- 172.21.15.6:0/1008853732 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2310040050 con 0x7f232006f8c0 2024-03-22T13:07:19.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.527+0000 7f2316a00640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f230006bbd0 0x7f230006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:19.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.528+0000 7f231de00640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f230006bbd0 0x7f230006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:19.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.529+0000 7f231de00640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f230006bbd0 0x7f230006e090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2320071a00 tx=0x7f230c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:19.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.530+0000 7f2316a00640 1 -- 172.21.15.6:0/1008853732 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f231007a540 con 0x7f232006f8c0 2024-03-22T13:07:19.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.551+0000 7f2316a00640 1 -- 172.21.15.6:0/1008853732 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2310033030 con 0x7f232006f8c0 2024-03-22T13:07:19.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.816+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23200c6400 con 0x7f232006f8c0 2024-03-22T13:07:19.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.884+0000 7f2316a00640 1 -- 172.21.15.6:0/1008853732 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f231007c4a0 con 0x7f232006f8c0 2024-03-22T13:07:19.884 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:19.885 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:19.885 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.887+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f230006bbd0 msgr2=0x7f230006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.887+0000 7f2325400640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f230006bbd0 0x7f230006e090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2320071a00 tx=0x7f230c008040 comp rx=0 tx=0).stop 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.887+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232006f8c0 msgr2=0x7f2320147aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.887+0000 7f2325400640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232006f8c0 0x7f2320147aa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2310000c00 tx=0x7f23100046c0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 shutdown_connections 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f232015bc80 0x7f2320145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2320152400 0x7f232015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f230006bbd0 0x7f230006e090 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 --2- 172.21.15.6:0/1008853732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232006f8c0 0x7f2320147aa0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 >> 172.21.15.6:0/1008853732 conn(0x7f23200cb930 msgr2=0x7f23200c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 shutdown_connections 2024-03-22T13:07:19.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:19.888+0000 7f2325400640 1 -- 172.21.15.6:0/1008853732 wait complete. 2024-03-22T13:07:20.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:20.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:21.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:21.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.702+0000 7f77db600640 1 Processor -- start 2024-03-22T13:07:21.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.702+0000 7f77db600640 1 -- start start 2024-03-22T13:07:21.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.703+0000 7f77db600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d4154240 0x7f77d415ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:21.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.703+0000 7f77db600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f77d40c7ee0 0x7f77d40c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:21.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.703+0000 7f77db600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f77d40c68c0 0x7f77d406b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:21.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.703+0000 7f77db600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d415e130 con 0x7f77d40c7ee0 2024-03-22T13:07:21.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.703+0000 7f77db600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d4154620 con 0x7f77d40c68c0 2024-03-22T13:07:21.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.703+0000 7f77db600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40c6e00 con 0x7f77d4154240 2024-03-22T13:07:21.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.704+0000 7f77d8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d4154240 0x7f77d415ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:21.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.704+0000 7f77d9600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f77d40c68c0 0x7f77d406b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:21.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.704+0000 7f77d8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d4154240 0x7f77d415ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36098/0 (socket says 172.21.15.6:36098) 2024-03-22T13:07:21.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.704+0000 7f77d8c00640 1 -- 172.21.15.6:0/3671775233 learned_addr learned my addr 172.21.15.6:0/3671775233 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:21.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.704+0000 7f77d3e00640 1 --2- 172.21.15.6:0/3671775233 >> v2:172.21.15.6:3300/0 conn(0x7f77d40c7ee0 0x7f77d40c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:21.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.706+0000 7f77d8c00640 1 -- 172.21.15.6:0/3671775233 >> v2:172.21.15.92:3300/0 conn(0x7f77d40c68c0 msgr2=0x7f77d406b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:21.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.706+0000 7f77d8c00640 1 --2- 172.21.15.6:0/3671775233 >> v2:172.21.15.92:3300/0 conn(0x7f77d40c68c0 0x7f77d406b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.706+0000 7f77d8c00640 1 -- 172.21.15.6:0/3671775233 >> v2:172.21.15.6:3300/0 conn(0x7f77d40c7ee0 msgr2=0x7f77d40c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:21.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.706+0000 7f77d8c00640 1 --2- 172.21.15.6:0/3671775233 >> v2:172.21.15.6:3300/0 conn(0x7f77d40c7ee0 0x7f77d40c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.706+0000 7f77d8c00640 1 -- 172.21.15.6:0/3671775233 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d40c70e0 con 0x7f77d4154240 2024-03-22T13:07:21.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.709+0000 7f77d8c00640 1 --2- 172.21.15.6:0/3671775233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d4154240 0x7f77d415ac90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f77d4157d50 tx=0x7f77c802b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e03c7307c05b58c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:21.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.713+0000 7f77d3400640 1 -- 172.21.15.6:0/3671775233 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8034070 con 0x7f77d4154240 2024-03-22T13:07:21.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.713+0000 7f77d3400640 1 -- 172.21.15.6:0/3671775233 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c8030860 con 0x7f77d4154240 2024-03-22T13:07:21.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.713+0000 7f77d3400640 1 -- 172.21.15.6:0/3671775233 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8030c30 con 0x7f77d4154240 2024-03-22T13:07:21.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.714+0000 7f77db600640 1 -- 172.21.15.6:0/3671775233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d4154240 msgr2=0x7f77d415ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:21.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.714+0000 7f77db600640 1 --2- 172.21.15.6:0/3671775233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d4154240 0x7f77d415ac90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f77d4157d50 tx=0x7f77c802b9a0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.715+0000 7f77db600640 1 -- 172.21.15.6:0/3671775233 shutdown_connections 2024-03-22T13:07:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.715+0000 7f77db600640 1 --2- 172.21.15.6:0/3671775233 >> v2:172.21.15.92:3300/0 conn(0x7f77d40c68c0 0x7f77d406b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.715+0000 7f77db600640 1 --2- 172.21.15.6:0/3671775233 >> v2:172.21.15.6:3300/0 conn(0x7f77d40c7ee0 0x7f77d40c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.715+0000 7f77db600640 1 --2- 172.21.15.6:0/3671775233 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d4154240 0x7f77d415ac90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.715+0000 7f77db600640 1 -- 172.21.15.6:0/3671775233 >> 172.21.15.6:0/3671775233 conn(0x7f77d40cb930 msgr2=0x7f77d40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.715+0000 7f77db600640 1 -- 172.21.15.6:0/3671775233 shutdown_connections 2024-03-22T13:07:21.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.715+0000 7f77db600640 1 -- 172.21.15.6:0/3671775233 wait complete. 2024-03-22T13:07:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.717+0000 7f77db600640 1 Processor -- start 2024-03-22T13:07:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.718+0000 7f77db600640 1 -- start start 2024-03-22T13:07:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.719+0000 7f77db600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d40c68c0 0x7f77d41f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.719+0000 7f77db600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77d40c7ee0 0x7f77d406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.719+0000 7f77db600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d41518a0 0x7f77d406fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.719+0000 7f77db600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d415ddc0 con 0x7f77d40c68c0 2024-03-22T13:07:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.719+0000 7f77db600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d415dae0 con 0x7f77d40c7ee0 2024-03-22T13:07:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.719+0000 7f77db600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d415e5c0 con 0x7f77d41518a0 2024-03-22T13:07:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.720+0000 7f77d9600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d41518a0 0x7f77d406fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.720+0000 7f77d8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d40c68c0 0x7f77d41f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.720+0000 7f77d3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77d40c7ee0 0x7f77d406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.720+0000 7f77d9600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d41518a0 0x7f77d406fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36100/0 (socket says 172.21.15.6:36100) 2024-03-22T13:07:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.720+0000 7f77d9600640 1 -- 172.21.15.6:0/46591943 learned_addr learned my addr 172.21.15.6:0/46591943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.725+0000 7f77d9600640 1 -- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77d40c7ee0 msgr2=0x7f77d406f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.725+0000 7f77d9600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77d40c7ee0 0x7f77d406f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.725+0000 7f77d9600640 1 -- 172.21.15.6:0/46591943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d40c68c0 msgr2=0x7f77d41f0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.725+0000 7f77d9600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d40c68c0 0x7f77d41f0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.725+0000 7f77d9600640 1 -- 172.21.15.6:0/46591943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d41575d0 con 0x7f77d41518a0 2024-03-22T13:07:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.729+0000 7f77d9600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d41518a0 0x7f77d406fad0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00ef30 tx=0x7f77cc00b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:21.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.733+0000 7f77d0c00640 1 -- 172.21.15.6:0/46591943 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc018020 con 0x7f77d41518a0 2024-03-22T13:07:21.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.733+0000 7f77d0c00640 1 -- 172.21.15.6:0/46591943 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77cc014400 con 0x7f77d41518a0 2024-03-22T13:07:21.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.733+0000 7f77d0c00640 1 -- 172.21.15.6:0/46591943 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc0147d0 con 0x7f77d41518a0 2024-03-22T13:07:21.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.734+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d41e6110 con 0x7f77d41518a0 2024-03-22T13:07:21.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.734+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d41f0350 con 0x7f77d41518a0 2024-03-22T13:07:21.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.735+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d40756c0 con 0x7f77d41518a0 2024-03-22T13:07:21.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.772+0000 7f77d0c00640 1 -- 172.21.15.6:0/46591943 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f77cc005d10 con 0x7f77d41518a0 2024-03-22T13:07:21.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.773+0000 7f77d0c00640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f77b406bc50 0x7f77b406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:21.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.774+0000 7f77d8c00640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f77b406bc50 0x7f77b406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.775+0000 7f77d8c00640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f77b406bc50 0x7f77b406e110 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f77c8036000 tx=0x7f77c8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:21.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:21.851+0000 7f77d0c00640 1 -- 172.21.15.6:0/46591943 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f77cc057c80 con 0x7f77d41518a0 2024-03-22T13:07:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.054+0000 7f77d0c00640 1 -- 172.21.15.6:0/46591943 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f77cc022070 con 0x7f77d41518a0 2024-03-22T13:07:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.309+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77d40c7420 con 0x7f77d41518a0 2024-03-22T13:07:22.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.342+0000 7f77d0c00640 1 -- 172.21.15.6:0/46591943 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f77cc059be0 con 0x7f77d41518a0 2024-03-22T13:07:22.342 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:22.342 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:22.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f77b406bc50 msgr2=0x7f77b406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f77b406bc50 0x7f77b406e110 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f77c8036000 tx=0x7f77c8008030 comp rx=0 tx=0).stop 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d41518a0 msgr2=0x7f77d406fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d41518a0 0x7f77d406fad0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00ef30 tx=0x7f77cc00b480 comp rx=0 tx=0).stop 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 shutdown_connections 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77d41518a0 0x7f77d406fad0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f77b406bc50 0x7f77b406e110 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77d40c7ee0 0x7f77d406f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 --2- 172.21.15.6:0/46591943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77d40c68c0 0x7f77d41f0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.344+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 >> 172.21.15.6:0/46591943 conn(0x7f77d40cb930 msgr2=0x7f77d4159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.345+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 shutdown_connections 2024-03-22T13:07:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:22.345+0000 7f77db600640 1 -- 172.21.15.6:0/46591943 wait complete. 2024-03-22T13:07:22.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:22.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:23.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.159+0000 7f12dfe00640 1 Processor -- start 2024-03-22T13:07:24.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.159+0000 7f12dfe00640 1 -- start start 2024-03-22T13:07:24.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.160+0000 7f12dfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12d80ca520 0x7f12d806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:24.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.160+0000 7f12dfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d806c7a0 con 0x7f12d80ccad0 2024-03-22T13:07:24.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.160+0000 7f12dfe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d806ca80 con 0x7f12d815b0e0 2024-03-22T13:07:24.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.160+0000 7f12dfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d80cd000 con 0x7f12d80ca520 2024-03-22T13:07:24.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.161+0000 7f12dca00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f12d815b0e0 0x7f12d80cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:54138/0 (socket says 172.21.15.6:54138) 2024-03-22T13:07:24.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.161+0000 7f12dca00640 1 -- 172.21.15.6:0/3307545877 learned_addr learned my addr 172.21.15.6:0/3307545877 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:24.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.161+0000 7f12dd400640 1 --2- 172.21.15.6:0/3307545877 >> v2:172.21.15.6:3300/0 conn(0x7f12d80ca520 0x7f12d806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:24.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.164+0000 7f12d7e00640 1 -- 172.21.15.6:0/3307545877 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871420944 0 0) 0x7f12d806ca80 con 0x7f12d815b0e0 2024-03-22T13:07:24.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.164+0000 7f12d7e00640 1 -- 172.21.15.6:0/3307545877 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c0003520 con 0x7f12d815b0e0 2024-03-22T13:07:24.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.167+0000 7f12dd400640 1 -- 172.21.15.6:0/3307545877 >> v1:172.21.15.92:6789/0 conn(0x7f12d815b0e0 legacy=0x7f12d80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:24.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.167+0000 7f12dd400640 1 -- 172.21.15.6:0/3307545877 >> v1:172.21.15.6:6789/0 conn(0x7f12d80ccad0 legacy=0x7f12d8164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:24.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.167+0000 7f12dd400640 1 -- 172.21.15.6:0/3307545877 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d806cd60 con 0x7f12d80ca520 2024-03-22T13:07:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.173+0000 7f12dd400640 1 --2- 172.21.15.6:0/3307545877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ca520 0x7f12d806d840 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0097a0 tx=0x7f12cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640aaa7ee055b2ed server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:24.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.178+0000 7f12d7e00640 1 -- 172.21.15.6:0/3307545877 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc034070 con 0x7f12d80ca520 2024-03-22T13:07:24.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.178+0000 7f12d7e00640 1 -- 172.21.15.6:0/3307545877 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12cc030610 con 0x7f12d80ca520 2024-03-22T13:07:24.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.178+0000 7f12d7e00640 1 -- 172.21.15.6:0/3307545877 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc0309e0 con 0x7f12d80ca520 2024-03-22T13:07:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.179+0000 7f12dfe00640 1 -- 172.21.15.6:0/3307545877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ca520 msgr2=0x7f12d806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.179+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3307545877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ca520 0x7f12d806d840 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0097a0 tx=0x7f12cc02b960 comp rx=0 tx=0).stop 2024-03-22T13:07:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.179+0000 7f12dfe00640 1 -- 172.21.15.6:0/3307545877 shutdown_connections 2024-03-22T13:07:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.179+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3307545877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ca520 0x7f12d806d840 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.179+0000 7f12dfe00640 1 -- 172.21.15.6:0/3307545877 >> 172.21.15.6:0/3307545877 conn(0x7f12d80b16b0 msgr2=0x7f12d80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.179+0000 7f12dfe00640 1 -- 172.21.15.6:0/3307545877 shutdown_connections 2024-03-22T13:07:24.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.179+0000 7f12dfe00640 1 -- 172.21.15.6:0/3307545877 wait complete. 2024-03-22T13:07:24.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.181+0000 7f12dfe00640 1 Processor -- start 2024-03-22T13:07:24.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.181+0000 7f12dfe00640 1 -- start start 2024-03-22T13:07:24.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.182+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12d80ca520 0x7f12d815db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:24.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.182+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ccad0 0x7f12d81574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:24.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.182+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12d815b0e0 0x7f12d81579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:24.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.182+0000 7f12dfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8154c20 con 0x7f12d80ccad0 2024-03-22T13:07:24.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.182+0000 7f12dfe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d806d320 con 0x7f12d815b0e0 2024-03-22T13:07:24.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.182+0000 7f12dfe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d815b980 con 0x7f12d80ca520 2024-03-22T13:07:24.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.183+0000 7f12dca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ccad0 0x7f12d81574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:24.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.183+0000 7f12dd400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12d80ca520 0x7f12d815db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:24.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.183+0000 7f12dca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ccad0 0x7f12d81574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58312/0 (socket says 172.21.15.6:58312) 2024-03-22T13:07:24.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.183+0000 7f12dca00640 1 -- 172.21.15.6:0/3289530076 learned_addr learned my addr 172.21.15.6:0/3289530076 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:24.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.184+0000 7f12dde00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12d815b0e0 0x7f12d81579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.188+0000 7f12dca00640 1 -- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12d80ca520 msgr2=0x7f12d815db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.188+0000 7f12dca00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12d80ca520 0x7f12d815db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.188+0000 7f12dca00640 1 -- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12d815b0e0 msgr2=0x7f12d81579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.188+0000 7f12dca00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12d815b0e0 0x7f12d81579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.188+0000 7f12dca00640 1 -- 172.21.15.6:0/3289530076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d81e5a30 con 0x7f12d80ccad0 2024-03-22T13:07:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.192+0000 7f12dca00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ccad0 0x7f12d81574a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f12c400c3c0 tx=0x7f12c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.196+0000 7f12d5600640 1 -- 172.21.15.6:0/3289530076 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4017070 con 0x7f12d80ccad0 2024-03-22T13:07:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.196+0000 7f12d5600640 1 -- 172.21.15.6:0/3289530076 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c4005ce0 con 0x7f12d80ccad0 2024-03-22T13:07:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.196+0000 7f12d5600640 1 -- 172.21.15.6:0/3289530076 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c40072f0 con 0x7f12d80ccad0 2024-03-22T13:07:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.196+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d81e5cd0 con 0x7f12d80ccad0 2024-03-22T13:07:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.196+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d81e6290 con 0x7f12d80ccad0 2024-03-22T13:07:24.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.197+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d8158170 con 0x7f12d80ccad0 2024-03-22T13:07:24.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.492+0000 7f12d5600640 1 -- 172.21.15.6:0/3289530076 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f12c401a070 con 0x7f12d80ccad0 2024-03-22T13:07:24.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.494+0000 7f12d5600640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12b806bc50 0x7f12b806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:24.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.494+0000 7f12dd400640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12b806bc50 0x7f12b806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:24.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.495+0000 7f12dd400640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12b806bc50 0x7f12b806e110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f12cc036000 tx=0x7f12cc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:24.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.525+0000 7f12d5600640 1 -- 172.21.15.6:0/3289530076 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f12c4056720 con 0x7f12d80ccad0 2024-03-22T13:07:24.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.526+0000 7f12d5600640 1 -- 172.21.15.6:0/3289530076 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f12d8158170 con 0x7f12d80ccad0 2024-03-22T13:07:24.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.806+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12d80cb080 con 0x7f12d80ccad0 2024-03-22T13:07:24.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.827+0000 7f12d5600640 1 -- 172.21.15.6:0/3289530076 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f12c4021020 con 0x7f12d80ccad0 2024-03-22T13:07:24.827 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:24.827 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:24.827 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:24.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12b806bc50 msgr2=0x7f12b806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12b806bc50 0x7f12b806e110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f12cc036000 tx=0x7f12cc03b040 comp rx=0 tx=0).stop 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ccad0 msgr2=0x7f12d81574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ccad0 0x7f12d81574a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f12c400c3c0 tx=0x7f12c400c880 comp rx=0 tx=0).stop 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 shutdown_connections 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f12b806bc50 0x7f12b806e110 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12d815b0e0 0x7f12d81579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d80ccad0 0x7f12d81574a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.829+0000 7f12dfe00640 1 --2- 172.21.15.6:0/3289530076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12d80ca520 0x7f12d815db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.830+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 >> 172.21.15.6:0/3289530076 conn(0x7f12d80b16b0 msgr2=0x7f12d80b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.830+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 shutdown_connections 2024-03-22T13:07:24.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:24.830+0000 7f12dfe00640 1 -- 172.21.15.6:0/3289530076 wait complete. 2024-03-22T13:07:25.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:25.450 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:26.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:26.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.645+0000 7fb319000640 1 Processor -- start 2024-03-22T13:07:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.645+0000 7fb319000640 1 -- start start 2024-03-22T13:07:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.646+0000 7fb319000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3141436f0 0x7fb3140cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.646+0000 7fb319000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140c9260 con 0x7fb31415d8e0 2024-03-22T13:07:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.646+0000 7fb319000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31415a2f0 con 0x7fb314071210 2024-03-22T13:07:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.646+0000 7fb319000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31415a640 con 0x7fb3141436f0 2024-03-22T13:07:26.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.674+0000 7fb313400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3141436f0 0x7fb3140cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:26.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.675+0000 7fb313400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3141436f0 0x7fb3140cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56122/0 (socket says 172.21.15.6:56122) 2024-03-22T13:07:26.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.675+0000 7fb313400640 1 -- 172.21.15.6:0/3806299708 learned_addr learned my addr 172.21.15.6:0/3806299708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.678+0000 7fb313400640 1 -- 172.21.15.6:0/3806299708 >> v1:172.21.15.92:6789/0 conn(0x7fb314071210 legacy=0x7fb314164c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.678+0000 7fb313400640 1 -- 172.21.15.6:0/3806299708 >> v1:172.21.15.6:6789/0 conn(0x7fb31415d8e0 legacy=0x7fb31415aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.678+0000 7fb313400640 1 -- 172.21.15.6:0/3806299708 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb314143ad0 con 0x7fb3141436f0 2024-03-22T13:07:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.684+0000 7fb313400640 1 --2- 172.21.15.6:0/3806299708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3141436f0 0x7fb3140cac30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb31415a7c0 tx=0x7fb30402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49eab733f999e373 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.690+0000 7fb312000640 1 -- 172.21.15.6:0/3806299708 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304034070 con 0x7fb3141436f0 2024-03-22T13:07:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.690+0000 7fb312000640 1 -- 172.21.15.6:0/3806299708 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb304030840 con 0x7fb3141436f0 2024-03-22T13:07:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.690+0000 7fb312000640 1 -- 172.21.15.6:0/3806299708 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304030c10 con 0x7fb3141436f0 2024-03-22T13:07:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.690+0000 7fb319000640 1 -- 172.21.15.6:0/3806299708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3141436f0 msgr2=0x7fb3140cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.690+0000 7fb319000640 1 --2- 172.21.15.6:0/3806299708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3141436f0 0x7fb3140cac30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb31415a7c0 tx=0x7fb30402b980 comp rx=0 tx=0).stop 2024-03-22T13:07:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.691+0000 7fb319000640 1 -- 172.21.15.6:0/3806299708 shutdown_connections 2024-03-22T13:07:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.691+0000 7fb319000640 1 --2- 172.21.15.6:0/3806299708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3141436f0 0x7fb3140cac30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.691+0000 7fb319000640 1 -- 172.21.15.6:0/3806299708 >> 172.21.15.6:0/3806299708 conn(0x7fb31406b4a0 msgr2=0x7fb31406f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.691+0000 7fb319000640 1 -- 172.21.15.6:0/3806299708 shutdown_connections 2024-03-22T13:07:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.691+0000 7fb319000640 1 -- 172.21.15.6:0/3806299708 wait complete. 2024-03-22T13:07:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.693+0000 7fb319000640 1 Processor -- start 2024-03-22T13:07:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.693+0000 7fb319000640 1 -- start start 2024-03-22T13:07:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.694+0000 7fb319000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb314071210 0x7fb3140ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.694+0000 7fb319000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb31415d8e0 0x7fb3140b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.694+0000 7fb319000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb31415b4b0 0x7fb3140b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.694+0000 7fb319000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31415e390 con 0x7fb314071210 2024-03-22T13:07:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.694+0000 7fb319000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31415e0b0 con 0x7fb31415d8e0 2024-03-22T13:07:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.694+0000 7fb319000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31415e670 con 0x7fb31415b4b0 2024-03-22T13:07:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.695+0000 7fb312a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb31415d8e0 0x7fb3140b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.695+0000 7fb313400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb314071210 0x7fb3140ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.695+0000 7fb313e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb31415b4b0 0x7fb3140b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.695+0000 7fb312a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb31415d8e0 0x7fb3140b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:56130/0 (socket says 172.21.15.6:56130) 2024-03-22T13:07:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.695+0000 7fb313e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb31415b4b0 0x7fb3140b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:39664/0 (socket says 172.21.15.6:39664) 2024-03-22T13:07:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.695+0000 7fb312a00640 1 -- 172.21.15.6:0/4224926970 learned_addr learned my addr 172.21.15.6:0/4224926970 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.695+0000 7fb313400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb314071210 0x7fb3140ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58326/0 (socket says 172.21.15.6:58326) 2024-03-22T13:07:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.699+0000 7fb312a00640 1 -- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb31415b4b0 msgr2=0x7fb3140b13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.699+0000 7fb312a00640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb31415b4b0 0x7fb3140b13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.699+0000 7fb312a00640 1 -- 172.21.15.6:0/4224926970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb314071210 msgr2=0x7fb3140ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.699+0000 7fb312a00640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb314071210 0x7fb3140ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.699+0000 7fb312a00640 1 -- 172.21.15.6:0/4224926970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3141eff90 con 0x7fb31415d8e0 2024-03-22T13:07:26.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.701+0000 7fb312a00640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb31415d8e0 0x7fb3140b0e80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb30000b3c0 tx=0x7fb30000b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:26.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.705+0000 7fb30b400640 1 -- 172.21.15.6:0/4224926970 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300011830 con 0x7fb31415d8e0 2024-03-22T13:07:26.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.705+0000 7fb30b400640 1 -- 172.21.15.6:0/4224926970 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb300016070 con 0x7fb31415d8e0 2024-03-22T13:07:26.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.705+0000 7fb30b400640 1 -- 172.21.15.6:0/4224926970 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300011bf0 con 0x7fb31415d8e0 2024-03-22T13:07:26.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.706+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3141f0250 con 0x7fb31415d8e0 2024-03-22T13:07:26.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.706+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3141f0810 con 0x7fb31415d8e0 2024-03-22T13:07:26.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.707+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31406c380 con 0x7fb31415d8e0 2024-03-22T13:07:26.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.733+0000 7fb30b400640 1 -- 172.21.15.6:0/4224926970 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb30001b460 con 0x7fb31415d8e0 2024-03-22T13:07:26.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.734+0000 7fb30b400640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb2f406bc20 0x7fb2f406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:26.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.734+0000 7fb313400640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb2f406bc20 0x7fb2f406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:26.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.735+0000 7fb313400640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb2f406bc20 0x7fb2f406e0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb304036000 tx=0x7fb304008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:26.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:26.983+0000 7fb30b400640 1 -- 172.21.15.6:0/4224926970 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb300057650 con 0x7fb31415d8e0 2024-03-22T13:07:27.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.002+0000 7fb30b400640 1 -- 172.21.15.6:0/4224926970 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb3000595b0 con 0x7fb31415d8e0 2024-03-22T13:07:27.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.266+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb314075650 con 0x7fb31415d8e0 2024-03-22T13:07:27.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.282+0000 7fb30b400640 1 -- 172.21.15.6:0/4224926970 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fb30005d650 con 0x7fb31415d8e0 2024-03-22T13:07:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:27.282 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:27.282 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:27.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb2f406bc20 msgr2=0x7fb2f406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:27.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb2f406bc20 0x7fb2f406e0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb304036000 tx=0x7fb304008030 comp rx=0 tx=0).stop 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb31415d8e0 msgr2=0x7fb3140b0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb31415d8e0 0x7fb3140b0e80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb30000b3c0 tx=0x7fb30000b880 comp rx=0 tx=0).stop 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 shutdown_connections 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb31415b4b0 0x7fb3140b13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb2f406bc20 0x7fb2f406e0e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb31415d8e0 0x7fb3140b0e80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 --2- 172.21.15.6:0/4224926970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb314071210 0x7fb3140ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 >> 172.21.15.6:0/4224926970 conn(0x7fb31406b4a0 msgr2=0x7fb314141f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 shutdown_connections 2024-03-22T13:07:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:27.285+0000 7fb319000640 1 -- 172.21.15.6:0/4224926970 wait complete. 2024-03-22T13:07:27.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:27.904 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:28.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:29.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.109+0000 7fab75000640 1 Processor -- start 2024-03-22T13:07:29.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.109+0000 7fab75000640 1 -- start start 2024-03-22T13:07:29.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.110+0000 7fab75000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab700cd300 0x7fab7015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:29.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.110+0000 7fab75000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7015b380 con 0x7fab7015aef0 2024-03-22T13:07:29.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.111+0000 7fab75000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab700cc380 con 0x7fab70154980 2024-03-22T13:07:29.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.111+0000 7fab75000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7015a520 con 0x7fab700cd300 2024-03-22T13:07:29.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.111+0000 7fab6d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fab700cd300 0x7fab7015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:29.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.111+0000 7fab6de00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fab70154980 0x7fab700ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:54162/0 (socket says 172.21.15.6:54162) 2024-03-22T13:07:29.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.111+0000 7fab6de00640 1 -- 172.21.15.6:0/4140649849 learned_addr learned my addr 172.21.15.6:0/4140649849 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:29.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.115+0000 7fab6ca00640 1 -- 172.21.15.6:0/4140649849 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655751531 0 0) 0x7fab700cc380 con 0x7fab70154980 2024-03-22T13:07:29.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.115+0000 7fab6ca00640 1 -- 172.21.15.6:0/4140649849 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab54003520 con 0x7fab70154980 2024-03-22T13:07:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.115+0000 7fab6d400640 1 -- 172.21.15.6:0/4140649849 >> v1:172.21.15.92:6789/0 conn(0x7fab70154980 legacy=0x7fab700ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.115+0000 7fab6d400640 1 -- 172.21.15.6:0/4140649849 >> v1:172.21.15.6:6789/0 conn(0x7fab7015aef0 legacy=0x7fab700ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:29.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.115+0000 7fab6d400640 1 -- 172.21.15.6:0/4140649849 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700cc660 con 0x7fab700cd300 2024-03-22T13:07:29.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.121+0000 7fab6d400640 1 --2- 172.21.15.6:0/4140649849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 0x7fab7015a9b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fab5c0097a0 tx=0x7fab5c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d6ee8c8e3694ee server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:29.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.127+0000 7fab6ca00640 1 -- 172.21.15.6:0/4140649849 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c034070 con 0x7fab700cd300 2024-03-22T13:07:29.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.127+0000 7fab6ca00640 1 -- 172.21.15.6:0/4140649849 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c030610 con 0x7fab700cd300 2024-03-22T13:07:29.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.127+0000 7fab6ca00640 1 -- 172.21.15.6:0/4140649849 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0309e0 con 0x7fab700cd300 2024-03-22T13:07:29.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.128+0000 7fab75000640 1 -- 172.21.15.6:0/4140649849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 msgr2=0x7fab7015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:29.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.128+0000 7fab75000640 1 --2- 172.21.15.6:0/4140649849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 0x7fab7015a9b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fab5c0097a0 tx=0x7fab5c02b960 comp rx=0 tx=0).stop 2024-03-22T13:07:29.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.128+0000 7fab75000640 1 -- 172.21.15.6:0/4140649849 shutdown_connections 2024-03-22T13:07:29.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.128+0000 7fab75000640 1 --2- 172.21.15.6:0/4140649849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 0x7fab7015a9b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.128+0000 7fab75000640 1 -- 172.21.15.6:0/4140649849 >> 172.21.15.6:0/4140649849 conn(0x7fab700b16c0 msgr2=0x7fab700b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:29.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.128+0000 7fab75000640 1 -- 172.21.15.6:0/4140649849 shutdown_connections 2024-03-22T13:07:29.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.128+0000 7fab75000640 1 -- 172.21.15.6:0/4140649849 wait complete. 2024-03-22T13:07:29.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.130+0000 7fab75000640 1 Processor -- start 2024-03-22T13:07:29.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.130+0000 7fab75000640 1 -- start start 2024-03-22T13:07:29.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.131+0000 7fab75000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 0x7fab701f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:29.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.131+0000 7fab75000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab70154980 0x7fab700b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:29.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.131+0000 7fab75000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab7015aef0 0x7fab70159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:29.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.131+0000 7fab75000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700ccf00 con 0x7fab700cd300 2024-03-22T13:07:29.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.131+0000 7fab75000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700ccc20 con 0x7fab7015aef0 2024-03-22T13:07:29.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.131+0000 7fab75000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700cea10 con 0x7fab70154980 2024-03-22T13:07:29.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.132+0000 7fab6d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab70154980 0x7fab700b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:29.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.132+0000 7fab6de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 0x7fab701f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:29.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.132+0000 7fab6e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab7015aef0 0x7fab70159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:29.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.132+0000 7fab6d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab70154980 0x7fab700b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:39668/0 (socket says 172.21.15.6:39668) 2024-03-22T13:07:29.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.132+0000 7fab6d400640 1 -- 172.21.15.6:0/4259844593 learned_addr learned my addr 172.21.15.6:0/4259844593 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:29.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.136+0000 7fab6d400640 1 -- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab7015aef0 msgr2=0x7fab70159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:29.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.136+0000 7fab6d400640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab7015aef0 0x7fab70159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.136+0000 7fab6d400640 1 -- 172.21.15.6:0/4259844593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 msgr2=0x7fab701f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:29.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.136+0000 7fab6d400640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 0x7fab701f0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.136+0000 7fab6d400640 1 -- 172.21.15.6:0/4259844593 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7015a130 con 0x7fab70154980 2024-03-22T13:07:29.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.139+0000 7fab6d400640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab70154980 0x7fab700b07b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009890 tx=0x7fab5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.143+0000 7fab66000640 1 -- 172.21.15.6:0/4259844593 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c034040 con 0x7fab70154980 2024-03-22T13:07:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.143+0000 7fab66000640 1 -- 172.21.15.6:0/4259844593 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c039990 con 0x7fab70154980 2024-03-22T13:07:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.143+0000 7fab66000640 1 -- 172.21.15.6:0/4259844593 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c039d60 con 0x7fab70154980 2024-03-22T13:07:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.143+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab70158470 con 0x7fab70154980 2024-03-22T13:07:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.143+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab70158a30 con 0x7fab70154980 2024-03-22T13:07:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.145+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab700756c0 con 0x7fab70154980 2024-03-22T13:07:29.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.441+0000 7fab66000640 1 -- 172.21.15.6:0/4259844593 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fab5c037070 con 0x7fab70154980 2024-03-22T13:07:29.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.443+0000 7fab66000640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab5006bc50 0x7fab5006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.443+0000 7fab6de00640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab5006bc50 0x7fab5006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.444+0000 7fab6de00640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab5006bc50 0x7fab5006e110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fab600011e0 tx=0x7fab60007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:29.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.446+0000 7fab66000640 1 -- 172.21.15.6:0/4259844593 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fab5c079910 con 0x7fab70154980 2024-03-22T13:07:29.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.460+0000 7fab66000640 1 -- 172.21.15.6:0/4259844593 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fab5c07b870 con 0x7fab70154980 2024-03-22T13:07:29.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.745+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab700b5b30 con 0x7fab70154980 2024-03-22T13:07:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.767+0000 7fab66000640 1 -- 172.21.15.6:0/4259844593 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fab5c07f910 con 0x7fab70154980 2024-03-22T13:07:29.767 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:29.767 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab5006bc50 msgr2=0x7fab5006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab5006bc50 0x7fab5006e110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fab600011e0 tx=0x7fab60007d30 comp rx=0 tx=0).stop 2024-03-22T13:07:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab70154980 msgr2=0x7fab700b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab70154980 0x7fab700b07b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009890 tx=0x7fab5c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 shutdown_connections 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab5006bc50 0x7fab5006e110 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab7015aef0 0x7fab70159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab70154980 0x7fab700b07b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 --2- 172.21.15.6:0/4259844593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab700cd300 0x7fab701f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.769+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 >> 172.21.15.6:0/4259844593 conn(0x7fab700b16c0 msgr2=0x7fab700744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.770+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 shutdown_connections 2024-03-22T13:07:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:29.770+0000 7fab75000640 1 -- 172.21.15.6:0/4259844593 wait complete. 2024-03-22T13:07:30.391 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:30.391 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:31.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:31.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.591+0000 7f0e47200640 1 Processor -- start 2024-03-22T13:07:31.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.591+0000 7f0e47200640 1 -- start start 2024-03-22T13:07:31.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.593+0000 7f0e47200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e480ce8a0 0x7f0e4806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:31.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.593+0000 7f0e47200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4815b240 con 0x7f0e480cc380 2024-03-22T13:07:31.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.593+0000 7f0e47200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e480cee00 con 0x7f0e4815ae30 2024-03-22T13:07:31.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.593+0000 7f0e47200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e480cf0c0 con 0x7f0e480ce8a0 2024-03-22T13:07:31.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.594+0000 7f0e46800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0e4815ae30 0x7f0e48164b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:54174/0 (socket says 172.21.15.6:54174) 2024-03-22T13:07:31.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.594+0000 7f0e46800640 1 -- 172.21.15.6:0/3019693622 learned_addr learned my addr 172.21.15.6:0/3019693622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:31.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.594+0000 7f0e45400640 1 --2- 172.21.15.6:0/3019693622 >> v2:172.21.15.6:3300/0 conn(0x7f0e480ce8a0 0x7f0e4806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:31.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.596+0000 7f0e44a00640 1 -- 172.21.15.6:0/3019693622 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513052105 0 0) 0x7f0e480cee00 con 0x7f0e4815ae30 2024-03-22T13:07:31.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.596+0000 7f0e44a00640 1 -- 172.21.15.6:0/3019693622 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e2c003520 con 0x7f0e4815ae30 2024-03-22T13:07:31.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.599+0000 7f0e45400640 1 -- 172.21.15.6:0/3019693622 >> v1:172.21.15.92:6789/0 conn(0x7f0e4815ae30 legacy=0x7f0e48164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:31.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.599+0000 7f0e45400640 1 -- 172.21.15.6:0/3019693622 >> v1:172.21.15.6:6789/0 conn(0x7f0e480cc380 legacy=0x7f0e480cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:31.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.599+0000 7f0e45400640 1 -- 172.21.15.6:0/3019693622 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e480cf3a0 con 0x7f0e480ce8a0 2024-03-22T13:07:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.606+0000 7f0e45400640 1 --2- 172.21.15.6:0/3019693622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 0x7f0e4806c230 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e340097a0 tx=0x7f0e3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd8371204afea88 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.612+0000 7f0e44a00640 1 -- 172.21.15.6:0/3019693622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34034070 con 0x7f0e480ce8a0 2024-03-22T13:07:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.613+0000 7f0e44a00640 1 -- 172.21.15.6:0/3019693622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e34030610 con 0x7f0e480ce8a0 2024-03-22T13:07:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.613+0000 7f0e44a00640 1 -- 172.21.15.6:0/3019693622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e340309e0 con 0x7f0e480ce8a0 2024-03-22T13:07:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.613+0000 7f0e47200640 1 -- 172.21.15.6:0/3019693622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 msgr2=0x7f0e4806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.613+0000 7f0e47200640 1 --2- 172.21.15.6:0/3019693622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 0x7f0e4806c230 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e340097a0 tx=0x7f0e3402b960 comp rx=0 tx=0).stop 2024-03-22T13:07:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.614+0000 7f0e47200640 1 -- 172.21.15.6:0/3019693622 shutdown_connections 2024-03-22T13:07:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.614+0000 7f0e47200640 1 --2- 172.21.15.6:0/3019693622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 0x7f0e4806c230 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.614+0000 7f0e47200640 1 -- 172.21.15.6:0/3019693622 >> 172.21.15.6:0/3019693622 conn(0x7f0e480b1480 msgr2=0x7f0e480b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.614+0000 7f0e47200640 1 -- 172.21.15.6:0/3019693622 shutdown_connections 2024-03-22T13:07:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.614+0000 7f0e47200640 1 -- 172.21.15.6:0/3019693622 wait complete. 2024-03-22T13:07:31.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.616+0000 7f0e47200640 1 Processor -- start 2024-03-22T13:07:31.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.616+0000 7f0e47200640 1 -- start start 2024-03-22T13:07:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.617+0000 7f0e47200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e480cc380 0x7f0e481533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.617+0000 7f0e47200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 0x7f0e481efff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.617+0000 7f0e47200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4815ae30 0x7f0e4814a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.617+0000 7f0e47200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4815a910 con 0x7f0e480ce8a0 2024-03-22T13:07:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.617+0000 7f0e47200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4815a630 con 0x7f0e480cc380 2024-03-22T13:07:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.617+0000 7f0e47200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481549d0 con 0x7f0e4815ae30 2024-03-22T13:07:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.617+0000 7f0e46800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4815ae30 0x7f0e4814a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.618+0000 7f0e45400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 0x7f0e481efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.618+0000 7f0e45e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e480cc380 0x7f0e481533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.618+0000 7f0e46800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4815ae30 0x7f0e4814a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:39680/0 (socket says 172.21.15.6:39680) 2024-03-22T13:07:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.618+0000 7f0e46800640 1 -- 172.21.15.6:0/2441161281 learned_addr learned my addr 172.21.15.6:0/2441161281 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.623+0000 7f0e46800640 1 -- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e480cc380 msgr2=0x7f0e481533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.623+0000 7f0e46800640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e480cc380 0x7f0e481533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:31.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.623+0000 7f0e46800640 1 -- 172.21.15.6:0/2441161281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 msgr2=0x7f0e481efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:31.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.623+0000 7f0e46800640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 0x7f0e481efff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:31.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.623+0000 7f0e46800640 1 -- 172.21.15.6:0/2441161281 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e481f4010 con 0x7f0e4815ae30 2024-03-22T13:07:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.626+0000 7f0e46800640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4815ae30 0x7f0e4814a6e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000cb70 tx=0x7f0e4000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.631+0000 7f0e3e000640 1 -- 172.21.15.6:0/2441161281 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40017070 con 0x7f0e4815ae30 2024-03-22T13:07:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.631+0000 7f0e3e000640 1 -- 172.21.15.6:0/2441161281 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4000ad40 con 0x7f0e4815ae30 2024-03-22T13:07:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.631+0000 7f0e3e000640 1 -- 172.21.15.6:0/2441161281 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40012620 con 0x7f0e4815ae30 2024-03-22T13:07:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.631+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e481f42f0 con 0x7f0e4815ae30 2024-03-22T13:07:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.631+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e481e7730 con 0x7f0e4815ae30 2024-03-22T13:07:31.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.633+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e48075650 con 0x7f0e4815ae30 2024-03-22T13:07:31.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.922+0000 7f0e3e000640 1 -- 172.21.15.6:0/2441161281 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0e4001a070 con 0x7f0e4815ae30 2024-03-22T13:07:31.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.924+0000 7f0e3e000640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2806bc50 0x7f0e2806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:31.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.925+0000 7f0e45e00640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2806bc50 0x7f0e2806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:31.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.926+0000 7f0e45e00640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2806bc50 0x7f0e2806e110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0e380016e0 tx=0x7f0e38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:31.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.927+0000 7f0e3e000640 1 -- 172.21.15.6:0/2441161281 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0e400571a0 con 0x7f0e4815ae30 2024-03-22T13:07:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:31.942+0000 7f0e3e000640 1 -- 172.21.15.6:0/2441161281 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0e40021020 con 0x7f0e4815ae30 2024-03-22T13:07:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.198+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e4806de60 con 0x7f0e4815ae30 2024-03-22T13:07:32.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.229+0000 7f0e3e000640 1 -- 172.21.15.6:0/2441161281 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f0e40059100 con 0x7f0e4815ae30 2024-03-22T13:07:32.229 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:32.229 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:32.230 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.233+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2806bc50 msgr2=0x7f0e2806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.233+0000 7f0e47200640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2806bc50 0x7f0e2806e110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0e380016e0 tx=0x7f0e38009040 comp rx=0 tx=0).stop 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.233+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4815ae30 msgr2=0x7f0e4814a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.233+0000 7f0e47200640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4815ae30 0x7f0e4814a6e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000cb70 tx=0x7f0e4000a430 comp rx=0 tx=0).stop 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.233+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 shutdown_connections 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.234+0000 7f0e47200640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4815ae30 0x7f0e4814a6e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.234+0000 7f0e47200640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e480ce8a0 0x7f0e481efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.234+0000 7f0e47200640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2806bc50 0x7f0e2806e110 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.234+0000 7f0e47200640 1 --2- 172.21.15.6:0/2441161281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e480cc380 0x7f0e481533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.234+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 >> 172.21.15.6:0/2441161281 conn(0x7f0e480b1480 msgr2=0x7f0e480cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.234+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 shutdown_connections 2024-03-22T13:07:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:32.234+0000 7f0e47200640 1 -- 172.21.15.6:0/2441161281 wait complete. 2024-03-22T13:07:32.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:07:32.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:07:32.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:32.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:33.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.045+0000 7f1b51600640 1 Processor -- start 2024-03-22T13:07:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.045+0000 7f1b51600640 1 -- start start 2024-03-22T13:07:34.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.046+0000 7f1b51600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:34.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.046+0000 7f1b51600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b4c0ce8a0 0x7f1b4c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:34.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.046+0000 7f1b51600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4c0cee00 con 0x7f1b4c0cc760 2024-03-22T13:07:34.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.046+0000 7f1b51600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4c0cf0c0 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.046+0000 7f1b51600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c0cf3a0 con 0x7f1b4c0cc380 2024-03-22T13:07:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.047+0000 7f1b4be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.047+0000 7f1b4be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c06c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:55586/0 (socket says 172.21.15.6:55586) 2024-03-22T13:07:34.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.047+0000 7f1b4be00640 1 -- 172.21.15.6:0/3974063470 learned_addr learned my addr 172.21.15.6:0/3974063470 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:34.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.048+0000 7f1b4b400640 1 --2- 172.21.15.6:0/3974063470 >> v2:172.21.15.6:3300/0 conn(0x7f1b4c0ce8a0 0x7f1b4c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:34.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.050+0000 7f1b4be00640 1 -- 172.21.15.6:0/3974063470 >> v2:172.21.15.6:3300/0 conn(0x7f1b4c0ce8a0 msgr2=0x7f1b4c15ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.050+0000 7f1b4be00640 1 --2- 172.21.15.6:0/3974063470 >> v2:172.21.15.6:3300/0 conn(0x7f1b4c0ce8a0 0x7f1b4c15ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.050+0000 7f1b4be00640 1 -- 172.21.15.6:0/3974063470 >> v1:172.21.15.6:6789/0 conn(0x7f1b4c0cc760 legacy=0x7f1b4c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.050+0000 7f1b4be00640 1 -- 172.21.15.6:0/3974063470 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c0cf680 con 0x7f1b4c0cc380 2024-03-22T13:07:34.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.052+0000 7f1b4be00640 1 --2- 172.21.15.6:0/3974063470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c06c230 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009870 tx=0x7f1b3c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf63d6ec09da4970 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:34.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b4aa00640 1 -- 172.21.15.6:0/3974063470 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c034070 con 0x7f1b4c0cc380 2024-03-22T13:07:34.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b4aa00640 1 -- 172.21.15.6:0/3974063470 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c030840 con 0x7f1b4c0cc380 2024-03-22T13:07:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b4aa00640 1 -- 172.21.15.6:0/3974063470 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c030c10 con 0x7f1b4c0cc380 2024-03-22T13:07:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b51600640 1 -- 172.21.15.6:0/3974063470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 msgr2=0x7f1b4c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:34.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b51600640 1 --2- 172.21.15.6:0/3974063470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c06c230 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009870 tx=0x7f1b3c02b980 comp rx=0 tx=0).stop 2024-03-22T13:07:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b51600640 1 -- 172.21.15.6:0/3974063470 shutdown_connections 2024-03-22T13:07:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b51600640 1 --2- 172.21.15.6:0/3974063470 >> v2:172.21.15.6:3300/0 conn(0x7f1b4c0ce8a0 0x7f1b4c15ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b51600640 1 --2- 172.21.15.6:0/3974063470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c06c230 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.058+0000 7f1b51600640 1 -- 172.21.15.6:0/3974063470 >> 172.21.15.6:0/3974063470 conn(0x7f1b4c0b1480 msgr2=0x7f1b4c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.059+0000 7f1b51600640 1 -- 172.21.15.6:0/3974063470 shutdown_connections 2024-03-22T13:07:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.059+0000 7f1b51600640 1 -- 172.21.15.6:0/3974063470 wait complete. 2024-03-22T13:07:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.061+0000 7f1b51600640 1 Processor -- start 2024-03-22T13:07:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.062+0000 7f1b51600640 1 -- start start 2024-03-22T13:07:34.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.063+0000 7f1b51600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c1f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:34.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.063+0000 7f1b51600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b4c0cc760 0x7f1b4c0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:34.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.063+0000 7f1b51600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4c0ce8a0 0x7f1b4c06c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:34.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.063+0000 7f1b51600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c1549d0 con 0x7f1b4c0cc760 2024-03-22T13:07:34.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.063+0000 7f1b51600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c15a910 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.063+0000 7f1b51600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c0ccb90 con 0x7f1b4c0cc380 2024-03-22T13:07:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.064+0000 7f1b50c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4c0ce8a0 0x7f1b4c06c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.064+0000 7f1b4b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b4c0cc760 0x7f1b4c0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.064+0000 7f1b4b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b4c0cc760 0x7f1b4c0b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48128/0 (socket says 172.21.15.6:48128) 2024-03-22T13:07:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.064+0000 7f1b4b400640 1 -- 172.21.15.6:0/2398328086 learned_addr learned my addr 172.21.15.6:0/2398328086 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.064+0000 7f1b4be00640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c1f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.069+0000 7f1b50c00640 1 -- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 msgr2=0x7f1b4c1f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.069+0000 7f1b50c00640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c1f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.069+0000 7f1b50c00640 1 -- 172.21.15.6:0/2398328086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b4c0cc760 msgr2=0x7f1b4c0b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.069+0000 7f1b50c00640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b4c0cc760 0x7f1b4c0b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.069+0000 7f1b50c00640 1 -- 172.21.15.6:0/2398328086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c1e55c0 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.073+0000 7f1b50c00640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4c0ce8a0 0x7f1b4c06c820 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c690 tx=0x7f1b4400cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:34.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.080+0000 7f1b43e00640 1 -- 172.21.15.6:0/2398328086 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44003b70 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.080+0000 7f1b43e00640 1 -- 172.21.15.6:0/2398328086 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44003d10 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.080+0000 7f1b43e00640 1 -- 172.21.15.6:0/2398328086 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44005310 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.080+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c1e4350 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.080+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c1e3ab0 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.081+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4c0b1dc0 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.362+0000 7f1b43e00640 1 -- 172.21.15.6:0/2398328086 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b440054b0 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.364+0000 7f1b43e00640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2c06bc50 0x7f1b2c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.365+0000 7f1b43e00640 1 -- 172.21.15.6:0/2398328086 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1b4408b540 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.365+0000 7f1b4be00640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2c06bc50 0x7f1b2c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:34.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.366+0000 7f1b4be00640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2c06bc50 0x7f1b2c06e110 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c036000 tx=0x7f1b3c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:34.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.384+0000 7f1b43e00640 1 -- 172.21.15.6:0/2398328086 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b4c0b1dc0 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.633+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b4c075650 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.648+0000 7f1b43e00640 1 -- 172.21.15.6:0/2398328086 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f1b440596c0 con 0x7f1b4c0ce8a0 2024-03-22T13:07:34.648 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:34.648 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:34.648 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:34.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.650+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2c06bc50 msgr2=0x7f1b2c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.650+0000 7f1b51600640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2c06bc50 0x7f1b2c06e110 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c036000 tx=0x7f1b3c03b040 comp rx=0 tx=0).stop 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.650+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4c0ce8a0 msgr2=0x7f1b4c06c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.650+0000 7f1b51600640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4c0ce8a0 0x7f1b4c06c820 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400c690 tx=0x7f1b4400cb50 comp rx=0 tx=0).stop 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 shutdown_connections 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2c06bc50 0x7f1b2c06e110 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4c0ce8a0 0x7f1b4c06c820 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b4c0cc760 0x7f1b4c0b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 --2- 172.21.15.6:0/2398328086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4c0cc380 0x7f1b4c1f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 >> 172.21.15.6:0/2398328086 conn(0x7f1b4c0b1480 msgr2=0x7f1b4c0ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 shutdown_connections 2024-03-22T13:07:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:34.651+0000 7f1b51600640 1 -- 172.21.15.6:0/2398328086 wait complete. 2024-03-22T13:07:35.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:35.273 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:36.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:36.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.484+0000 7ff38be00640 1 Processor -- start 2024-03-22T13:07:36.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.484+0000 7ff38be00640 1 -- start start 2024-03-22T13:07:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.485+0000 7ff38be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff384152400 0x7ff38415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.485+0000 7ff38be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff38406f8c0 0x7ff3840c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.485+0000 7ff38be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3841590b0 con 0x7ff38415bc80 2024-03-22T13:07:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.485+0000 7ff38be00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3840724d0 con 0x7ff38406f8c0 2024-03-22T13:07:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.485+0000 7ff38be00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840721f0 con 0x7ff384152400 2024-03-22T13:07:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.486+0000 7ff389e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff38406f8c0 0x7ff3840c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.486+0000 7ff389400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff384152400 0x7ff38415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.487+0000 7ff389e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff38406f8c0 0x7ff3840c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60318/0 (socket says 172.21.15.6:60318) 2024-03-22T13:07:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.487+0000 7ff389e00640 1 -- 172.21.15.6:0/2124029450 learned_addr learned my addr 172.21.15.6:0/2124029450 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.489+0000 7ff389e00640 1 -- 172.21.15.6:0/2124029450 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff384152400 msgr2=0x7ff38415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.489+0000 7ff389e00640 1 --2- 172.21.15.6:0/2124029450 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff384152400 0x7ff38415b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.489+0000 7ff389e00640 1 -- 172.21.15.6:0/2124029450 >> v1:172.21.15.6:6789/0 conn(0x7ff38415bc80 legacy=0x7ff3840716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.489+0000 7ff389e00640 1 -- 172.21.15.6:0/2124029450 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3841582f0 con 0x7ff38406f8c0 2024-03-22T13:07:36.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.491+0000 7ff389e00640 1 --2- 172.21.15.6:0/2124029450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff38406f8c0 0x7ff3840c5040 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009e50 tx=0x7ff37c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b27f64cac97ab4d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.495+0000 7ff383e00640 1 -- 172.21.15.6:0/2124029450 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c02cb00 con 0x7ff38406f8c0 2024-03-22T13:07:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.495+0000 7ff383e00640 1 -- 172.21.15.6:0/2124029450 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c02cca0 con 0x7ff38406f8c0 2024-03-22T13:07:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.495+0000 7ff383e00640 1 -- 172.21.15.6:0/2124029450 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c034610 con 0x7ff38406f8c0 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 -- 172.21.15.6:0/2124029450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff38406f8c0 msgr2=0x7ff3840c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 --2- 172.21.15.6:0/2124029450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff38406f8c0 0x7ff3840c5040 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009e50 tx=0x7ff37c02bf60 comp rx=0 tx=0).stop 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 -- 172.21.15.6:0/2124029450 shutdown_connections 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 --2- 172.21.15.6:0/2124029450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff38406f8c0 0x7ff3840c5040 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 --2- 172.21.15.6:0/2124029450 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff384152400 0x7ff38415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 -- 172.21.15.6:0/2124029450 >> 172.21.15.6:0/2124029450 conn(0x7ff3840cb930 msgr2=0x7ff3840cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 -- 172.21.15.6:0/2124029450 shutdown_connections 2024-03-22T13:07:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.496+0000 7ff38be00640 1 -- 172.21.15.6:0/2124029450 wait complete. 2024-03-22T13:07:36.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.498+0000 7ff38be00640 1 Processor -- start 2024-03-22T13:07:36.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.498+0000 7ff38be00640 1 -- start start 2024-03-22T13:07:36.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.499+0000 7ff38be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38406f8c0 0x7ff38414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:36.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.499+0000 7ff38be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff384152400 0x7ff38414d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:36.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.499+0000 7ff38be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff38415bc80 0x7ff38414dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:36.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.499+0000 7ff38be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff384158b90 con 0x7ff38406f8c0 2024-03-22T13:07:36.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.499+0000 7ff38be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841588b0 con 0x7ff384152400 2024-03-22T13:07:36.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.499+0000 7ff38be00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841527e0 con 0x7ff38415bc80 2024-03-22T13:07:36.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.500+0000 7ff388a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff384152400 0x7ff38414d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:36.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.500+0000 7ff389e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff38415bc80 0x7ff38414dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:36.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.500+0000 7ff388a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff384152400 0x7ff38414d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60324/0 (socket says 172.21.15.6:60324) 2024-03-22T13:07:36.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.500+0000 7ff388a00640 1 -- 172.21.15.6:0/1352382044 learned_addr learned my addr 172.21.15.6:0/1352382044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:36.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.505+0000 7ff388a00640 1 -- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff38415bc80 msgr2=0x7ff38414dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:36.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.505+0000 7ff388a00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff38415bc80 0x7ff38414dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:36.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.505+0000 7ff388a00640 1 -- 172.21.15.6:0/1352382044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38406f8c0 msgr2=0x7ff38414cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:36.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.505+0000 7ff388a00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38406f8c0 0x7ff38414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:36.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.505+0000 7ff388a00640 1 -- 172.21.15.6:0/1352382044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff384159a60 con 0x7ff384152400 2024-03-22T13:07:36.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.509+0000 7ff388a00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff384152400 0x7ff38414d590 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff37000b3c0 tx=0x7ff37000b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:36.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.516+0000 7ff381600640 1 -- 172.21.15.6:0/1352382044 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370011830 con 0x7ff384152400 2024-03-22T13:07:36.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.516+0000 7ff381600640 1 -- 172.21.15.6:0/1352382044 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff370016070 con 0x7ff384152400 2024-03-22T13:07:36.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.516+0000 7ff381600640 1 -- 172.21.15.6:0/1352382044 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370011c30 con 0x7ff384152400 2024-03-22T13:07:36.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.516+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38414e0d0 con 0x7ff384152400 2024-03-22T13:07:36.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.516+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff384156ce0 con 0x7ff384152400 2024-03-22T13:07:36.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.517+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3840756c0 con 0x7ff384152400 2024-03-22T13:07:36.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.840+0000 7ff381600640 1 -- 172.21.15.6:0/1352382044 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff37001b460 con 0x7ff384152400 2024-03-22T13:07:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.843+0000 7ff381600640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff36406bc20 0x7ff36406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.843+0000 7ff389400640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff36406bc20 0x7ff36406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.844+0000 7ff389400640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff36406bc20 0x7ff36406e0e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff378006a90 tx=0x7ff378006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:36.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.846+0000 7ff381600640 1 -- 172.21.15.6:0/1352382044 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff370057650 con 0x7ff384152400 2024-03-22T13:07:36.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:36.862+0000 7ff381600640 1 -- 172.21.15.6:0/1352382044 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff384159a60 con 0x7ff384152400 2024-03-22T13:07:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.108+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3840c6400 con 0x7ff384152400 2024-03-22T13:07:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.123+0000 7ff381600640 1 -- 172.21.15.6:0/1352382044 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7ff370059cb0 con 0x7ff384152400 2024-03-22T13:07:37.123 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:37.123 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff36406bc20 msgr2=0x7ff36406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff36406bc20 0x7ff36406e0e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff378006a90 tx=0x7ff378006a20 comp rx=0 tx=0).stop 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff384152400 msgr2=0x7ff38414d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff384152400 0x7ff38414d590 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff37000b3c0 tx=0x7ff37000b880 comp rx=0 tx=0).stop 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 shutdown_connections 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff38415bc80 0x7ff38414dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff36406bc20 0x7ff36406e0e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff384152400 0x7ff38414d590 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 --2- 172.21.15.6:0/1352382044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38406f8c0 0x7ff38414cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 >> 172.21.15.6:0/1352382044 conn(0x7ff3840cb930 msgr2=0x7ff38414fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 shutdown_connections 2024-03-22T13:07:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:37.126+0000 7ff38be00640 1 -- 172.21.15.6:0/1352382044 wait complete. 2024-03-22T13:07:37.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:37.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:38.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:38.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.942+0000 7fe581600640 1 Processor -- start 2024-03-22T13:07:38.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.943+0000 7fe581600640 1 -- start start 2024-03-22T13:07:38.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.944+0000 7fe581600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe57c152400 0x7fe57c15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:38.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.944+0000 7fe581600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c06f8c0 0x7fe57c0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:38.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.944+0000 7fe581600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c1590b0 con 0x7fe57c15bc80 2024-03-22T13:07:38.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.944+0000 7fe581600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57c0724d0 con 0x7fe57c152400 2024-03-22T13:07:38.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.944+0000 7fe581600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0721f0 con 0x7fe57c06f8c0 2024-03-22T13:07:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.945+0000 7fe57b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c06f8c0 0x7fe57c0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.945+0000 7fe57a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe57c15bc80 0x7fe57c0716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46628/0 (socket says 172.21.15.6:46628) 2024-03-22T13:07:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.945+0000 7fe57a000640 1 -- 172.21.15.6:0/2566108329 learned_addr learned my addr 172.21.15.6:0/2566108329 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:38.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.945+0000 7fe57aa00640 1 --2- 172.21.15.6:0/2566108329 >> v2:172.21.15.6:3300/0 conn(0x7fe57c152400 0x7fe57c15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:38.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.949+0000 7fe57b400640 1 -- 172.21.15.6:0/2566108329 >> v2:172.21.15.6:3300/0 conn(0x7fe57c152400 msgr2=0x7fe57c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:38.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.949+0000 7fe57b400640 1 --2- 172.21.15.6:0/2566108329 >> v2:172.21.15.6:3300/0 conn(0x7fe57c152400 0x7fe57c15b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:38.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.949+0000 7fe57b400640 1 -- 172.21.15.6:0/2566108329 >> v1:172.21.15.6:6789/0 conn(0x7fe57c15bc80 legacy=0x7fe57c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:38.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.949+0000 7fe57b400640 1 -- 172.21.15.6:0/2566108329 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c1582f0 con 0x7fe57c06f8c0 2024-03-22T13:07:38.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.952+0000 7fe57b400640 1 --2- 172.21.15.6:0/2566108329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c06f8c0 0x7fe57c0c5040 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe574009d90 tx=0x7fe57402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc72d7db3daa8671 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:38.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.956+0000 7fe579600640 1 -- 172.21.15.6:0/2566108329 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574039070 con 0x7fe57c06f8c0 2024-03-22T13:07:38.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.956+0000 7fe579600640 1 -- 172.21.15.6:0/2566108329 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57402cce0 con 0x7fe57c06f8c0 2024-03-22T13:07:38.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.956+0000 7fe579600640 1 -- 172.21.15.6:0/2566108329 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574034660 con 0x7fe57c06f8c0 2024-03-22T13:07:38.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.957+0000 7fe581600640 1 -- 172.21.15.6:0/2566108329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c06f8c0 msgr2=0x7fe57c0c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:38.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.957+0000 7fe581600640 1 --2- 172.21.15.6:0/2566108329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c06f8c0 0x7fe57c0c5040 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe574009d90 tx=0x7fe57402bf30 comp rx=0 tx=0).stop 2024-03-22T13:07:38.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.957+0000 7fe581600640 1 -- 172.21.15.6:0/2566108329 shutdown_connections 2024-03-22T13:07:38.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.957+0000 7fe581600640 1 --2- 172.21.15.6:0/2566108329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c06f8c0 0x7fe57c0c5040 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:38.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.957+0000 7fe581600640 1 --2- 172.21.15.6:0/2566108329 >> v2:172.21.15.6:3300/0 conn(0x7fe57c152400 0x7fe57c15b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:38.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.958+0000 7fe581600640 1 -- 172.21.15.6:0/2566108329 >> 172.21.15.6:0/2566108329 conn(0x7fe57c0cb930 msgr2=0x7fe57c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:38.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.958+0000 7fe581600640 1 -- 172.21.15.6:0/2566108329 shutdown_connections 2024-03-22T13:07:38.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.958+0000 7fe581600640 1 -- 172.21.15.6:0/2566108329 wait complete. 2024-03-22T13:07:38.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.959+0000 7fe581600640 1 Processor -- start 2024-03-22T13:07:38.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.959+0000 7fe581600640 1 -- start start 2024-03-22T13:07:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.960+0000 7fe581600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe57c06f8c0 0x7fe57c1f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.960+0000 7fe581600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c152400 0x7fe57c0cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.960+0000 7fe581600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe57c15bc80 0x7fe57c159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.960+0000 7fe581600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c158b90 con 0x7fe57c06f8c0 2024-03-22T13:07:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.960+0000 7fe581600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c1588b0 con 0x7fe57c15bc80 2024-03-22T13:07:38.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.960+0000 7fe581600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c1527e0 con 0x7fe57c152400 2024-03-22T13:07:38.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.961+0000 7fe57aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe57c06f8c0 0x7fe57c1f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:38.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.961+0000 7fe57a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c152400 0x7fe57c0cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:38.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.961+0000 7fe57b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe57c15bc80 0x7fe57c159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:38.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.961+0000 7fe57aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe57c06f8c0 0x7fe57c1f3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48138/0 (socket says 172.21.15.6:48138) 2024-03-22T13:07:38.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.961+0000 7fe57aa00640 1 -- 172.21.15.6:0/355086660 learned_addr learned my addr 172.21.15.6:0/355086660 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.965+0000 7fe57aa00640 1 -- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c152400 msgr2=0x7fe57c0cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.965+0000 7fe57aa00640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c152400 0x7fe57c0cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.965+0000 7fe57aa00640 1 -- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe57c15bc80 msgr2=0x7fe57c159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.965+0000 7fe57aa00640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe57c15bc80 0x7fe57c159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.965+0000 7fe57aa00640 1 -- 172.21.15.6:0/355086660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c156400 con 0x7fe57c06f8c0 2024-03-22T13:07:38.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.968+0000 7fe57aa00640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe57c06f8c0 0x7fe57c1f3fd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e9b0 tx=0x7fe56c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.973+0000 7fe572a00640 1 -- 172.21.15.6:0/355086660 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c017070 con 0x7fe57c06f8c0 2024-03-22T13:07:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.973+0000 7fe572a00640 1 -- 172.21.15.6:0/355086660 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c00c9a0 con 0x7fe57c06f8c0 2024-03-22T13:07:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.973+0000 7fe572a00640 1 -- 172.21.15.6:0/355086660 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c00cd90 con 0x7fe57c06f8c0 2024-03-22T13:07:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.973+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c1f0360 con 0x7fe57c06f8c0 2024-03-22T13:07:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.973+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c1f0920 con 0x7fe57c06f8c0 2024-03-22T13:07:38.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.974+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c0756c0 con 0x7fe57c06f8c0 2024-03-22T13:07:38.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.997+0000 7fe572a00640 1 -- 172.21.15.6:0/355086660 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe56c01a070 con 0x7fe57c06f8c0 2024-03-22T13:07:38.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.999+0000 7fe572a00640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe55c06bc20 0x7fe55c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:38.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:38.999+0000 7fe57a000640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe55c06bc20 0x7fe55c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:38.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.000+0000 7fe57a000640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe55c06bc20 0x7fe55c06e0e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe57c155cf0 tx=0x7fe568007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.284+0000 7fe572a00640 1 -- 172.21.15.6:0/355086660 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe56c0576d0 con 0x7fe57c06f8c0 2024-03-22T13:07:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.284+0000 7fe572a00640 1 -- 172.21.15.6:0/355086660 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe56c059630 con 0x7fe57c06f8c0 2024-03-22T13:07:39.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.541+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe57c151c20 con 0x7fe57c06f8c0 2024-03-22T13:07:39.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.615+0000 7fe572a00640 1 -- 172.21.15.6:0/355086660 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fe56c05d6d0 con 0x7fe57c06f8c0 2024-03-22T13:07:39.615 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:39.615 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:39.615 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.618+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe55c06bc20 msgr2=0x7fe55c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.618+0000 7fe581600640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe55c06bc20 0x7fe55c06e0e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe57c155cf0 tx=0x7fe568007e80 comp rx=0 tx=0).stop 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.618+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe57c06f8c0 msgr2=0x7fe57c1f3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.618+0000 7fe581600640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe57c06f8c0 0x7fe57c1f3fd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e9b0 tx=0x7fe56c00ee70 comp rx=0 tx=0).stop 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 shutdown_connections 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe55c06bc20 0x7fe55c06e0e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe57c15bc80 0x7fe57c159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe57c152400 0x7fe57c0cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 --2- 172.21.15.6:0/355086660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe57c06f8c0 0x7fe57c1f3fd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 >> 172.21.15.6:0/355086660 conn(0x7fe57c0cb930 msgr2=0x7fe57c06b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 shutdown_connections 2024-03-22T13:07:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:39.619+0000 7fe581600640 1 -- 172.21.15.6:0/355086660 wait complete. 2024-03-22T13:07:40.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:40.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:41.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:41.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.441+0000 7f5b33200640 1 Processor -- start 2024-03-22T13:07:41.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.441+0000 7f5b33200640 1 -- start start 2024-03-22T13:07:41.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.442+0000 7f5b33200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5b340cc380 0x7f5b3406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:41.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.442+0000 7f5b33200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b3415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:41.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.442+0000 7f5b33200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b340cee00 con 0x7f5b340cc760 2024-03-22T13:07:41.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.442+0000 7f5b33200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b340cf0c0 con 0x7f5b340cc380 2024-03-22T13:07:41.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.442+0000 7f5b33200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b340cf3a0 con 0x7f5b340ce8a0 2024-03-22T13:07:41.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.443+0000 7f5b29400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b3415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:41.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.443+0000 7f5b31e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5b340cc380 0x7f5b3406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:41.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.443+0000 7f5b32800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5b340cc760 0x7f5b340cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46636/0 (socket says 172.21.15.6:46636) 2024-03-22T13:07:41.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.443+0000 7f5b32800640 1 -- 172.21.15.6:0/3395473560 learned_addr learned my addr 172.21.15.6:0/3395473560 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.446+0000 7f5b31e00640 1 -- 172.21.15.6:0/3395473560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 msgr2=0x7f5b3415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.446+0000 7f5b31e00640 1 --2- 172.21.15.6:0/3395473560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b3415ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.446+0000 7f5b31e00640 1 -- 172.21.15.6:0/3395473560 >> v1:172.21.15.6:6789/0 conn(0x7f5b340cc760 legacy=0x7f5b340cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.446+0000 7f5b31e00640 1 -- 172.21.15.6:0/3395473560 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b340cf680 con 0x7f5b340cc380 2024-03-22T13:07:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.448+0000 7f5b31e00640 1 --2- 172.21.15.6:0/3395473560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc380 0x7f5b3406c230 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009d90 tx=0x7f5b2402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d37b1d4f219c631 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.451+0000 7f5b31400640 1 -- 172.21.15.6:0/3395473560 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24039070 con 0x7f5b340cc380 2024-03-22T13:07:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b31400640 1 -- 172.21.15.6:0/3395473560 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2402cc60 con 0x7f5b340cc380 2024-03-22T13:07:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b31400640 1 -- 172.21.15.6:0/3395473560 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24034660 con 0x7f5b340cc380 2024-03-22T13:07:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b33200640 1 -- 172.21.15.6:0/3395473560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc380 msgr2=0x7f5b3406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b33200640 1 --2- 172.21.15.6:0/3395473560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc380 0x7f5b3406c230 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009d90 tx=0x7f5b2402beb0 comp rx=0 tx=0).stop 2024-03-22T13:07:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b33200640 1 -- 172.21.15.6:0/3395473560 shutdown_connections 2024-03-22T13:07:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b33200640 1 --2- 172.21.15.6:0/3395473560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b3415ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b33200640 1 --2- 172.21.15.6:0/3395473560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc380 0x7f5b3406c230 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b33200640 1 -- 172.21.15.6:0/3395473560 >> 172.21.15.6:0/3395473560 conn(0x7f5b340b1480 msgr2=0x7f5b340b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.454+0000 7f5b33200640 1 -- 172.21.15.6:0/3395473560 shutdown_connections 2024-03-22T13:07:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.455+0000 7f5b33200640 1 -- 172.21.15.6:0/3395473560 wait complete. 2024-03-22T13:07:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.456+0000 7f5b33200640 1 Processor -- start 2024-03-22T13:07:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.456+0000 7f5b33200640 1 -- start start 2024-03-22T13:07:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.457+0000 7f5b33200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b340cc380 0x7f5b341e3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.457+0000 7f5b33200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc760 0x7f5b340c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.457+0000 7f5b33200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b340c1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.457+0000 7f5b33200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b341549d0 con 0x7f5b340cc380 2024-03-22T13:07:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.457+0000 7f5b33200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3415a910 con 0x7f5b340cc760 2024-03-22T13:07:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.457+0000 7f5b33200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b340ccb90 con 0x7f5b340ce8a0 2024-03-22T13:07:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.458+0000 7f5b29400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc760 0x7f5b340c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.458+0000 7f5b32800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b340c1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.458+0000 7f5b31e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b340cc380 0x7f5b341e3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.458+0000 7f5b29400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc760 0x7f5b340c0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:60342/0 (socket says 172.21.15.6:60342) 2024-03-22T13:07:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.458+0000 7f5b29400640 1 -- 172.21.15.6:0/4240070429 learned_addr learned my addr 172.21.15.6:0/4240070429 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:41.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.463+0000 7f5b29400640 1 -- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 msgr2=0x7f5b340c1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:41.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.463+0000 7f5b29400640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b340c1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:41.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.463+0000 7f5b29400640 1 -- 172.21.15.6:0/4240070429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b340cc380 msgr2=0x7f5b341e3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:41.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.463+0000 7f5b29400640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b340cc380 0x7f5b341e3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:41.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.463+0000 7f5b29400640 1 -- 172.21.15.6:0/4240070429 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3415bfb0 con 0x7f5b340cc760 2024-03-22T13:07:41.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.466+0000 7f5b29400640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc760 0x7f5b340c0ef0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00e980 tx=0x7f5b1c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:41.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.469+0000 7f5b2aa00640 1 -- 172.21.15.6:0/4240070429 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c017070 con 0x7f5b340cc760 2024-03-22T13:07:41.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.469+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3415d290 con 0x7f5b340cc760 2024-03-22T13:07:41.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.469+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3415d850 con 0x7f5b340cc760 2024-03-22T13:07:41.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.471+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b34075650 con 0x7f5b340cc760 2024-03-22T13:07:41.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.681+0000 7f5b2aa00640 1 -- 172.21.15.6:0/4240070429 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c00ca00 con 0x7f5b340cc760 2024-03-22T13:07:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.753+0000 7f5b2aa00640 1 -- 172.21.15.6:0/4240070429 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c021060 con 0x7f5b340cc760 2024-03-22T13:07:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.756+0000 7f5b2aa00640 1 -- 172.21.15.6:0/4240070429 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b1c01a070 con 0x7f5b340cc760 2024-03-22T13:07:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.758+0000 7f5b2aa00640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b1006bbd0 0x7f5b1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.759+0000 7f5b2aa00640 1 -- 172.21.15.6:0/4240070429 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5b1c057740 con 0x7f5b340cc760 2024-03-22T13:07:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.759+0000 7f5b31e00640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b1006bbd0 0x7f5b1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.760+0000 7f5b31e00640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b1006bbd0 0x7f5b1006e090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5b24036000 tx=0x7f5b24030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:41.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:41.770+0000 7f5b2aa00640 1 -- 172.21.15.6:0/4240070429 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b3415bfb0 con 0x7f5b340cc760 2024-03-22T13:07:42.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.019+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b34151ca0 con 0x7f5b340cc760 2024-03-22T13:07:42.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.040+0000 7f5b2aa00640 1 -- 172.21.15.6:0/4240070429 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f5b1c0596a0 con 0x7f5b340cc760 2024-03-22T13:07:42.040 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:42.040 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:42.040 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b1006bbd0 msgr2=0x7f5b1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b1006bbd0 0x7f5b1006e090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5b24036000 tx=0x7f5b24030000 comp rx=0 tx=0).stop 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc760 msgr2=0x7f5b340c0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc760 0x7f5b340c0ef0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00e980 tx=0x7f5b1c00ee40 comp rx=0 tx=0).stop 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 shutdown_connections 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b340ce8a0 0x7f5b340c1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b1006bbd0 0x7f5b1006e090 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b340cc760 0x7f5b340c0ef0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 --2- 172.21.15.6:0/4240070429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b340cc380 0x7f5b341e3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 >> 172.21.15.6:0/4240070429 conn(0x7f5b340b1480 msgr2=0x7f5b340cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 shutdown_connections 2024-03-22T13:07:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:42.043+0000 7f5b33200640 1 -- 172.21.15.6:0/4240070429 wait complete. 2024-03-22T13:07:42.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:42.665 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:43.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:43.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.869+0000 7f395cc00640 1 Processor -- start 2024-03-22T13:07:43.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.869+0000 7f395cc00640 1 -- start start 2024-03-22T13:07:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.870+0000 7f395cc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cc640 0x7f395806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.870+0000 7f395cc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f39580cca20 0x7f39580ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.870+0000 7f395cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f395815b7a0 0x7f395815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.870+0000 7f395cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395806d330 con 0x7f395815b7a0 2024-03-22T13:07:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.870+0000 7f395cc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395806dd00 con 0x7f39580cca20 2024-03-22T13:07:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.870+0000 7f395cc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395806c440 con 0x7f39580cc640 2024-03-22T13:07:43.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.884+0000 7f3956a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f39580cca20 0x7f39580ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:43.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.884+0000 7f3956a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f39580cca20 0x7f39580ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51768/0 (socket says 172.21.15.6:51768) 2024-03-22T13:07:43.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.884+0000 7f3956a00640 1 -- 172.21.15.6:0/1854065723 learned_addr learned my addr 172.21.15.6:0/1854065723 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:43.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.887+0000 7f3956a00640 1 -- 172.21.15.6:0/1854065723 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cc640 msgr2=0x7f395806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:43.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.887+0000 7f3956a00640 1 --2- 172.21.15.6:0/1854065723 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cc640 0x7f395806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.888+0000 7f3956a00640 1 -- 172.21.15.6:0/1854065723 >> v2:172.21.15.6:3300/0 conn(0x7f395815b7a0 msgr2=0x7f395815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:43.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.888+0000 7f3956a00640 1 --2- 172.21.15.6:0/1854065723 >> v2:172.21.15.6:3300/0 conn(0x7f395815b7a0 0x7f395815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.888+0000 7f3956a00640 1 -- 172.21.15.6:0/1854065723 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395806c720 con 0x7f39580cca20 2024-03-22T13:07:43.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.890+0000 7f3956a00640 1 --2- 172.21.15.6:0/1854065723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39580cca20 0x7f39580ca340 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f395808d3b0 tx=0x7f394402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17973f9b37454f4d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:43.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.894+0000 7f3956000640 1 -- 172.21.15.6:0/1854065723 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944034070 con 0x7f39580cca20 2024-03-22T13:07:43.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.894+0000 7f3956000640 1 -- 172.21.15.6:0/1854065723 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3944030860 con 0x7f39580cca20 2024-03-22T13:07:43.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.894+0000 7f3956000640 1 -- 172.21.15.6:0/1854065723 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944030c30 con 0x7f39580cca20 2024-03-22T13:07:43.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 -- 172.21.15.6:0/1854065723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39580cca20 msgr2=0x7f39580ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 --2- 172.21.15.6:0/1854065723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39580cca20 0x7f39580ca340 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f395808d3b0 tx=0x7f394402b9a0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 -- 172.21.15.6:0/1854065723 shutdown_connections 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 --2- 172.21.15.6:0/1854065723 >> v2:172.21.15.6:3300/0 conn(0x7f395815b7a0 0x7f395815d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 --2- 172.21.15.6:0/1854065723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39580cca20 0x7f39580ca340 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 --2- 172.21.15.6:0/1854065723 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cc640 0x7f395806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 -- 172.21.15.6:0/1854065723 >> 172.21.15.6:0/1854065723 conn(0x7f39580cf9c0 msgr2=0x7f39580cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 -- 172.21.15.6:0/1854065723 shutdown_connections 2024-03-22T13:07:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.895+0000 7f395cc00640 1 -- 172.21.15.6:0/1854065723 wait complete. 2024-03-22T13:07:43.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.897+0000 7f395cc00640 1 Processor -- start 2024-03-22T13:07:43.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.898+0000 7f395cc00640 1 -- start start 2024-03-22T13:07:43.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.898+0000 7f395cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39580cc640 0x7f39580b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:43.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.898+0000 7f395cc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cca20 0x7f39580c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:43.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.898+0000 7f395cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f395815b7a0 0x7f39581f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:43.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.898+0000 7f395cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395806cfc0 con 0x7f39580cc640 2024-03-22T13:07:43.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.898+0000 7f395cc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395806cce0 con 0x7f395815b7a0 2024-03-22T13:07:43.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.898+0000 7f395cc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395815b000 con 0x7f39580cca20 2024-03-22T13:07:43.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.899+0000 7f3957e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f395815b7a0 0x7f39581f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:43.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.900+0000 7f3957e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f395815b7a0 0x7f39581f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51778/0 (socket says 172.21.15.6:51778) 2024-03-22T13:07:43.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.900+0000 7f3957e00640 1 -- 172.21.15.6:0/1426149047 learned_addr learned my addr 172.21.15.6:0/1426149047 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:43.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.905+0000 7f3957e00640 1 -- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cca20 msgr2=0x7f39580c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:43.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.905+0000 7f3957e00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cca20 0x7f39580c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.905+0000 7f3957e00640 1 -- 172.21.15.6:0/1426149047 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39580cc640 msgr2=0x7f39580b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:43.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.905+0000 7f3957e00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39580cc640 0x7f39580b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:43.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.905+0000 7f3957e00640 1 -- 172.21.15.6:0/1426149047 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39581f0050 con 0x7f395815b7a0 2024-03-22T13:07:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.908+0000 7f3957e00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f395815b7a0 0x7f39581f35c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f395000ba40 tx=0x7f395000bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.913+0000 7f394f400640 1 -- 172.21.15.6:0/1426149047 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950019070 con 0x7f395815b7a0 2024-03-22T13:07:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.913+0000 7f394f400640 1 -- 172.21.15.6:0/1426149047 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395000cb80 con 0x7f395815b7a0 2024-03-22T13:07:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.913+0000 7f394f400640 1 -- 172.21.15.6:0/1426149047 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950014440 con 0x7f395815b7a0 2024-03-22T13:07:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.913+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39581578e0 con 0x7f395815b7a0 2024-03-22T13:07:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.913+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39580c7120 con 0x7f395815b7a0 2024-03-22T13:07:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.914+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3958075650 con 0x7f395815b7a0 2024-03-22T13:07:43.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.939+0000 7f394f400640 1 -- 172.21.15.6:0/1426149047 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f395001c070 con 0x7f395815b7a0 2024-03-22T13:07:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.939+0000 7f394f400640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f393806bc20 0x7f393806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.940+0000 7f3957400640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f393806bc20 0x7f393806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.941+0000 7f3957400640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f393806bc20 0x7f393806e0e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3948009710 tx=0x7f3948006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:43.941+0000 7f394f400640 1 -- 172.21.15.6:0/1426149047 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3950057100 con 0x7f395815b7a0 2024-03-22T13:07:44.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.209+0000 7f394f400640 1 -- 172.21.15.6:0/1426149047 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3950023020 con 0x7f395815b7a0 2024-03-22T13:07:44.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.460+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39580c78e0 con 0x7f395815b7a0 2024-03-22T13:07:44.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.481+0000 7f394f400640 1 -- 172.21.15.6:0/1426149047 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f3950059760 con 0x7f395815b7a0 2024-03-22T13:07:44.481 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:44.481 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:44.481 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f393806bc20 msgr2=0x7f393806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f393806bc20 0x7f393806e0e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3948009710 tx=0x7f3948006c90 comp rx=0 tx=0).stop 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f395815b7a0 msgr2=0x7f39581f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f395815b7a0 0x7f39581f35c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f395000ba40 tx=0x7f395000bf00 comp rx=0 tx=0).stop 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 shutdown_connections 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f393806bc20 0x7f393806e0e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f395815b7a0 0x7f39581f35c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39580cca20 0x7f39580c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 --2- 172.21.15.6:0/1426149047 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39580cc640 0x7f39580b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 >> 172.21.15.6:0/1426149047 conn(0x7f39580cf9c0 msgr2=0x7f395815e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 shutdown_connections 2024-03-22T13:07:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:44.485+0000 7f395cc00640 1 -- 172.21.15.6:0/1426149047 wait complete. 2024-03-22T13:07:45.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:45.116 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:46.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:46.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.313+0000 7f3e79400640 1 Processor -- start 2024-03-22T13:07:46.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.313+0000 7f3e79400640 1 -- start start 2024-03-22T13:07:46.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.315+0000 7f3e79400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e741582a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:46.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.315+0000 7f3e79400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e740c8c30 0x7f3e740cad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:46.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.315+0000 7f3e79400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3e740c9070 0x7f3e740cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:46.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.315+0000 7f3e79400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e740ca8b0 con 0x7f3e740c8c30 2024-03-22T13:07:46.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.315+0000 7f3e79400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e74158b50 con 0x7f3e740c9070 2024-03-22T13:07:46.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.315+0000 7f3e79400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74158e30 con 0x7f3e74152240 2024-03-22T13:07:46.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.316+0000 7f3e73200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3e740c9070 0x7f3e740cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:46.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.316+0000 7f3e72800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e741582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:46.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.316+0000 7f3e71e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e740c8c30 0x7f3e740cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:46.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.316+0000 7f3e73200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3e740c9070 0x7f3e740cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:51794/0 (socket says 172.21.15.6:51794) 2024-03-22T13:07:46.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.316+0000 7f3e73200640 1 -- 172.21.15.6:0/2242428502 learned_addr learned my addr 172.21.15.6:0/2242428502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:46.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.319+0000 7f3e73200640 1 -- 172.21.15.6:0/2242428502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 msgr2=0x7f3e741582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:46.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.319+0000 7f3e73200640 1 --2- 172.21.15.6:0/2242428502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e741582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.319+0000 7f3e73200640 1 -- 172.21.15.6:0/2242428502 >> v2:172.21.15.6:3300/0 conn(0x7f3e740c8c30 msgr2=0x7f3e740cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:46.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.319+0000 7f3e73200640 1 --2- 172.21.15.6:0/2242428502 >> v2:172.21.15.6:3300/0 conn(0x7f3e740c8c30 0x7f3e740cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.319+0000 7f3e73200640 1 -- 172.21.15.6:0/2242428502 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74159110 con 0x7f3e740c9070 2024-03-22T13:07:46.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.321+0000 7f3e73200640 1 --2- 172.21.15.6:0/2242428502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 0x7f3e740cb280 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3e74068e60 tx=0x7f3e6c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=586284ebf680917a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.326+0000 7f3e71400640 1 -- 172.21.15.6:0/2242428502 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c02cb00 con 0x7f3e740c9070 2024-03-22T13:07:46.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.326+0000 7f3e71400640 1 -- 172.21.15.6:0/2242428502 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6c02cca0 con 0x7f3e740c9070 2024-03-22T13:07:46.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.326+0000 7f3e71400640 1 -- 172.21.15.6:0/2242428502 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c034610 con 0x7f3e740c9070 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 -- 172.21.15.6:0/2242428502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 msgr2=0x7f3e740cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 --2- 172.21.15.6:0/2242428502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 0x7f3e740cb280 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3e74068e60 tx=0x7f3e6c02bf60 comp rx=0 tx=0).stop 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 -- 172.21.15.6:0/2242428502 shutdown_connections 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 --2- 172.21.15.6:0/2242428502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 0x7f3e740cb280 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 --2- 172.21.15.6:0/2242428502 >> v2:172.21.15.6:3300/0 conn(0x7f3e740c8c30 0x7f3e740cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 --2- 172.21.15.6:0/2242428502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e741582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 -- 172.21.15.6:0/2242428502 >> 172.21.15.6:0/2242428502 conn(0x7f3e74071420 msgr2=0x7f3e74071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 -- 172.21.15.6:0/2242428502 shutdown_connections 2024-03-22T13:07:46.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.327+0000 7f3e79400640 1 -- 172.21.15.6:0/2242428502 wait complete. 2024-03-22T13:07:46.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.329+0000 7f3e79400640 1 Processor -- start 2024-03-22T13:07:46.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.329+0000 7f3e79400640 1 -- start start 2024-03-22T13:07:46.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.329+0000 7f3e79400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e740c8c30 0x7f3e74146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:46.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.329+0000 7f3e79400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 0x7f3e740cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:46.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.330+0000 7f3e79400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e740ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:46.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.330+0000 7f3e79400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e741587e0 con 0x7f3e740c8c30 2024-03-22T13:07:46.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.330+0000 7f3e79400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e740ca540 con 0x7f3e740c9070 2024-03-22T13:07:46.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.330+0000 7f3e79400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74152620 con 0x7f3e74152240 2024-03-22T13:07:46.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.330+0000 7f3e72800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e740c8c30 0x7f3e74146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:46.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.330+0000 7f3e71e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 0x7f3e740cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:46.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.330+0000 7f3e73200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e740ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:46.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.331+0000 7f3e72800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e740c8c30 0x7f3e74146600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59188/0 (socket says 172.21.15.6:59188) 2024-03-22T13:07:46.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.331+0000 7f3e72800640 1 -- 172.21.15.6:0/1950442488 learned_addr learned my addr 172.21.15.6:0/1950442488 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:46.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.334+0000 7f3e72800640 1 -- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 msgr2=0x7f3e740ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:46.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.334+0000 7f3e72800640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e740ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.334+0000 7f3e72800640 1 -- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 msgr2=0x7f3e740cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:46.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.334+0000 7f3e72800640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 0x7f3e740cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.335+0000 7f3e72800640 1 -- 172.21.15.6:0/1950442488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e741f0210 con 0x7f3e740c8c30 2024-03-22T13:07:46.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.337+0000 7f3e72800640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e740c8c30 0x7f3e74146600 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400e9b0 tx=0x7f3e6400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:46.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.342+0000 7f3e6aa00640 1 -- 172.21.15.6:0/1950442488 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64017070 con 0x7f3e740c8c30 2024-03-22T13:07:46.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.342+0000 7f3e6aa00640 1 -- 172.21.15.6:0/1950442488 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6400c9a0 con 0x7f3e740c8c30 2024-03-22T13:07:46.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.342+0000 7f3e6aa00640 1 -- 172.21.15.6:0/1950442488 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6400cd90 con 0x7f3e740c8c30 2024-03-22T13:07:46.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.342+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e741f04f0 con 0x7f3e740c8c30 2024-03-22T13:07:46.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.342+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e741f0ab0 con 0x7f3e740c8c30 2024-03-22T13:07:46.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.344+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e740756c0 con 0x7f3e740c8c30 2024-03-22T13:07:46.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.382+0000 7f3e6aa00640 1 -- 172.21.15.6:0/1950442488 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3e6401a070 con 0x7f3e740c8c30 2024-03-22T13:07:46.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.384+0000 7f3e6aa00640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e5406bc20 0x7f3e5406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:46.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.384+0000 7f3e71e00640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e5406bc20 0x7f3e5406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:46.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.385+0000 7f3e71e00640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e5406bc20 0x7f3e5406e0e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3e74146590 tx=0x7f3e60009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:46.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.672+0000 7f3e6aa00640 1 -- 172.21.15.6:0/1950442488 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3e640576d0 con 0x7f3e740c8c30 2024-03-22T13:07:46.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.672+0000 7f3e6aa00640 1 -- 172.21.15.6:0/1950442488 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3e64059630 con 0x7f3e740c8c30 2024-03-22T13:07:46.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.915+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e74155ef0 con 0x7f3e740c8c30 2024-03-22T13:07:46.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.937+0000 7f3e6aa00640 1 -- 172.21.15.6:0/1950442488 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f3e6405d6d0 con 0x7f3e740c8c30 2024-03-22T13:07:46.937 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:46.937 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:46.937 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e5406bc20 msgr2=0x7f3e5406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e5406bc20 0x7f3e5406e0e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3e74146590 tx=0x7f3e60009190 comp rx=0 tx=0).stop 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e740c8c30 msgr2=0x7f3e74146600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e740c8c30 0x7f3e74146600 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400e9b0 tx=0x7f3e6400ee70 comp rx=0 tx=0).stop 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 shutdown_connections 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e74152240 0x7f3e740ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3e5406bc20 0x7f3e5406e0e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e740c9070 0x7f3e740cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 --2- 172.21.15.6:0/1950442488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e740c8c30 0x7f3e74146600 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.939+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 >> 172.21.15.6:0/1950442488 conn(0x7f3e74071420 msgr2=0x7f3e74156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.940+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 shutdown_connections 2024-03-22T13:07:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:46.940+0000 7f3e79400640 1 -- 172.21.15.6:0/1950442488 wait complete. 2024-03-22T13:07:47.561 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:47.561 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:48.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.764+0000 7f80c6000640 1 Processor -- start 2024-03-22T13:07:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.764+0000 7f80c6000640 1 -- start start 2024-03-22T13:07:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.765+0000 7f80c6000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00ce8a0 0x7f80c006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.765+0000 7f80c6000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f80c015ae30 0x7f80c00cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.765+0000 7f80c6000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c015b260 con 0x7f80c00cc380 2024-03-22T13:07:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.766+0000 7f80c6000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c00cee00 con 0x7f80c015ae30 2024-03-22T13:07:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.766+0000 7f80c6000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00cf0c0 con 0x7f80c00ce8a0 2024-03-22T13:07:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.766+0000 7f80bfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00ce8a0 0x7f80c006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.767+0000 7f80c5600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f80c015ae30 0x7f80c00cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.767+0000 7f80bfe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00ce8a0 0x7f80c006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59632/0 (socket says 172.21.15.6:59632) 2024-03-22T13:07:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.767+0000 7f80bfe00640 1 -- 172.21.15.6:0/3156580988 learned_addr learned my addr 172.21.15.6:0/3156580988 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.770+0000 7f80bfe00640 1 -- 172.21.15.6:0/3156580988 >> v2:172.21.15.6:3300/0 conn(0x7f80c015ae30 msgr2=0x7f80c00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.770+0000 7f80bfe00640 1 --2- 172.21.15.6:0/3156580988 >> v2:172.21.15.6:3300/0 conn(0x7f80c015ae30 0x7f80c00cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.770+0000 7f80bfe00640 1 -- 172.21.15.6:0/3156580988 >> v1:172.21.15.6:6789/0 conn(0x7f80c00cc380 legacy=0x7f80c00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.770+0000 7f80bfe00640 1 -- 172.21.15.6:0/3156580988 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c00cf3a0 con 0x7f80c00ce8a0 2024-03-22T13:07:48.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.773+0000 7f80bfe00640 1 --2- 172.21.15.6:0/3156580988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00ce8a0 0x7f80c006c230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009870 tx=0x7f80ac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=550e5778ece991d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:48.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.776+0000 7f80bf400640 1 -- 172.21.15.6:0/3156580988 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac034070 con 0x7f80c00ce8a0 2024-03-22T13:07:48.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.777+0000 7f80bf400640 1 -- 172.21.15.6:0/3156580988 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac030840 con 0x7f80c00ce8a0 2024-03-22T13:07:48.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 -- 172.21.15.6:0/3156580988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00ce8a0 msgr2=0x7f80c006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:48.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 --2- 172.21.15.6:0/3156580988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00ce8a0 0x7f80c006c230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009870 tx=0x7f80ac02b980 comp rx=0 tx=0).stop 2024-03-22T13:07:48.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80bf400640 1 -- 172.21.15.6:0/3156580988 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac030c10 con 0x7f80c00ce8a0 2024-03-22T13:07:48.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 -- 172.21.15.6:0/3156580988 shutdown_connections 2024-03-22T13:07:48.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 --2- 172.21.15.6:0/3156580988 >> v2:172.21.15.6:3300/0 conn(0x7f80c015ae30 0x7f80c00cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:48.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 --2- 172.21.15.6:0/3156580988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00ce8a0 0x7f80c006c230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:48.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 -- 172.21.15.6:0/3156580988 >> 172.21.15.6:0/3156580988 conn(0x7f80c00b1480 msgr2=0x7f80c00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:48.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 -- 172.21.15.6:0/3156580988 shutdown_connections 2024-03-22T13:07:48.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.778+0000 7f80c6000640 1 -- 172.21.15.6:0/3156580988 wait complete. 2024-03-22T13:07:48.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.780+0000 7f80c6000640 1 Processor -- start 2024-03-22T13:07:48.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.780+0000 7f80c6000640 1 -- start start 2024-03-22T13:07:48.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.781+0000 7f80c6000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00cc380 0x7f80c00c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:48.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.781+0000 7f80c6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c00ce8a0 0x7f80c01e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:48.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.781+0000 7f80c6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80c015ae30 0x7f80c01e4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:48.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.781+0000 7f80c6000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c015a910 con 0x7f80c00ce8a0 2024-03-22T13:07:48.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.781+0000 7f80c6000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c015a630 con 0x7f80c015ae30 2024-03-22T13:07:48.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.781+0000 7f80c6000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c01549d0 con 0x7f80c00cc380 2024-03-22T13:07:48.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.782+0000 7f80c4c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00cc380 0x7f80c00c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:48.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.782+0000 7f80c5600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80c015ae30 0x7f80c01e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:48.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.782+0000 7f80bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c00ce8a0 0x7f80c01e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:48.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.782+0000 7f80c4c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00cc380 0x7f80c00c7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59636/0 (socket says 172.21.15.6:59636) 2024-03-22T13:07:48.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.782+0000 7f80c4c00640 1 -- 172.21.15.6:0/345310584 learned_addr learned my addr 172.21.15.6:0/345310584 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:48.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.787+0000 7f80c5600640 1 -- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00cc380 msgr2=0x7f80c00c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:48.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.787+0000 7f80c5600640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00cc380 0x7f80c00c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:48.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.787+0000 7f80c5600640 1 -- 172.21.15.6:0/345310584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c00ce8a0 msgr2=0x7f80c01e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:48.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.787+0000 7f80c5600640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c00ce8a0 0x7f80c01e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:48.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.787+0000 7f80c5600640 1 -- 172.21.15.6:0/345310584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c01f4430 con 0x7f80c015ae30 2024-03-22T13:07:48.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.790+0000 7f80c5600640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80c015ae30 0x7f80c01e4100 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009d90 tx=0x7f80b800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:48.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.794+0000 7f80bcc00640 1 -- 172.21.15.6:0/345310584 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b8018020 con 0x7f80c015ae30 2024-03-22T13:07:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.794+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c01f0ee0 con 0x7f80c015ae30 2024-03-22T13:07:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.794+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c01f0120 con 0x7f80c015ae30 2024-03-22T13:07:48.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.795+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c0159700 con 0x7f80c015ae30 2024-03-22T13:07:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.798+0000 7f80bcc00640 1 -- 172.21.15.6:0/345310584 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b8014480 con 0x7f80c015ae30 2024-03-22T13:07:48.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:48.798+0000 7f80bcc00640 1 -- 172.21.15.6:0/345310584 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b8014830 con 0x7f80c015ae30 2024-03-22T13:07:49.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.078+0000 7f80bcc00640 1 -- 172.21.15.6:0/345310584 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f80b8005d10 con 0x7f80c015ae30 2024-03-22T13:07:49.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.080+0000 7f80bcc00640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f80a006bbd0 0x7f80a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:49.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.081+0000 7f80c4c00640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f80a006bbd0 0x7f80a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:49.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.082+0000 7f80c4c00640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f80a006bbd0 0x7f80a006e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f80b40017a0 tx=0x7f80b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:49.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.103+0000 7f80bcc00640 1 -- 172.21.15.6:0/345310584 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f80b80581a0 con 0x7f80c015ae30 2024-03-22T13:07:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.114+0000 7f80bcc00640 1 -- 172.21.15.6:0/345310584 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f80c0159700 con 0x7f80c015ae30 2024-03-22T13:07:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.376+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80c0075650 con 0x7f80c015ae30 2024-03-22T13:07:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.400+0000 7f80bcc00640 1 -- 172.21.15.6:0/345310584 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f80b8022030 con 0x7f80c015ae30 2024-03-22T13:07:49.400 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:49.400 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f80a006bbd0 msgr2=0x7f80a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f80a006bbd0 0x7f80a006e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f80b40017a0 tx=0x7f80b4009040 comp rx=0 tx=0).stop 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80c015ae30 msgr2=0x7f80c01e4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80c015ae30 0x7f80c01e4100 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009d90 tx=0x7f80b800c490 comp rx=0 tx=0).stop 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 shutdown_connections 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f80a006bbd0 0x7f80a006e090 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80c015ae30 0x7f80c01e4100 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c00ce8a0 0x7f80c01e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 --2- 172.21.15.6:0/345310584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80c00cc380 0x7f80c00c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.403+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 >> 172.21.15.6:0/345310584 conn(0x7f80c00b1480 msgr2=0x7f80c00cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:49.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.404+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 shutdown_connections 2024-03-22T13:07:49.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:49.404+0000 7f80c6000640 1 -- 172.21.15.6:0/345310584 wait complete. 2024-03-22T13:07:50.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:50.025 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:51.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:51.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.222+0000 7fc184a00640 1 Processor -- start 2024-03-22T13:07:51.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.222+0000 7fc184a00640 1 -- start start 2024-03-22T13:07:51.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.223+0000 7fc184a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1800ce8a0 0x7fc18006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:51.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.223+0000 7fc184a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18015b260 con 0x7fc18015ae30 2024-03-22T13:07:51.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.224+0000 7fc184a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1800cee00 con 0x7fc1800cc380 2024-03-22T13:07:51.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.224+0000 7fc184a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1800cf0c0 con 0x7fc1800ce8a0 2024-03-22T13:07:51.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.224+0000 7fc17ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1800ce8a0 0x7fc18006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:51.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.224+0000 7fc17f400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc1800cc380 0x7fc1800cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:46266/0 (socket says 172.21.15.6:46266) 2024-03-22T13:07:51.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.225+0000 7fc17ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1800ce8a0 0x7fc18006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59640/0 (socket says 172.21.15.6:59640) 2024-03-22T13:07:51.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.225+0000 7fc17f400640 1 -- 172.21.15.6:0/2407075592 learned_addr learned my addr 172.21.15.6:0/2407075592 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:51.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.224+0000 7fc17fe00640 1 --1- 172.21.15.6:0/2407075592 >> v1:172.21.15.6:6789/0 conn(0x7fc18015ae30 0x7fc180164b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54742/0 (socket says 172.21.15.6:54742) 2024-03-22T13:07:51.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.227+0000 7fc17ea00640 1 -- 172.21.15.6:0/2407075592 >> v1:172.21.15.92:6789/0 conn(0x7fc1800cc380 legacy=0x7fc1800cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:51.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.227+0000 7fc17ea00640 1 -- 172.21.15.6:0/2407075592 >> v1:172.21.15.6:6789/0 conn(0x7fc18015ae30 legacy=0x7fc180164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:51.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.227+0000 7fc17ea00640 1 -- 172.21.15.6:0/2407075592 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1800cf3a0 con 0x7fc1800ce8a0 2024-03-22T13:07:51.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.229+0000 7fc17ea00640 1 --2- 172.21.15.6:0/2407075592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1800ce8a0 0x7fc18006c230 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc1680097d0 tx=0x7fc16802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee95070939f6ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:51.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.234+0000 7fc17e000640 1 -- 172.21.15.6:0/2407075592 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168035020 con 0x7fc1800ce8a0 2024-03-22T13:07:51.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.234+0000 7fc17e000640 1 -- 172.21.15.6:0/2407075592 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc168031890 con 0x7fc1800ce8a0 2024-03-22T13:07:51.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.234+0000 7fc17e000640 1 -- 172.21.15.6:0/2407075592 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168031c60 con 0x7fc1800ce8a0 2024-03-22T13:07:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.234+0000 7fc184a00640 1 -- 172.21.15.6:0/2407075592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1800ce8a0 msgr2=0x7fc18006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.235+0000 7fc184a00640 1 --2- 172.21.15.6:0/2407075592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1800ce8a0 0x7fc18006c230 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc1680097d0 tx=0x7fc16802b930 comp rx=0 tx=0).stop 2024-03-22T13:07:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.235+0000 7fc184a00640 1 -- 172.21.15.6:0/2407075592 shutdown_connections 2024-03-22T13:07:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.235+0000 7fc184a00640 1 --2- 172.21.15.6:0/2407075592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1800ce8a0 0x7fc18006c230 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.235+0000 7fc184a00640 1 -- 172.21.15.6:0/2407075592 >> 172.21.15.6:0/2407075592 conn(0x7fc1800b1480 msgr2=0x7fc1800b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.235+0000 7fc184a00640 1 -- 172.21.15.6:0/2407075592 shutdown_connections 2024-03-22T13:07:51.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.235+0000 7fc184a00640 1 -- 172.21.15.6:0/2407075592 wait complete. 2024-03-22T13:07:51.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.237+0000 7fc184a00640 1 Processor -- start 2024-03-22T13:07:51.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.238+0000 7fc184a00640 1 -- start start 2024-03-22T13:07:51.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.239+0000 7fc184a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1800cc380 0x7fc18006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:51.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.239+0000 7fc184a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1800ce8a0 0x7fc1801f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:51.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.239+0000 7fc184a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc18015ae30 0x7fc1800c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:51.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.239+0000 7fc184a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18015a910 con 0x7fc1800cc380 2024-03-22T13:07:51.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.239+0000 7fc184a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18015a630 con 0x7fc1800ce8a0 2024-03-22T13:07:51.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.239+0000 7fc184a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1801549d0 con 0x7fc18015ae30 2024-03-22T13:07:51.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.240+0000 7fc17fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc18015ae30 0x7fc1800c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:51.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.240+0000 7fc17f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1800cc380 0x7fc18006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:51.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.240+0000 7fc17ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1800ce8a0 0x7fc1801f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:51.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.240+0000 7fc17fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc18015ae30 0x7fc1800c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:59654/0 (socket says 172.21.15.6:59654) 2024-03-22T13:07:51.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.240+0000 7fc17fe00640 1 -- 172.21.15.6:0/3666459855 learned_addr learned my addr 172.21.15.6:0/3666459855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:51.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.243+0000 7fc17fe00640 1 -- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1800ce8a0 msgr2=0x7fc1801f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:51.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.243+0000 7fc17fe00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1800ce8a0 0x7fc1801f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.243+0000 7fc17fe00640 1 -- 172.21.15.6:0/3666459855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1800cc380 msgr2=0x7fc18006cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:51.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.243+0000 7fc17fe00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1800cc380 0x7fc18006cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.243+0000 7fc17fe00640 1 -- 172.21.15.6:0/3666459855 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18015d9a0 con 0x7fc18015ae30 2024-03-22T13:07:51.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.246+0000 7fc17fe00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc18015ae30 0x7fc1800c7620 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc17800c430 tx=0x7fc17800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.250+0000 7fc177400640 1 -- 172.21.15.6:0/3666459855 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1780074e0 con 0x7fc18015ae30 2024-03-22T13:07:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.250+0000 7fc177400640 1 -- 172.21.15.6:0/3666459855 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc178007680 con 0x7fc18015ae30 2024-03-22T13:07:51.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.250+0000 7fc177400640 1 -- 172.21.15.6:0/3666459855 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1780049e0 con 0x7fc18015ae30 2024-03-22T13:07:51.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.250+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18015dc80 con 0x7fc18015ae30 2024-03-22T13:07:51.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.250+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18015b690 con 0x7fc18015ae30 2024-03-22T13:07:51.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.251+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18015a2e0 con 0x7fc18015ae30 2024-03-22T13:07:51.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.271+0000 7fc177400640 1 -- 172.21.15.6:0/3666459855 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc17801a020 con 0x7fc18015ae30 2024-03-22T13:07:51.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.273+0000 7fc177400640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc15c06bc50 0x7fc15c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:51.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.274+0000 7fc17f400640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc15c06bc50 0x7fc15c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.274+0000 7fc17f400640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc15c06bc50 0x7fc15c06e110 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc170005c00 tx=0x7fc170009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.514+0000 7fc177400640 1 -- 172.21.15.6:0/3666459855 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc178056e40 con 0x7fc18015ae30 2024-03-22T13:07:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.526+0000 7fc177400640 1 -- 172.21.15.6:0/3666459855 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc18015a2e0 con 0x7fc18015ae30 2024-03-22T13:07:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.783+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc180075650 con 0x7fc18015ae30 2024-03-22T13:07:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.796+0000 7fc177400640 1 -- 172.21.15.6:0/3666459855 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fc178021070 con 0x7fc18015ae30 2024-03-22T13:07:51.796 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:51.796 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc15c06bc50 msgr2=0x7fc15c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc15c06bc50 0x7fc15c06e110 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc170005c00 tx=0x7fc170009040 comp rx=0 tx=0).stop 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc18015ae30 msgr2=0x7fc1800c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc18015ae30 0x7fc1800c7620 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc17800c430 tx=0x7fc17800c8f0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 shutdown_connections 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc18015ae30 0x7fc1800c7620 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc15c06bc50 0x7fc15c06e110 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1800ce8a0 0x7fc1801f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 --2- 172.21.15.6:0/3666459855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1800cc380 0x7fc18006cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 >> 172.21.15.6:0/3666459855 conn(0x7fc1800b1480 msgr2=0x7fc180158e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:51.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 shutdown_connections 2024-03-22T13:07:51.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:51.799+0000 7fc184a00640 1 -- 172.21.15.6:0/3666459855 wait complete. 2024-03-22T13:07:52.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:52.421 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:53.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:53.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.627+0000 7fc4b4600640 1 Processor -- start 2024-03-22T13:07:53.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.627+0000 7fc4b4600640 1 -- start start 2024-03-22T13:07:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.629+0000 7fc4b4600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac06fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.629+0000 7fc4b4600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac070100 con 0x7fc4ac06c210 2024-03-22T13:07:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.629+0000 7fc4b4600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac0708c0 con 0x7fc4ac154490 2024-03-22T13:07:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.629+0000 7fc4b4600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac06b1b0 con 0x7fc4ac06c720 2024-03-22T13:07:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.629+0000 7fc4b1200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac06fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.629+0000 7fc4b2600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc4ac06c210 0x7fc4ac15d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58812/0 (socket says 172.21.15.6:58812) 2024-03-22T13:07:53.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.630+0000 7fc4b2600640 1 -- 172.21.15.6:0/1868517253 learned_addr learned my addr 172.21.15.6:0/1868517253 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.633+0000 7fc4abe00640 1 -- 172.21.15.6:0/1868517253 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730538521 0 0) 0x7fc4ac070100 con 0x7fc4ac06c210 2024-03-22T13:07:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.633+0000 7fc4abe00640 1 -- 172.21.15.6:0/1868517253 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc494003520 con 0x7fc4ac06c210 2024-03-22T13:07:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.633+0000 7fc4abe00640 1 -- 172.21.15.6:0/1868517253 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627949381 0 0) 0x7fc4ac0708c0 con 0x7fc4ac154490 2024-03-22T13:07:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.633+0000 7fc4abe00640 1 -- 172.21.15.6:0/1868517253 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ac070100 con 0x7fc4ac154490 2024-03-22T13:07:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.633+0000 7fc4b1200640 1 -- 172.21.15.6:0/1868517253 >> v1:172.21.15.92:6789/0 conn(0x7fc4ac154490 legacy=0x7fc4ac0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.633+0000 7fc4b1200640 1 -- 172.21.15.6:0/1868517253 >> v1:172.21.15.6:6789/0 conn(0x7fc4ac06c210 legacy=0x7fc4ac15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:53.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.633+0000 7fc4b1200640 1 -- 172.21.15.6:0/1868517253 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac06b450 con 0x7fc4ac06c720 2024-03-22T13:07:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.636+0000 7fc4b1200640 1 --2- 172.21.15.6:0/1868517253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac06fae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc4980097d0 tx=0x7fc49802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a0be22adc562ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.640+0000 7fc4abe00640 1 -- 172.21.15.6:0/1868517253 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498035020 con 0x7fc4ac06c720 2024-03-22T13:07:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.640+0000 7fc4abe00640 1 -- 172.21.15.6:0/1868517253 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc498031890 con 0x7fc4ac06c720 2024-03-22T13:07:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.640+0000 7fc4abe00640 1 -- 172.21.15.6:0/1868517253 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498031c60 con 0x7fc4ac06c720 2024-03-22T13:07:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.641+0000 7fc4b4600640 1 -- 172.21.15.6:0/1868517253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 msgr2=0x7fc4ac06fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.641+0000 7fc4b4600640 1 --2- 172.21.15.6:0/1868517253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac06fae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc4980097d0 tx=0x7fc49802b930 comp rx=0 tx=0).stop 2024-03-22T13:07:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.641+0000 7fc4b4600640 1 -- 172.21.15.6:0/1868517253 shutdown_connections 2024-03-22T13:07:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.641+0000 7fc4b4600640 1 --2- 172.21.15.6:0/1868517253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac06fae0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.641+0000 7fc4b4600640 1 -- 172.21.15.6:0/1868517253 >> 172.21.15.6:0/1868517253 conn(0x7fc4ac0c8cf0 msgr2=0x7fc4ac0c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.641+0000 7fc4b4600640 1 -- 172.21.15.6:0/1868517253 shutdown_connections 2024-03-22T13:07:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.641+0000 7fc4b4600640 1 -- 172.21.15.6:0/1868517253 wait complete. 2024-03-22T13:07:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.643+0000 7fc4b4600640 1 Processor -- start 2024-03-22T13:07:53.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.643+0000 7fc4b4600640 1 -- start start 2024-03-22T13:07:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.644+0000 7fc4b4600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ac06c210 0x7fc4ac15d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.644+0000 7fc4b4600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.644+0000 7fc4b4600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4ac154490 0x7fc4ac14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.644+0000 7fc4b4600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac06bcf0 con 0x7fc4ac06c210 2024-03-22T13:07:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.644+0000 7fc4b4600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac06ba10 con 0x7fc4ac154490 2024-03-22T13:07:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.644+0000 7fc4b4600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac15dfb0 con 0x7fc4ac06c720 2024-03-22T13:07:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.645+0000 7fc4b1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ac06c210 0x7fc4ac15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.645+0000 7fc4b1200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.645+0000 7fc4b2600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4ac154490 0x7fc4ac14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.645+0000 7fc4b1200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac1f4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:45284/0 (socket says 172.21.15.6:45284) 2024-03-22T13:07:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.645+0000 7fc4b1200640 1 -- 172.21.15.6:0/3057995663 learned_addr learned my addr 172.21.15.6:0/3057995663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.648+0000 7fc4b1200640 1 -- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4ac154490 msgr2=0x7fc4ac14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.648+0000 7fc4b1200640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4ac154490 0x7fc4ac14e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.648+0000 7fc4b1200640 1 -- 172.21.15.6:0/3057995663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ac06c210 msgr2=0x7fc4ac15d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.649+0000 7fc4b1200640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ac06c210 0x7fc4ac15d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.649+0000 7fc4b1200640 1 -- 172.21.15.6:0/3057995663 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac15b960 con 0x7fc4ac06c720 2024-03-22T13:07:53.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.651+0000 7fc4b1200640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac1f4420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc49802bf40 tx=0x7fc4980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:53.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.657+0000 7fc4a9600640 1 -- 172.21.15.6:0/3057995663 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498035040 con 0x7fc4ac06c720 2024-03-22T13:07:53.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.657+0000 7fc4a9600640 1 -- 172.21.15.6:0/3057995663 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc498030c90 con 0x7fc4ac06c720 2024-03-22T13:07:53.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.657+0000 7fc4a9600640 1 -- 172.21.15.6:0/3057995663 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498044670 con 0x7fc4ac06c720 2024-03-22T13:07:53.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.657+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac15bc40 con 0x7fc4ac06c720 2024-03-22T13:07:53.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.657+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac1f3c20 con 0x7fc4ac06c720 2024-03-22T13:07:53.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.658+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac0756c0 con 0x7fc4ac06c720 2024-03-22T13:07:53.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.950+0000 7fc4a9600640 1 -- 172.21.15.6:0/3057995663 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc498038020 con 0x7fc4ac06c720 2024-03-22T13:07:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.952+0000 7fc4a9600640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc48c06bc50 0x7fc48c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.953+0000 7fc4b1c00640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc48c06bc50 0x7fc48c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.953+0000 7fc4a9600640 1 -- 172.21.15.6:0/3057995663 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc498030e40 con 0x7fc4ac06c720 2024-03-22T13:07:53.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.954+0000 7fc4b1c00640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc48c06bc50 0x7fc48c06e110 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00011e0 tx=0x7fc4a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:53.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:53.963+0000 7fc4a9600640 1 -- 172.21.15.6:0/3057995663 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc49802d040 con 0x7fc4ac06c720 2024-03-22T13:07:54.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.243+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4ac0c4530 con 0x7fc4ac06c720 2024-03-22T13:07:54.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.263+0000 7fc4a9600640 1 -- 172.21.15.6:0/3057995663 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fc49807b800 con 0x7fc4ac06c720 2024-03-22T13:07:54.263 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:54.263 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:54.263 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:54.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.265+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc48c06bc50 msgr2=0x7fc48c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.265+0000 7fc4b4600640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc48c06bc50 0x7fc48c06e110 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00011e0 tx=0x7fc4a0009040 comp rx=0 tx=0).stop 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.265+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 msgr2=0x7fc4ac1f4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.265+0000 7fc4b4600640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac1f4420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc49802bf40 tx=0x7fc4980046c0 comp rx=0 tx=0).stop 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 shutdown_connections 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc48c06bc50 0x7fc48c06e110 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4ac154490 0x7fc4ac14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4ac06c720 0x7fc4ac1f4420 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 --2- 172.21.15.6:0/3057995663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4ac06c210 0x7fc4ac15d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 >> 172.21.15.6:0/3057995663 conn(0x7fc4ac0c8cf0 msgr2=0x7fc4ac152700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 shutdown_connections 2024-03-22T13:07:54.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:54.266+0000 7fc4b4600640 1 -- 172.21.15.6:0/3057995663 wait complete. 2024-03-22T13:07:54.887 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:54.887 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:55.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.085+0000 7fef5fe00640 1 Processor -- start 2024-03-22T13:07:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.085+0000 7fef5fe00640 1 -- start start 2024-03-22T13:07:56.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.088+0000 7fef5fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fef58072310 0x7fef58159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:56.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.088+0000 7fef5fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580c48e0 con 0x7fef580c7f00 2024-03-22T13:07:56.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.088+0000 7fef5fe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580716d0 con 0x7fef58150a70 2024-03-22T13:07:56.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.088+0000 7fef5fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef58071990 con 0x7fef58072310 2024-03-22T13:07:56.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.089+0000 7fef5d400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fef58150a70 0x7fef5806f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37368/0 (socket says 172.21.15.6:37368) 2024-03-22T13:07:56.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.089+0000 7fef5d400640 1 -- 172.21.15.6:0/1886416444 learned_addr learned my addr 172.21.15.6:0/1886416444 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:56.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.089+0000 7fef5ca00640 1 --2- 172.21.15.6:0/1886416444 >> v2:172.21.15.6:3300/0 conn(0x7fef58072310 0x7fef58159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:56.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.091+0000 7fef57e00640 1 -- 172.21.15.6:0/1886416444 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883961568 0 0) 0x7fef580716d0 con 0x7fef58150a70 2024-03-22T13:07:56.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.091+0000 7fef57e00640 1 -- 172.21.15.6:0/1886416444 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef40003440 con 0x7fef58150a70 2024-03-22T13:07:56.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.095+0000 7fef5ca00640 1 -- 172.21.15.6:0/1886416444 >> v1:172.21.15.92:6789/0 conn(0x7fef58150a70 legacy=0x7fef5806f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:56.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.095+0000 7fef5ca00640 1 -- 172.21.15.6:0/1886416444 >> v1:172.21.15.6:6789/0 conn(0x7fef580c7f00 legacy=0x7fef58161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:56.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.095+0000 7fef5ca00640 1 -- 172.21.15.6:0/1886416444 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef58071c70 con 0x7fef58072310 2024-03-22T13:07:56.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.100+0000 7fef5ca00640 1 --2- 172.21.15.6:0/1886416444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58072310 0x7fef58159cd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fef440097a0 tx=0x7fef4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471e5974968dc8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:56.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.106+0000 7fef57e00640 1 -- 172.21.15.6:0/1886416444 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44035020 con 0x7fef58072310 2024-03-22T13:07:56.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.106+0000 7fef57e00640 1 -- 172.21.15.6:0/1886416444 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef44031830 con 0x7fef58072310 2024-03-22T13:07:56.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.106+0000 7fef57e00640 1 -- 172.21.15.6:0/1886416444 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44031c00 con 0x7fef58072310 2024-03-22T13:07:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.107+0000 7fef5fe00640 1 -- 172.21.15.6:0/1886416444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58072310 msgr2=0x7fef58159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.107+0000 7fef5fe00640 1 --2- 172.21.15.6:0/1886416444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58072310 0x7fef58159cd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fef440097a0 tx=0x7fef4402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.107+0000 7fef5fe00640 1 -- 172.21.15.6:0/1886416444 shutdown_connections 2024-03-22T13:07:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.107+0000 7fef5fe00640 1 --2- 172.21.15.6:0/1886416444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58072310 0x7fef58159cd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.107+0000 7fef5fe00640 1 -- 172.21.15.6:0/1886416444 >> 172.21.15.6:0/1886416444 conn(0x7fef580cb930 msgr2=0x7fef580cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.107+0000 7fef5fe00640 1 -- 172.21.15.6:0/1886416444 shutdown_connections 2024-03-22T13:07:56.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.107+0000 7fef5fe00640 1 -- 172.21.15.6:0/1886416444 wait complete. 2024-03-22T13:07:56.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.109+0000 7fef5fe00640 1 Processor -- start 2024-03-22T13:07:56.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.109+0000 7fef5fe00640 1 -- start start 2024-03-22T13:07:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.110+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef58072310 0x7fef58144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.110+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef580c7f00 0x7fef581f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.110+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58150a70 0x7fef581f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.110+0000 7fef5fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58150e50 con 0x7fef58150a70 2024-03-22T13:07:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.110+0000 7fef5fe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580c4b50 con 0x7fef580c7f00 2024-03-22T13:07:56.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.110+0000 7fef5fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5806f7d0 con 0x7fef58072310 2024-03-22T13:07:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.111+0000 7fef5ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef580c7f00 0x7fef581f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.111+0000 7fef5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58150a70 0x7fef581f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.111+0000 7fef5d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef58072310 0x7fef58144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.111+0000 7fef5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58150a70 0x7fef581f3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60402/0 (socket says 172.21.15.6:60402) 2024-03-22T13:07:56.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.111+0000 7fef5de00640 1 -- 172.21.15.6:0/3020598180 learned_addr learned my addr 172.21.15.6:0/3020598180 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:56.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.115+0000 7fef5de00640 1 -- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef58072310 msgr2=0x7fef58144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:56.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.115+0000 7fef5de00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef58072310 0x7fef58144cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.115+0000 7fef5de00640 1 -- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef580c7f00 msgr2=0x7fef581f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:56.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.115+0000 7fef5de00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef580c7f00 0x7fef581f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.115+0000 7fef5de00640 1 -- 172.21.15.6:0/3020598180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef581f3ba0 con 0x7fef58150a70 2024-03-22T13:07:56.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.118+0000 7fef5de00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58150a70 0x7fef581f3540 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fef5000c7a0 tx=0x7fef5000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:56.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.124+0000 7fef55600640 1 -- 172.21.15.6:0/3020598180 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef50003b70 con 0x7fef58150a70 2024-03-22T13:07:56.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.124+0000 7fef55600640 1 -- 172.21.15.6:0/3020598180 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef50003d30 con 0x7fef58150a70 2024-03-22T13:07:56.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.124+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef581464c0 con 0x7fef58150a70 2024-03-22T13:07:56.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.124+0000 7fef55600640 1 -- 172.21.15.6:0/3020598180 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5000aa90 con 0x7fef58150a70 2024-03-22T13:07:56.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.124+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef581eba90 con 0x7fef58150a70 2024-03-22T13:07:56.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.125+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef580756c0 con 0x7fef58150a70 2024-03-22T13:07:56.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.433+0000 7fef55600640 1 -- 172.21.15.6:0/3020598180 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef5000ac30 con 0x7fef58150a70 2024-03-22T13:07:56.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.436+0000 7fef55600640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fef3806bc50 0x7fef3806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:56.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.436+0000 7fef5d400640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fef3806bc50 0x7fef3806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:56.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.437+0000 7fef5d400640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fef3806bc50 0x7fef3806e110 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0011e0 tx=0x7fef4c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:56.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.451+0000 7fef55600640 1 -- 172.21.15.6:0/3020598180 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fef50057820 con 0x7fef58150a70 2024-03-22T13:07:56.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.452+0000 7fef55600640 1 -- 172.21.15.6:0/3020598180 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef50059780 con 0x7fef58150a70 2024-03-22T13:07:56.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.714+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef580c3780 con 0x7fef58150a70 2024-03-22T13:07:56.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.744+0000 7fef55600640 1 -- 172.21.15.6:0/3020598180 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fef5005d820 con 0x7fef58150a70 2024-03-22T13:07:56.745 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:56.745 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:56.745 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:56.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.747+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fef3806bc50 msgr2=0x7fef3806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.747+0000 7fef5fe00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fef3806bc50 0x7fef3806e110 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0011e0 tx=0x7fef4c007d30 comp rx=0 tx=0).stop 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.747+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58150a70 msgr2=0x7fef581f3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.747+0000 7fef5fe00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58150a70 0x7fef581f3540 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fef5000c7a0 tx=0x7fef5000cc60 comp rx=0 tx=0).stop 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 shutdown_connections 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef58150a70 0x7fef581f3540 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fef3806bc50 0x7fef3806e110 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef580c7f00 0x7fef581f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 --2- 172.21.15.6:0/3020598180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef58072310 0x7fef58144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 >> 172.21.15.6:0/3020598180 conn(0x7fef580cb930 msgr2=0x7fef580c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 shutdown_connections 2024-03-22T13:07:56.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:56.748+0000 7fef5fe00640 1 -- 172.21.15.6:0/3020598180 wait complete. 2024-03-22T13:07:57.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:57.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:07:58.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:07:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.597+0000 7fc3a9600640 1 Processor -- start 2024-03-22T13:07:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.598+0000 7fc3a9600640 1 -- start start 2024-03-22T13:07:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.599+0000 7fc3a9600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3a4154240 0x7fc3a415ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.599+0000 7fc3a9600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c68c0 0x7fc3a406b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.599+0000 7fc3a9600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a40c4740 con 0x7fc3a40c7ee0 2024-03-22T13:07:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.599+0000 7fc3a9600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a415e130 con 0x7fc3a4154240 2024-03-22T13:07:58.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.599+0000 7fc3a9600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a4154620 con 0x7fc3a40c68c0 2024-03-22T13:07:58.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.630+0000 7fc3a2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3a4154240 0x7fc3a415ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:58.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.630+0000 7fc3a2000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc3a40c7ee0 0x7fc3a40c4030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58820/0 (socket says 172.21.15.6:58820) 2024-03-22T13:07:58.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.630+0000 7fc3a2000640 1 -- 172.21.15.6:0/1537983731 learned_addr learned my addr 172.21.15.6:0/1537983731 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:07:58.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.637+0000 7fc3a2a00640 1 -- 172.21.15.6:0/1537983731 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c68c0 msgr2=0x7fc3a406b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:07:58.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.637+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1537983731 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c68c0 0x7fc3a406b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:58.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.637+0000 7fc3a2a00640 1 -- 172.21.15.6:0/1537983731 >> v1:172.21.15.6:6789/0 conn(0x7fc3a40c7ee0 legacy=0x7fc3a40c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:58.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.637+0000 7fc3a2a00640 1 -- 172.21.15.6:0/1537983731 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40c6e00 con 0x7fc3a4154240 2024-03-22T13:07:58.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.644+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1537983731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154240 0x7fc3a415ac90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc3940097a0 tx=0x7fc39402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4900ddd78bc6b846 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:58.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.661+0000 7fc3a1600640 1 -- 172.21.15.6:0/1537983731 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394034070 con 0x7fc3a4154240 2024-03-22T13:07:58.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.661+0000 7fc3a1600640 1 -- 172.21.15.6:0/1537983731 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394030800 con 0x7fc3a4154240 2024-03-22T13:07:58.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.661+0000 7fc3a1600640 1 -- 172.21.15.6:0/1537983731 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394030bd0 con 0x7fc3a4154240 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 -- 172.21.15.6:0/1537983731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154240 msgr2=0x7fc3a415ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1537983731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154240 0x7fc3a415ac90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc3940097a0 tx=0x7fc39402b940 comp rx=0 tx=0).stop 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 -- 172.21.15.6:0/1537983731 shutdown_connections 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1537983731 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c68c0 0x7fc3a406b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1537983731 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154240 0x7fc3a415ac90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 -- 172.21.15.6:0/1537983731 >> 172.21.15.6:0/1537983731 conn(0x7fc3a40cb930 msgr2=0x7fc3a40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 -- 172.21.15.6:0/1537983731 shutdown_connections 2024-03-22T13:07:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.662+0000 7fc3a9600640 1 -- 172.21.15.6:0/1537983731 wait complete. 2024-03-22T13:07:58.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.665+0000 7fc3a9600640 1 Processor -- start 2024-03-22T13:07:58.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.665+0000 7fc3a9600640 1 -- start start 2024-03-22T13:07:58.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.666+0000 7fc3a9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a40c68c0 0x7fc3a414b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:58.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.666+0000 7fc3a9600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c7ee0 0x7fc3a4074730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:58.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.666+0000 7fc3a9600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a4154240 0x7fc3a414d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:58.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.667+0000 7fc3a9600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415dae0 con 0x7fc3a40c68c0 2024-03-22T13:07:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.667+0000 7fc3a9600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415d800 con 0x7fc3a4154240 2024-03-22T13:07:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.667+0000 7fc3a9600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415ddc0 con 0x7fc3a40c7ee0 2024-03-22T13:07:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.667+0000 7fc3a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a40c68c0 0x7fc3a414b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.667+0000 7fc3a2000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c7ee0 0x7fc3a4074730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.668+0000 7fc3a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a40c68c0 0x7fc3a414b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60416/0 (socket says 172.21.15.6:60416) 2024-03-22T13:07:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.668+0000 7fc3a2a00640 1 -- 172.21.15.6:0/1761709996 learned_addr learned my addr 172.21.15.6:0/1761709996 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:07:58.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.672+0000 7fc3a2a00640 1 -- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c7ee0 msgr2=0x7fc3a4074730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:58.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.672+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c7ee0 0x7fc3a4074730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:58.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.672+0000 7fc3a2a00640 1 -- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a4154240 msgr2=0x7fc3a414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:58.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.672+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a4154240 0x7fc3a414d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:58.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.672+0000 7fc3a2a00640 1 -- 172.21.15.6:0/1761709996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a414bcd0 con 0x7fc3a40c68c0 2024-03-22T13:07:58.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.675+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a40c68c0 0x7fc3a414b490 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc3940098c0 tx=0x7fc3940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.681+0000 7fc39aa00640 1 -- 172.21.15.6:0/1761709996 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394034040 con 0x7fc3a40c68c0 2024-03-22T13:07:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.681+0000 7fc39aa00640 1 -- 172.21.15.6:0/1761709996 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3940443f0 con 0x7fc3a40c68c0 2024-03-22T13:07:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.681+0000 7fc39aa00640 1 -- 172.21.15.6:0/1761709996 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3940447e0 con 0x7fc3a40c68c0 2024-03-22T13:07:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.681+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a41e3b10 con 0x7fc3a40c68c0 2024-03-22T13:07:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.682+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a41e40d0 con 0x7fc3a40c68c0 2024-03-22T13:07:58.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:58.683+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a40756c0 con 0x7fc3a40c68c0 2024-03-22T13:07:59.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.012+0000 7fc39aa00640 1 -- 172.21.15.6:0/1761709996 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc394037070 con 0x7fc3a40c68c0 2024-03-22T13:07:59.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.013+0000 7fc39aa00640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc38406bc20 0x7fc38406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:07:59.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.014+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc38406bc20 0x7fc38406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:07:59.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.015+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc38406bc20 0x7fc38406e0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc3a41edc70 tx=0x7fc390007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:07:59.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.055+0000 7fc39aa00640 1 -- 172.21.15.6:0/1761709996 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc3940307a0 con 0x7fc3a40c68c0 2024-03-22T13:07:59.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.055+0000 7fc39aa00640 1 -- 172.21.15.6:0/1761709996 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc39403f070 con 0x7fc3a40c68c0 2024-03-22T13:07:59.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.306+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3a40c7ab0 con 0x7fc3a40c68c0 2024-03-22T13:07:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.326+0000 7fc39aa00640 1 -- 172.21.15.6:0/1761709996 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fc39407b4c0 con 0x7fc3a40c68c0 2024-03-22T13:07:59.325 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:07:59.326 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:07:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:07:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc38406bc20 msgr2=0x7fc38406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc38406bc20 0x7fc38406e0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc3a41edc70 tx=0x7fc390007e80 comp rx=0 tx=0).stop 2024-03-22T13:07:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a40c68c0 msgr2=0x7fc3a414b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:07:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a40c68c0 0x7fc3a414b490 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc3940098c0 tx=0x7fc3940046c0 comp rx=0 tx=0).stop 2024-03-22T13:07:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 shutdown_connections 2024-03-22T13:07:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc38406bc20 0x7fc38406e0e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a4154240 0x7fc3a414d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:59.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a40c7ee0 0x7fc3a4074730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:59.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 --2- 172.21.15.6:0/1761709996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a40c68c0 0x7fc3a414b490 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:07:59.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 >> 172.21.15.6:0/1761709996 conn(0x7fc3a40cb930 msgr2=0x7fc3a40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:07:59.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 shutdown_connections 2024-03-22T13:07:59.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:07:59.328+0000 7fc3a9600640 1 -- 172.21.15.6:0/1761709996 wait complete. 2024-03-22T13:07:59.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:07:59.949 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:00.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:01.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.152+0000 7efd82e00640 1 Processor -- start 2024-03-22T13:08:01.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.152+0000 7efd82e00640 1 -- start start 2024-03-22T13:08:01.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.154+0000 7efd82e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd7c0cc790 0x7efd7c0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:01.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.154+0000 7efd82e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c0ce230 con 0x7efd7c15ab40 2024-03-22T13:08:01.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.154+0000 7efd82e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c0ce510 con 0x7efd7c0cc3b0 2024-03-22T13:08:01.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.154+0000 7efd82e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c0cd0f0 con 0x7efd7c0cc790 2024-03-22T13:08:01.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.155+0000 7efd7be00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efd7c0cc3b0 0x7efd7c0cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37380/0 (socket says 172.21.15.6:37380) 2024-03-22T13:08:01.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.155+0000 7efd7be00640 1 -- 172.21.15.6:0/2736071048 learned_addr learned my addr 172.21.15.6:0/2736071048 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:01.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.155+0000 7efd7b400640 1 --2- 172.21.15.6:0/2736071048 >> v2:172.21.15.6:3300/0 conn(0x7efd7c0cc790 0x7efd7c0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:01.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.158+0000 7efd7aa00640 1 -- 172.21.15.6:0/2736071048 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702177013 0 0) 0x7efd7c0ce510 con 0x7efd7c0cc3b0 2024-03-22T13:08:01.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.159+0000 7efd7aa00640 1 -- 172.21.15.6:0/2736071048 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd60003520 con 0x7efd7c0cc3b0 2024-03-22T13:08:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.159+0000 7efd7b400640 1 -- 172.21.15.6:0/2736071048 >> v1:172.21.15.92:6789/0 conn(0x7efd7c0cc3b0 legacy=0x7efd7c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.159+0000 7efd7b400640 1 -- 172.21.15.6:0/2736071048 >> v1:172.21.15.6:6789/0 conn(0x7efd7c15ab40 legacy=0x7efd7c15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:01.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.159+0000 7efd7b400640 1 -- 172.21.15.6:0/2736071048 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0ce7f0 con 0x7efd7c0cc790 2024-03-22T13:08:01.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.165+0000 7efd7b400640 1 --2- 172.21.15.6:0/2736071048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c0ccbb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efd680097a0 tx=0x7efd6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8704aa466a80b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:01.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.172+0000 7efd7aa00640 1 -- 172.21.15.6:0/2736071048 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68035020 con 0x7efd7c0cc790 2024-03-22T13:08:01.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.172+0000 7efd7aa00640 1 -- 172.21.15.6:0/2736071048 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd68031830 con 0x7efd7c0cc790 2024-03-22T13:08:01.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.172+0000 7efd7aa00640 1 -- 172.21.15.6:0/2736071048 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68031c00 con 0x7efd7c0cc790 2024-03-22T13:08:01.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.173+0000 7efd82e00640 1 -- 172.21.15.6:0/2736071048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 msgr2=0x7efd7c0ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:01.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.173+0000 7efd82e00640 1 --2- 172.21.15.6:0/2736071048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c0ccbb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efd680097a0 tx=0x7efd6802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.173+0000 7efd82e00640 1 -- 172.21.15.6:0/2736071048 shutdown_connections 2024-03-22T13:08:01.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.173+0000 7efd82e00640 1 --2- 172.21.15.6:0/2736071048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c0ccbb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.173+0000 7efd82e00640 1 -- 172.21.15.6:0/2736071048 >> 172.21.15.6:0/2736071048 conn(0x7efd7c0b16d0 msgr2=0x7efd7c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:01.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.173+0000 7efd82e00640 1 -- 172.21.15.6:0/2736071048 shutdown_connections 2024-03-22T13:08:01.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.173+0000 7efd82e00640 1 -- 172.21.15.6:0/2736071048 wait complete. 2024-03-22T13:08:01.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.177+0000 7efd82e00640 1 Processor -- start 2024-03-22T13:08:01.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.177+0000 7efd82e00640 1 -- start start 2024-03-22T13:08:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.178+0000 7efd82e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd7c0cc3b0 0x7efd7c159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.178+0000 7efd82e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c1f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.178+0000 7efd82e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7c15ab40 0x7efd7c1f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.178+0000 7efd82e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c06dc10 con 0x7efd7c0cc790 2024-03-22T13:08:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.178+0000 7efd82e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0cedb0 con 0x7efd7c15ab40 2024-03-22T13:08:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.178+0000 7efd82e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0cf940 con 0x7efd7c0cc3b0 2024-03-22T13:08:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.179+0000 7efd7b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c1f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.179+0000 7efd80e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7c15ab40 0x7efd7c1f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.179+0000 7efd7b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c1f1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60438/0 (socket says 172.21.15.6:60438) 2024-03-22T13:08:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.179+0000 7efd7b400640 1 -- 172.21.15.6:0/2564240886 learned_addr learned my addr 172.21.15.6:0/2564240886 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.180+0000 7efd7be00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd7c0cc3b0 0x7efd7c159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:01.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.183+0000 7efd80e00640 1 -- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd7c0cc3b0 msgr2=0x7efd7c159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:01.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.183+0000 7efd80e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd7c0cc3b0 0x7efd7c159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.183+0000 7efd80e00640 1 -- 172.21.15.6:0/2564240886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 msgr2=0x7efd7c1f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:01.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.183+0000 7efd80e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c1f1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.183+0000 7efd80e00640 1 -- 172.21.15.6:0/2564240886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c15a4c0 con 0x7efd7c15ab40 2024-03-22T13:08:01.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.186+0000 7efd80e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7c15ab40 0x7efd7c1f1e50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c730 tx=0x7efd7400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.191+0000 7efd73e00640 1 -- 172.21.15.6:0/2564240886 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74003b70 con 0x7efd7c15ab40 2024-03-22T13:08:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.191+0000 7efd73e00640 1 -- 172.21.15.6:0/2564240886 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd74003d10 con 0x7efd7c15ab40 2024-03-22T13:08:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.191+0000 7efd73e00640 1 -- 172.21.15.6:0/2564240886 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74005310 con 0x7efd7c15ab40 2024-03-22T13:08:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.191+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c0b3670 con 0x7efd7c15ab40 2024-03-22T13:08:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.191+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c0c9790 con 0x7efd7c15ab40 2024-03-22T13:08:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.193+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c0756c0 con 0x7efd7c15ab40 2024-03-22T13:08:01.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.231+0000 7efd73e00640 1 -- 172.21.15.6:0/2564240886 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd740056c0 con 0x7efd7c15ab40 2024-03-22T13:08:01.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.233+0000 7efd73e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd5c06bc20 0x7efd5c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.233+0000 7efd7be00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd5c06bc20 0x7efd5c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.234+0000 7efd7be00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd5c06bc20 0x7efd5c06e0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0011e0 tx=0x7efd6c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:01.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.496+0000 7efd73e00640 1 -- 172.21.15.6:0/2564240886 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7efd740578e0 con 0x7efd7c15ab40 2024-03-22T13:08:01.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.515+0000 7efd73e00640 1 -- 172.21.15.6:0/2564240886 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd74059840 con 0x7efd7c15ab40 2024-03-22T13:08:01.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.795+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd7c06de80 con 0x7efd7c15ab40 2024-03-22T13:08:01.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.815+0000 7efd73e00640 1 -- 172.21.15.6:0/2564240886 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7efd7405d8e0 con 0x7efd7c15ab40 2024-03-22T13:08:01.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:01.815 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:01.815 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.818+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd5c06bc20 msgr2=0x7efd5c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.818+0000 7efd82e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd5c06bc20 0x7efd5c06e0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0011e0 tx=0x7efd6c007d30 comp rx=0 tx=0).stop 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.818+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7c15ab40 msgr2=0x7efd7c1f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.818+0000 7efd82e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7c15ab40 0x7efd7c1f1e50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c730 tx=0x7efd7400cbf0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 shutdown_connections 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7efd5c06bc20 0x7efd5c06e0e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd7c15ab40 0x7efd7c1f1e50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd7c0cc790 0x7efd7c1f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 --2- 172.21.15.6:0/2564240886 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd7c0cc3b0 0x7efd7c159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 >> 172.21.15.6:0/2564240886 conn(0x7efd7c0b16d0 msgr2=0x7efd7c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 shutdown_connections 2024-03-22T13:08:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:01.819+0000 7efd82e00640 1 -- 172.21.15.6:0/2564240886 wait complete. 2024-03-22T13:08:02.447 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:02.448 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:02.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:08:02.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:08:03.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.641+0000 7fd219200640 1 Processor -- start 2024-03-22T13:08:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.642+0000 7fd219200640 1 -- start start 2024-03-22T13:08:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.643+0000 7fd219200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd2140cc3b0 0x7fd2140cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.643+0000 7fd219200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cf770 0x7fd2140cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.643+0000 7fd219200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd2140cccd0 0x7fd21415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.643+0000 7fd219200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2140ce230 con 0x7fd2140cccd0 2024-03-22T13:08:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.643+0000 7fd219200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2140ce510 con 0x7fd2140cc3b0 2024-03-22T13:08:03.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.643+0000 7fd219200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2140ce7f0 con 0x7fd2140cf770 2024-03-22T13:08:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.644+0000 7fd211e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd2140cc3b0 0x7fd2140cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.644+0000 7fd211400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cf770 0x7fd2140cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.644+0000 7fd212800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd2140cccd0 0x7fd21415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.644+0000 7fd211400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cf770 0x7fd2140cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51482/0 (socket says 172.21.15.6:51482) 2024-03-22T13:08:03.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.644+0000 7fd211400640 1 -- 172.21.15.6:0/4056570001 learned_addr learned my addr 172.21.15.6:0/4056570001 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.648+0000 7fd212800640 1 -- 172.21.15.6:0/4056570001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cf770 msgr2=0x7fd2140cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.648+0000 7fd212800640 1 --2- 172.21.15.6:0/4056570001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cf770 0x7fd2140cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.648+0000 7fd212800640 1 -- 172.21.15.6:0/4056570001 >> v2:172.21.15.92:3300/0 conn(0x7fd2140cc3b0 msgr2=0x7fd2140cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.648+0000 7fd212800640 1 --2- 172.21.15.6:0/4056570001 >> v2:172.21.15.92:3300/0 conn(0x7fd2140cc3b0 0x7fd2140cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.648+0000 7fd212800640 1 -- 172.21.15.6:0/4056570001 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2140cead0 con 0x7fd2140cccd0 2024-03-22T13:08:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.650+0000 7fd212800640 1 --2- 172.21.15.6:0/4056570001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cccd0 0x7fd21415ab40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd214073780 tx=0x7fd20c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c3e77aca17a231 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.670+0000 7fd210a00640 1 -- 172.21.15.6:0/4056570001 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c039070 con 0x7fd2140cccd0 2024-03-22T13:08:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.670+0000 7fd210a00640 1 -- 172.21.15.6:0/4056570001 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c02cb40 con 0x7fd2140cccd0 2024-03-22T13:08:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.670+0000 7fd210a00640 1 -- 172.21.15.6:0/4056570001 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c034440 con 0x7fd2140cccd0 2024-03-22T13:08:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.671+0000 7fd219200640 1 -- 172.21.15.6:0/4056570001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cccd0 msgr2=0x7fd21415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.671+0000 7fd219200640 1 --2- 172.21.15.6:0/4056570001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cccd0 0x7fd21415ab40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd214073780 tx=0x7fd20c02bd90 comp rx=0 tx=0).stop 2024-03-22T13:08:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.672+0000 7fd219200640 1 -- 172.21.15.6:0/4056570001 shutdown_connections 2024-03-22T13:08:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.672+0000 7fd219200640 1 --2- 172.21.15.6:0/4056570001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cccd0 0x7fd21415ab40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.672+0000 7fd219200640 1 --2- 172.21.15.6:0/4056570001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cf770 0x7fd2140cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.672+0000 7fd219200640 1 --2- 172.21.15.6:0/4056570001 >> v2:172.21.15.92:3300/0 conn(0x7fd2140cc3b0 0x7fd2140cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.672+0000 7fd219200640 1 -- 172.21.15.6:0/4056570001 >> 172.21.15.6:0/4056570001 conn(0x7fd2140b16d0 msgr2=0x7fd2140b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.672+0000 7fd219200640 1 -- 172.21.15.6:0/4056570001 shutdown_connections 2024-03-22T13:08:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.672+0000 7fd219200640 1 -- 172.21.15.6:0/4056570001 wait complete. 2024-03-22T13:08:03.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.674+0000 7fd219200640 1 Processor -- start 2024-03-22T13:08:03.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.674+0000 7fd219200640 1 -- start start 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.675+0000 7fd219200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2140cc3b0 0x7fd2140c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.675+0000 7fd219200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cccd0 0x7fd2140c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.675+0000 7fd219200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cf770 0x7fd2141507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.675+0000 7fd219200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21415a5d0 con 0x7fd2140cf770 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.675+0000 7fd219200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21406dc10 con 0x7fd2140cc3b0 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.675+0000 7fd219200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd214154bb0 con 0x7fd2140cccd0 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.676+0000 7fd211e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2140cc3b0 0x7fd2140c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.676+0000 7fd211400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cccd0 0x7fd2140c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.676+0000 7fd211400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cccd0 0x7fd2140c5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51486/0 (socket says 172.21.15.6:51486) 2024-03-22T13:08:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.676+0000 7fd211400640 1 -- 172.21.15.6:0/1440492495 learned_addr learned my addr 172.21.15.6:0/1440492495 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.677+0000 7fd212800640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cf770 0x7fd2141507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.681+0000 7fd211400640 1 -- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2140cc3b0 msgr2=0x7fd2140c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.681+0000 7fd211400640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2140cc3b0 0x7fd2140c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.681+0000 7fd211400640 1 -- 172.21.15.6:0/1440492495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cf770 msgr2=0x7fd2141507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.681+0000 7fd211400640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cf770 0x7fd2141507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.681+0000 7fd211400640 1 -- 172.21.15.6:0/1440492495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2141f0130 con 0x7fd2140cccd0 2024-03-22T13:08:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.684+0000 7fd211400640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cccd0 0x7fd2140c5b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd20000e8a0 tx=0x7fd20000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:03.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.690+0000 7fd20a000640 1 -- 172.21.15.6:0/1440492495 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200017070 con 0x7fd2140cccd0 2024-03-22T13:08:03.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.690+0000 7fd20a000640 1 -- 172.21.15.6:0/1440492495 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20000c940 con 0x7fd2140cccd0 2024-03-22T13:08:03.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.690+0000 7fd20a000640 1 -- 172.21.15.6:0/1440492495 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20000cd10 con 0x7fd2140cccd0 2024-03-22T13:08:03.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.690+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2141f03d0 con 0x7fd2140cccd0 2024-03-22T13:08:03.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.690+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2141f0990 con 0x7fd2140cccd0 2024-03-22T13:08:03.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.691+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd214154760 con 0x7fd2140cccd0 2024-03-22T13:08:03.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.724+0000 7fd20a000640 1 -- 172.21.15.6:0/1440492495 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd20001a070 con 0x7fd2140cccd0 2024-03-22T13:08:03.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.725+0000 7fd20a000640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd1f406bc20 0x7fd1f406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:03.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.726+0000 7fd211e00640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd1f406bc20 0x7fd1f406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:03.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.727+0000 7fd211e00640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd1f406bc20 0x7fd1f406e0e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd204009ac0 tx=0x7fd204009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:03.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:03.730+0000 7fd20a000640 1 -- 172.21.15.6:0/1440492495 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd200057580 con 0x7fd2140cccd0 2024-03-22T13:08:04.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.011+0000 7fd20a000640 1 -- 172.21.15.6:0/1440492495 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd214154760 con 0x7fd2140cccd0 2024-03-22T13:08:04.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.286+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2140756c0 con 0x7fd2140cccd0 2024-03-22T13:08:04.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.302+0000 7fd20a000640 1 -- 172.21.15.6:0/1440492495 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fd2000594e0 con 0x7fd2140cccd0 2024-03-22T13:08:04.303 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:04.303 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:04.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd1f406bc20 msgr2=0x7fd1f406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd1f406bc20 0x7fd1f406e0e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd204009ac0 tx=0x7fd204009190 comp rx=0 tx=0).stop 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cccd0 msgr2=0x7fd2140c5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cccd0 0x7fd2140c5b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd20000e8a0 tx=0x7fd20000ed60 comp rx=0 tx=0).stop 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 shutdown_connections 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd2140cf770 0x7fd2141507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2140cccd0 0x7fd2140c5b10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd1f406bc20 0x7fd1f406e0e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 --2- 172.21.15.6:0/1440492495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2140cc3b0 0x7fd2140c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.305+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 >> 172.21.15.6:0/1440492495 conn(0x7fd2140b16d0 msgr2=0x7fd21406cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.306+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 shutdown_connections 2024-03-22T13:08:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:04.306+0000 7fd219200640 1 -- 172.21.15.6:0/1440492495 wait complete. 2024-03-22T13:08:04.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:04.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:05.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.123+0000 7f561b200640 1 Processor -- start 2024-03-22T13:08:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.123+0000 7f561b200640 1 -- start start 2024-03-22T13:08:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.124+0000 7f561b200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f561c0cc380 0x7f561c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.125+0000 7f561b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c15b240 con 0x7f561c0ce8a0 2024-03-22T13:08:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.125+0000 7f561b200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c0cee00 con 0x7f561c15ae30 2024-03-22T13:08:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.125+0000 7f561b200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f561c0cf0c0 con 0x7f561c0cc380 2024-03-22T13:08:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.125+0000 7f5619400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f561c0ce8a0 0x7f561c0cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53442/0 (socket says 172.21.15.6:53442) 2024-03-22T13:08:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.125+0000 7f5619400640 1 -- 172.21.15.6:0/4151009213 learned_addr learned my addr 172.21.15.6:0/4151009213 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:06.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.126+0000 7f5619e00640 1 --2- 172.21.15.6:0/4151009213 >> v2:172.21.15.92:3300/0 conn(0x7f561c0cc380 0x7f561c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.129+0000 7f5619e00640 1 -- 172.21.15.6:0/4151009213 >> v1:172.21.15.92:6789/0 conn(0x7f561c15ae30 legacy=0x7f561c164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.129+0000 7f5619e00640 1 -- 172.21.15.6:0/4151009213 >> v1:172.21.15.6:6789/0 conn(0x7f561c0ce8a0 legacy=0x7f561c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.129+0000 7f5619e00640 1 -- 172.21.15.6:0/4151009213 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0cf3a0 con 0x7f561c0cc380 2024-03-22T13:08:06.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.135+0000 7f5619e00640 1 --2- 172.21.15.6:0/4151009213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0cc380 0x7f561c06c230 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f560c0097a0 tx=0x7f560c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=629213df3e6688d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:06.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.138+0000 7f5618a00640 1 -- 172.21.15.6:0/4151009213 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c034070 con 0x7f561c0cc380 2024-03-22T13:08:06.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.139+0000 7f5618a00640 1 -- 172.21.15.6:0/4151009213 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c030610 con 0x7f561c0cc380 2024-03-22T13:08:06.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.140+0000 7f5618a00640 1 -- 172.21.15.6:0/4151009213 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c0309e0 con 0x7f561c0cc380 2024-03-22T13:08:06.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.140+0000 7f561b200640 1 -- 172.21.15.6:0/4151009213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0cc380 msgr2=0x7f561c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:06.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.140+0000 7f561b200640 1 --2- 172.21.15.6:0/4151009213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0cc380 0x7f561c06c230 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f560c0097a0 tx=0x7f560c02b960 comp rx=0 tx=0).stop 2024-03-22T13:08:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.140+0000 7f561b200640 1 -- 172.21.15.6:0/4151009213 shutdown_connections 2024-03-22T13:08:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.140+0000 7f561b200640 1 --2- 172.21.15.6:0/4151009213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0cc380 0x7f561c06c230 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.140+0000 7f561b200640 1 -- 172.21.15.6:0/4151009213 >> 172.21.15.6:0/4151009213 conn(0x7f561c0b1480 msgr2=0x7f561c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.141+0000 7f561b200640 1 -- 172.21.15.6:0/4151009213 shutdown_connections 2024-03-22T13:08:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.141+0000 7f561b200640 1 -- 172.21.15.6:0/4151009213 wait complete. 2024-03-22T13:08:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.143+0000 7f561b200640 1 Processor -- start 2024-03-22T13:08:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.144+0000 7f561b200640 1 -- start start 2024-03-22T13:08:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.145+0000 7f561b200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561c0cc380 0x7f561c143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.145+0000 7f561b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0ce8a0 0x7f561c149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.145+0000 7f561b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f561c15ae30 0x7f561c149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.145+0000 7f561b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c15a910 con 0x7f561c15ae30 2024-03-22T13:08:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.145+0000 7f561b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c15a630 con 0x7f561c0ce8a0 2024-03-22T13:08:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.145+0000 7f561b200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c1549d0 con 0x7f561c0cc380 2024-03-22T13:08:06.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.146+0000 7f5619400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0ce8a0 0x7f561c149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:06.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.146+0000 7f5619e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561c0cc380 0x7f561c143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:06.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.146+0000 7f561a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f561c15ae30 0x7f561c149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:06.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.146+0000 7f5619e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561c0cc380 0x7f561c143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51502/0 (socket says 172.21.15.6:51502) 2024-03-22T13:08:06.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.146+0000 7f5619e00640 1 -- 172.21.15.6:0/3819458007 learned_addr learned my addr 172.21.15.6:0/3819458007 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.151+0000 7f5619e00640 1 -- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0ce8a0 msgr2=0x7f561c149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.151+0000 7f5619e00640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0ce8a0 0x7f561c149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.151+0000 7f5619e00640 1 -- 172.21.15.6:0/3819458007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f561c15ae30 msgr2=0x7f561c149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.151+0000 7f5619e00640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f561c15ae30 0x7f561c149980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.151+0000 7f5619e00640 1 -- 172.21.15.6:0/3819458007 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c1e4fc0 con 0x7f561c0cc380 2024-03-22T13:08:06.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.154+0000 7f5619e00640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561c0cc380 0x7f561c143fd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f560c02bf70 tx=0x7f560c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.159+0000 7f5612000640 1 -- 172.21.15.6:0/3819458007 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c034040 con 0x7f561c0cc380 2024-03-22T13:08:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.159+0000 7f5612000640 1 -- 172.21.15.6:0/3819458007 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c03acc0 con 0x7f561c0cc380 2024-03-22T13:08:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.159+0000 7f5612000640 1 -- 172.21.15.6:0/3819458007 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c044630 con 0x7f561c0cc380 2024-03-22T13:08:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.160+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561c147da0 con 0x7f561c0cc380 2024-03-22T13:08:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.160+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561c148340 con 0x7f561c0cc380 2024-03-22T13:08:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.161+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561c075650 con 0x7f561c0cc380 2024-03-22T13:08:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.188+0000 7f5612000640 1 -- 172.21.15.6:0/3819458007 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f560c037070 con 0x7f561c0cc380 2024-03-22T13:08:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.189+0000 7f5612000640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55fc06bc20 0x7f55fc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.190+0000 7f5619400640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55fc06bc20 0x7f55fc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.190+0000 7f5612000640 1 -- 172.21.15.6:0/3819458007 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f560c0794f0 con 0x7f561c0cc380 2024-03-22T13:08:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.190+0000 7f5619400640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55fc06bc20 0x7f55fc06e0e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f561c0c24b0 tx=0x7f5608009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:06.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.489+0000 7f5612000640 1 -- 172.21.15.6:0/3819458007 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f560c03f070 con 0x7f561c0cc380 2024-03-22T13:08:06.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.745+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f561c0b1d80 con 0x7f561c0cc380 2024-03-22T13:08:06.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.760+0000 7f5612000640 1 -- 172.21.15.6:0/3819458007 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f560c07b450 con 0x7f561c0cc380 2024-03-22T13:08:06.761 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:06.761 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:06.761 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.763+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55fc06bc20 msgr2=0x7f55fc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.763+0000 7f561b200640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55fc06bc20 0x7f55fc06e0e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f561c0c24b0 tx=0x7f5608009040 comp rx=0 tx=0).stop 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.763+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561c0cc380 msgr2=0x7f561c143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.763+0000 7f561b200640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561c0cc380 0x7f561c143fd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f560c02bf70 tx=0x7f560c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 shutdown_connections 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f561c15ae30 0x7f561c149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55fc06bc20 0x7f55fc06e0e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f561c0ce8a0 0x7f561c149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 --2- 172.21.15.6:0/3819458007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561c0cc380 0x7f561c143fd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 >> 172.21.15.6:0/3819458007 conn(0x7f561c0b1480 msgr2=0x7f561c06c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 shutdown_connections 2024-03-22T13:08:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:06.764+0000 7f561b200640 1 -- 172.21.15.6:0/3819458007 wait complete. 2024-03-22T13:08:07.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:07.395 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:08.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.599+0000 7fdf23200640 1 Processor -- start 2024-03-22T13:08:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.599+0000 7fdf23200640 1 -- start start 2024-03-22T13:08:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.601+0000 7fdf23200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdf1c152400 0x7fdf1c15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.601+0000 7fdf23200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c06f8c0 0x7fdf1c0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.601+0000 7fdf23200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1c1590b0 con 0x7fdf1c15bc80 2024-03-22T13:08:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.601+0000 7fdf23200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1c0724d0 con 0x7fdf1c152400 2024-03-22T13:08:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.601+0000 7fdf23200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0721f0 con 0x7fdf1c06f8c0 2024-03-22T13:08:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.602+0000 7fdf21200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c06f8c0 0x7fdf1c0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.602+0000 7fdf21200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c06f8c0 0x7fdf1c0c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51504/0 (socket says 172.21.15.6:51504) 2024-03-22T13:08:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.602+0000 7fdf21200640 1 -- 172.21.15.6:0/173899452 learned_addr learned my addr 172.21.15.6:0/173899452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.604+0000 7fdf21200640 1 -- 172.21.15.6:0/173899452 >> v2:172.21.15.6:3300/0 conn(0x7fdf1c152400 msgr2=0x7fdf1c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.604+0000 7fdf21200640 1 --2- 172.21.15.6:0/173899452 >> v2:172.21.15.6:3300/0 conn(0x7fdf1c152400 0x7fdf1c15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.604+0000 7fdf21200640 1 -- 172.21.15.6:0/173899452 >> v1:172.21.15.6:6789/0 conn(0x7fdf1c15bc80 legacy=0x7fdf1c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.604+0000 7fdf21200640 1 -- 172.21.15.6:0/173899452 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c1582f0 con 0x7fdf1c06f8c0 2024-03-22T13:08:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.606+0000 7fdf21200640 1 --2- 172.21.15.6:0/173899452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c06f8c0 0x7fdf1c0c5040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009e50 tx=0x7fdf1402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dce8120f9c56dbb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.610+0000 7fdf1aa00640 1 -- 172.21.15.6:0/173899452 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1402cb00 con 0x7fdf1c06f8c0 2024-03-22T13:08:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.610+0000 7fdf1aa00640 1 -- 172.21.15.6:0/173899452 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1402cca0 con 0x7fdf1c06f8c0 2024-03-22T13:08:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.610+0000 7fdf1aa00640 1 -- 172.21.15.6:0/173899452 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14034610 con 0x7fdf1c06f8c0 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 -- 172.21.15.6:0/173899452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c06f8c0 msgr2=0x7fdf1c0c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 --2- 172.21.15.6:0/173899452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c06f8c0 0x7fdf1c0c5040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009e50 tx=0x7fdf1402bf60 comp rx=0 tx=0).stop 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 -- 172.21.15.6:0/173899452 shutdown_connections 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 --2- 172.21.15.6:0/173899452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c06f8c0 0x7fdf1c0c5040 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 --2- 172.21.15.6:0/173899452 >> v2:172.21.15.6:3300/0 conn(0x7fdf1c152400 0x7fdf1c15b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 -- 172.21.15.6:0/173899452 >> 172.21.15.6:0/173899452 conn(0x7fdf1c0cb930 msgr2=0x7fdf1c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 -- 172.21.15.6:0/173899452 shutdown_connections 2024-03-22T13:08:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.611+0000 7fdf23200640 1 -- 172.21.15.6:0/173899452 wait complete. 2024-03-22T13:08:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.614+0000 7fdf23200640 1 Processor -- start 2024-03-22T13:08:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.614+0000 7fdf23200640 1 -- start start 2024-03-22T13:08:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.615+0000 7fdf23200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf1c06f8c0 0x7fdf1c14d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf23200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c152400 0x7fdf1c14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf23200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf1c15bc80 0x7fdf1c1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf23200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c158b90 con 0x7fdf1c15bc80 2024-03-22T13:08:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf23200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c1588b0 con 0x7fdf1c06f8c0 2024-03-22T13:08:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf23200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c1527e0 con 0x7fdf1c152400 2024-03-22T13:08:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf1b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c152400 0x7fdf1c14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf1be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf1c06f8c0 0x7fdf1c14d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf1b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c152400 0x7fdf1c14dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51506/0 (socket says 172.21.15.6:51506) 2024-03-22T13:08:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.616+0000 7fdf1b400640 1 -- 172.21.15.6:0/2033214061 learned_addr learned my addr 172.21.15.6:0/2033214061 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:08.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.620+0000 7fdf1b400640 1 -- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf1c06f8c0 msgr2=0x7fdf1c14d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:08.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.620+0000 7fdf1b400640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf1c06f8c0 0x7fdf1c14d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:08.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.620+0000 7fdf1b400640 1 -- 172.21.15.6:0/2033214061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf1c15bc80 msgr2=0x7fdf1c1560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:08.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.620+0000 7fdf1b400640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf1c15bc80 0x7fdf1c1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:08.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.620+0000 7fdf1b400640 1 -- 172.21.15.6:0/2033214061 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c1509f0 con 0x7fdf1c152400 2024-03-22T13:08:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.623+0000 7fdf1b400640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c152400 0x7fdf1c14dad0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800b3e0 tx=0x7fdf0800b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.627+0000 7fdf13e00640 1 -- 172.21.15.6:0/2033214061 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08015020 con 0x7fdf1c152400 2024-03-22T13:08:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.627+0000 7fdf13e00640 1 -- 172.21.15.6:0/2033214061 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf08011830 con 0x7fdf1c152400 2024-03-22T13:08:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.627+0000 7fdf13e00640 1 -- 172.21.15.6:0/2033214061 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08011c20 con 0x7fdf1c152400 2024-03-22T13:08:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.627+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c0c44a0 con 0x7fdf1c152400 2024-03-22T13:08:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.628+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c150f70 con 0x7fdf1c152400 2024-03-22T13:08:08.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.629+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c0756c0 con 0x7fdf1c152400 2024-03-22T13:08:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.648+0000 7fdf13e00640 1 -- 172.21.15.6:0/2033214061 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf08018020 con 0x7fdf1c152400 2024-03-22T13:08:08.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.649+0000 7fdf13e00640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdefc06bc20 0x7fdefc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.649+0000 7fdf1be00640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdefc06bc20 0x7fdefc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.650+0000 7fdf1be00640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdefc06bc20 0x7fdefc06e0e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009990 tx=0x7fdf0c009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:08.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.938+0000 7fdf13e00640 1 -- 172.21.15.6:0/2033214061 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdf08057360 con 0x7fdf1c152400 2024-03-22T13:08:08.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:08.958+0000 7fdf13e00640 1 -- 172.21.15.6:0/2033214061 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf0801f020 con 0x7fdf1c152400 2024-03-22T13:08:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.211+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf1c0c6400 con 0x7fdf1c152400 2024-03-22T13:08:09.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.232+0000 7fdf13e00640 1 -- 172.21.15.6:0/2033214061 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fdf0800f2e0 con 0x7fdf1c152400 2024-03-22T13:08:09.233 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:09.233 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:09.233 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdefc06bc20 msgr2=0x7fdefc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdefc06bc20 0x7fdefc06e0e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009990 tx=0x7fdf0c009110 comp rx=0 tx=0).stop 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c152400 msgr2=0x7fdf1c14dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c152400 0x7fdf1c14dad0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800b3e0 tx=0x7fdf0800b7e0 comp rx=0 tx=0).stop 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 shutdown_connections 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf1c15bc80 0x7fdf1c1560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf1c152400 0x7fdf1c14dad0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdefc06bc20 0x7fdefc06e0e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 --2- 172.21.15.6:0/2033214061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf1c06f8c0 0x7fdf1c14d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 >> 172.21.15.6:0/2033214061 conn(0x7fdf1c0cb930 msgr2=0x7fdf1c0c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 shutdown_connections 2024-03-22T13:08:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:09.235+0000 7fdf23200640 1 -- 172.21.15.6:0/2033214061 wait complete. 2024-03-22T13:08:09.855 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:09.855 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:10.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.054+0000 7fe966a00640 1 Processor -- start 2024-03-22T13:08:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.054+0000 7fe966a00640 1 -- start start 2024-03-22T13:08:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.055+0000 7fe966a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe960071210 0x7fe960071660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.055+0000 7fe966a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9600c9260 con 0x7fe96015d8e0 2024-03-22T13:08:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.055+0000 7fe966a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96015a2f0 con 0x7fe9601436f0 2024-03-22T13:08:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.055+0000 7fe966a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96015a640 con 0x7fe960071210 2024-03-22T13:08:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.056+0000 7fe966000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe960071210 0x7fe960071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:11.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.057+0000 7fe966000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe960071210 0x7fe960071660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:55466/0 (socket says 172.21.15.6:55466) 2024-03-22T13:08:11.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.057+0000 7fe966000640 1 -- 172.21.15.6:0/1713121266 learned_addr learned my addr 172.21.15.6:0/1713121266 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:11.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.059+0000 7fe966000640 1 -- 172.21.15.6:0/1713121266 >> v1:172.21.15.92:6789/0 conn(0x7fe9601436f0 legacy=0x7fe9600cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:11.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.059+0000 7fe966000640 1 -- 172.21.15.6:0/1713121266 >> v1:172.21.15.6:6789/0 conn(0x7fe96015d8e0 legacy=0x7fe96015aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:11.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.059+0000 7fe966000640 1 -- 172.21.15.6:0/1713121266 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe960143ad0 con 0x7fe960071210 2024-03-22T13:08:11.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.065+0000 7fe966000640 1 --2- 172.21.15.6:0/1713121266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe960071210 0x7fe960071660 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe96015a7c0 tx=0x7fe95802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=843413f65d46fa42 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:11.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.069+0000 7fe95fe00640 1 -- 172.21.15.6:0/1713121266 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95802cb00 con 0x7fe960071210 2024-03-22T13:08:11.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.071+0000 7fe95fe00640 1 -- 172.21.15.6:0/1713121266 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95802cca0 con 0x7fe960071210 2024-03-22T13:08:11.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.071+0000 7fe95fe00640 1 -- 172.21.15.6:0/1713121266 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958034610 con 0x7fe960071210 2024-03-22T13:08:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.071+0000 7fe966a00640 1 -- 172.21.15.6:0/1713121266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe960071210 msgr2=0x7fe960071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.071+0000 7fe966a00640 1 --2- 172.21.15.6:0/1713121266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe960071210 0x7fe960071660 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe96015a7c0 tx=0x7fe95802bf60 comp rx=0 tx=0).stop 2024-03-22T13:08:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.071+0000 7fe966a00640 1 -- 172.21.15.6:0/1713121266 shutdown_connections 2024-03-22T13:08:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.072+0000 7fe966a00640 1 --2- 172.21.15.6:0/1713121266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe960071210 0x7fe960071660 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.072+0000 7fe966a00640 1 -- 172.21.15.6:0/1713121266 >> 172.21.15.6:0/1713121266 conn(0x7fe96006b4a0 msgr2=0x7fe96006f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:11.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.072+0000 7fe966a00640 1 -- 172.21.15.6:0/1713121266 shutdown_connections 2024-03-22T13:08:11.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.072+0000 7fe966a00640 1 -- 172.21.15.6:0/1713121266 wait complete. 2024-03-22T13:08:11.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.073+0000 7fe966a00640 1 Processor -- start 2024-03-22T13:08:11.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.074+0000 7fe966a00640 1 -- start start 2024-03-22T13:08:11.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.074+0000 7fe966a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe960071210 0x7fe96015bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe966a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9601436f0 0x7fe9600cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe966a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe96015d8e0 0x7fe9600cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe966a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96015e390 con 0x7fe9601436f0 2024-03-22T13:08:11.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe966a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96015e0b0 con 0x7fe96015d8e0 2024-03-22T13:08:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe966a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96015e670 con 0x7fe960071210 2024-03-22T13:08:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe966000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe96015d8e0 0x7fe9600cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe964c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9601436f0 0x7fe9600cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.075+0000 7fe965600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe960071210 0x7fe96015bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.076+0000 7fe966000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe96015d8e0 0x7fe9600cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:55468/0 (socket says 172.21.15.6:55468) 2024-03-22T13:08:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.076+0000 7fe966000640 1 -- 172.21.15.6:0/3958745838 learned_addr learned my addr 172.21.15.6:0/3958745838 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:11.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.079+0000 7fe966000640 1 -- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe960071210 msgr2=0x7fe96015bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:11.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.079+0000 7fe966000640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe960071210 0x7fe96015bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:11.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.079+0000 7fe966000640 1 -- 172.21.15.6:0/3958745838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9601436f0 msgr2=0x7fe9600cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:11.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.079+0000 7fe966000640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9601436f0 0x7fe9600cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:11.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.079+0000 7fe966000640 1 -- 172.21.15.6:0/3958745838 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600cd640 con 0x7fe96015d8e0 2024-03-22T13:08:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.082+0000 7fe966000640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe96015d8e0 0x7fe9600cdbc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe958009f10 tx=0x7fe958004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:11.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.089+0000 7fe95d600640 1 -- 172.21.15.6:0/3958745838 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958043020 con 0x7fe96015d8e0 2024-03-22T13:08:11.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.089+0000 7fe95d600640 1 -- 172.21.15.6:0/3958745838 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95802ce50 con 0x7fe96015d8e0 2024-03-22T13:08:11.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.089+0000 7fe95d600640 1 -- 172.21.15.6:0/3958745838 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95803e670 con 0x7fe96015d8e0 2024-03-22T13:08:11.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.089+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9600cd360 con 0x7fe96015d8e0 2024-03-22T13:08:11.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.089+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9601f0400 con 0x7fe96015d8e0 2024-03-22T13:08:11.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.090+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe960075650 con 0x7fe96015d8e0 2024-03-22T13:08:11.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.129+0000 7fe95d600640 1 -- 172.21.15.6:0/3958745838 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe958046020 con 0x7fe96015d8e0 2024-03-22T13:08:11.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.130+0000 7fe95d600640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe94006bc20 0x7fe94006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:11.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.131+0000 7fe965600640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe94006bc20 0x7fe94006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.132+0000 7fe965600640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe94006bc20 0x7fe94006e0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe954000f60 tx=0x7fe954009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.133+0000 7fe95d600640 1 -- 172.21.15.6:0/3958745838 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe958079ca0 con 0x7fe96015d8e0 2024-03-22T13:08:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.383+0000 7fe95d600640 1 -- 172.21.15.6:0/3958745838 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe95804d030 con 0x7fe96015d8e0 2024-03-22T13:08:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.651+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe960157f40 con 0x7fe96015d8e0 2024-03-22T13:08:11.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.745+0000 7fe95d600640 1 -- 172.21.15.6:0/3958745838 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fe95807bc00 con 0x7fe96015d8e0 2024-03-22T13:08:11.746 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:11.746 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:11.746 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe94006bc20 msgr2=0x7fe94006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe94006bc20 0x7fe94006e0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe954000f60 tx=0x7fe954009040 comp rx=0 tx=0).stop 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe96015d8e0 msgr2=0x7fe9600cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe96015d8e0 0x7fe9600cdbc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe958009f10 tx=0x7fe958004a40 comp rx=0 tx=0).stop 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 shutdown_connections 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe94006bc20 0x7fe94006e0e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe96015d8e0 0x7fe9600cdbc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9601436f0 0x7fe9600cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 --2- 172.21.15.6:0/3958745838 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe960071210 0x7fe96015bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:11.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 >> 172.21.15.6:0/3958745838 conn(0x7fe96006b4a0 msgr2=0x7fe9600cab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:11.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 shutdown_connections 2024-03-22T13:08:11.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:11.748+0000 7fe966a00640 1 -- 172.21.15.6:0/3958745838 wait complete. 2024-03-22T13:08:12.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:12.371 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:13.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:13.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.573+0000 7fc339a00640 1 Processor -- start 2024-03-22T13:08:13.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.573+0000 7fc339a00640 1 -- start start 2024-03-22T13:08:13.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.575+0000 7fc339a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc334152400 0x7fc33415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:13.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.575+0000 7fc339a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc33415bc80 0x7fc3340716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:13.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.575+0000 7fc339a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc33406f8c0 0x7fc334071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:13.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.575+0000 7fc339a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3340724d0 con 0x7fc33415bc80 2024-03-22T13:08:13.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.575+0000 7fc339a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3341590b0 con 0x7fc33406f8c0 2024-03-22T13:08:13.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.575+0000 7fc339a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3340721f0 con 0x7fc334152400 2024-03-22T13:08:13.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.576+0000 7fc333400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc33406f8c0 0x7fc334071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:13.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.576+0000 7fc332a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc334152400 0x7fc33415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:13.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.576+0000 7fc332000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc33415bc80 0x7fc3340716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:13.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.576+0000 7fc332a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc334152400 0x7fc33415b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34934/0 (socket says 172.21.15.6:34934) 2024-03-22T13:08:13.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.576+0000 7fc332a00640 1 -- 172.21.15.6:0/2084464405 learned_addr learned my addr 172.21.15.6:0/2084464405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.580+0000 7fc333400640 1 -- 172.21.15.6:0/2084464405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc334152400 msgr2=0x7fc33415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.580+0000 7fc333400640 1 --2- 172.21.15.6:0/2084464405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc334152400 0x7fc33415b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.580+0000 7fc333400640 1 -- 172.21.15.6:0/2084464405 >> v2:172.21.15.6:3300/0 conn(0x7fc33415bc80 msgr2=0x7fc3340716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.580+0000 7fc333400640 1 --2- 172.21.15.6:0/2084464405 >> v2:172.21.15.6:3300/0 conn(0x7fc33415bc80 0x7fc3340716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.580+0000 7fc333400640 1 -- 172.21.15.6:0/2084464405 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3341582f0 con 0x7fc33406f8c0 2024-03-22T13:08:13.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.583+0000 7fc333400640 1 --2- 172.21.15.6:0/2084464405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33406f8c0 0x7fc334071c10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc33406e7f0 tx=0x7fc32c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d66b0bf1c211b662 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.587+0000 7fc331600640 1 -- 172.21.15.6:0/2084464405 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c039070 con 0x7fc33406f8c0 2024-03-22T13:08:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.587+0000 7fc331600640 1 -- 172.21.15.6:0/2084464405 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c02cc60 con 0x7fc33406f8c0 2024-03-22T13:08:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.587+0000 7fc331600640 1 -- 172.21.15.6:0/2084464405 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c034660 con 0x7fc33406f8c0 2024-03-22T13:08:13.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.588+0000 7fc339a00640 1 -- 172.21.15.6:0/2084464405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33406f8c0 msgr2=0x7fc334071c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:13.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.588+0000 7fc339a00640 1 --2- 172.21.15.6:0/2084464405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33406f8c0 0x7fc334071c10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc33406e7f0 tx=0x7fc32c02beb0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.588+0000 7fc339a00640 1 -- 172.21.15.6:0/2084464405 shutdown_connections 2024-03-22T13:08:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.588+0000 7fc339a00640 1 --2- 172.21.15.6:0/2084464405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33406f8c0 0x7fc334071c10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.588+0000 7fc339a00640 1 --2- 172.21.15.6:0/2084464405 >> v2:172.21.15.6:3300/0 conn(0x7fc33415bc80 0x7fc3340716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.588+0000 7fc339a00640 1 --2- 172.21.15.6:0/2084464405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc334152400 0x7fc33415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.589+0000 7fc339a00640 1 -- 172.21.15.6:0/2084464405 >> 172.21.15.6:0/2084464405 conn(0x7fc3340cb930 msgr2=0x7fc3340cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.589+0000 7fc339a00640 1 -- 172.21.15.6:0/2084464405 shutdown_connections 2024-03-22T13:08:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.589+0000 7fc339a00640 1 -- 172.21.15.6:0/2084464405 wait complete. 2024-03-22T13:08:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.591+0000 7fc339a00640 1 Processor -- start 2024-03-22T13:08:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.592+0000 7fc339a00640 1 -- start start 2024-03-22T13:08:13.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.593+0000 7fc339a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc33406f8c0 0x7fc3341571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:13.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.593+0000 7fc339a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc334152400 0x7fc3341561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:13.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.593+0000 7fc339a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc33415bc80 0x7fc334150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:13.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.593+0000 7fc339a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc334158b90 con 0x7fc33406f8c0 2024-03-22T13:08:13.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.593+0000 7fc339a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3341588b0 con 0x7fc334152400 2024-03-22T13:08:13.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.593+0000 7fc339a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3341527e0 con 0x7fc33415bc80 2024-03-22T13:08:13.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.594+0000 7fc332000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc334152400 0x7fc3341561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:13.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.594+0000 7fc333400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc33415bc80 0x7fc334150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:13.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.594+0000 7fc332a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc33406f8c0 0x7fc3341571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:13.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.594+0000 7fc332000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc334152400 0x7fc3341561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52060/0 (socket says 172.21.15.6:52060) 2024-03-22T13:08:13.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.594+0000 7fc332000640 1 -- 172.21.15.6:0/3791658808 learned_addr learned my addr 172.21.15.6:0/3791658808 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.599+0000 7fc332000640 1 -- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc33415bc80 msgr2=0x7fc334150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.599+0000 7fc332000640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc33415bc80 0x7fc334150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.599+0000 7fc332000640 1 -- 172.21.15.6:0/3791658808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc33406f8c0 msgr2=0x7fc3341571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.599+0000 7fc332000640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc33406f8c0 0x7fc3341571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.599+0000 7fc332000640 1 -- 172.21.15.6:0/3791658808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3341f0880 con 0x7fc334152400 2024-03-22T13:08:13.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.603+0000 7fc332000640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc334152400 0x7fc3341561b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc32000e840 tx=0x7fc32000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.608+0000 7fc32aa00640 1 -- 172.21.15.6:0/3791658808 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320017070 con 0x7fc334152400 2024-03-22T13:08:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.608+0000 7fc32aa00640 1 -- 172.21.15.6:0/3791658808 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32000c940 con 0x7fc334152400 2024-03-22T13:08:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.608+0000 7fc32aa00640 1 -- 172.21.15.6:0/3791658808 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32000cd10 con 0x7fc334152400 2024-03-22T13:08:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.608+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3341f0b60 con 0x7fc334152400 2024-03-22T13:08:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.608+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3341f1120 con 0x7fc334152400 2024-03-22T13:08:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.609+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3340756c0 con 0x7fc334152400 2024-03-22T13:08:13.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.641+0000 7fc32aa00640 1 -- 172.21.15.6:0/3791658808 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc32001a070 con 0x7fc334152400 2024-03-22T13:08:13.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.643+0000 7fc32aa00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc31406bc20 0x7fc31406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:13.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.643+0000 7fc332a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc31406bc20 0x7fc31406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:13.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.644+0000 7fc332a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc31406bc20 0x7fc31406e0e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc324006cc0 tx=0x7fc324009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.888+0000 7fc32aa00640 1 -- 172.21.15.6:0/3791658808 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc320057530 con 0x7fc334152400 2024-03-22T13:08:13.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:13.899+0000 7fc32aa00640 1 -- 172.21.15.6:0/3791658808 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc320059490 con 0x7fc334152400 2024-03-22T13:08:14.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.178+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3340c6400 con 0x7fc334152400 2024-03-22T13:08:14.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.199+0000 7fc32aa00640 1 -- 172.21.15.6:0/3791658808 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fc32005d530 con 0x7fc334152400 2024-03-22T13:08:14.200 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:14.200 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:14.200 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:14.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.201+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc31406bc20 msgr2=0x7fc31406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.201+0000 7fc339a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc31406bc20 0x7fc31406e0e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc324006cc0 tx=0x7fc324009240 comp rx=0 tx=0).stop 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.201+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc334152400 msgr2=0x7fc3341561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.201+0000 7fc339a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc334152400 0x7fc3341561b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc32000e840 tx=0x7fc32000ed00 comp rx=0 tx=0).stop 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 shutdown_connections 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc33415bc80 0x7fc334150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc31406bc20 0x7fc31406e0e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc334152400 0x7fc3341561b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 --2- 172.21.15.6:0/3791658808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc33406f8c0 0x7fc3341571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 >> 172.21.15.6:0/3791658808 conn(0x7fc3340cb930 msgr2=0x7fc334147e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 shutdown_connections 2024-03-22T13:08:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:14.202+0000 7fc339a00640 1 -- 172.21.15.6:0/3791658808 wait complete. 2024-03-22T13:08:14.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:14.825 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:15.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:16.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.023+0000 7fa814a00640 1 Processor -- start 2024-03-22T13:08:16.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.023+0000 7fa814a00640 1 -- start start 2024-03-22T13:08:16.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.024+0000 7fa814a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8100ce8a0 0x7fa81006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:16.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.024+0000 7fa814a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa81015ae30 0x7fa8100cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:16.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.024+0000 7fa814a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81015b260 con 0x7fa8100cc380 2024-03-22T13:08:16.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.024+0000 7fa814a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8100cee00 con 0x7fa81015ae30 2024-03-22T13:08:16.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.024+0000 7fa814a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8100cf0c0 con 0x7fa8100ce8a0 2024-03-22T13:08:16.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.025+0000 7fa80ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8100ce8a0 0x7fa81006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:16.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.025+0000 7fa80fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa81015ae30 0x7fa8100cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:16.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.025+0000 7fa80f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8100cc380 0x7fa8100cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39860/0 (socket says 172.21.15.6:39860) 2024-03-22T13:08:16.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.025+0000 7fa80f400640 1 -- 172.21.15.6:0/3231651909 learned_addr learned my addr 172.21.15.6:0/3231651909 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:16.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.028+0000 7fa80ea00640 1 -- 172.21.15.6:0/3231651909 >> v2:172.21.15.92:3300/0 conn(0x7fa81015ae30 msgr2=0x7fa8100cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:16.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.028+0000 7fa80ea00640 1 --2- 172.21.15.6:0/3231651909 >> v2:172.21.15.92:3300/0 conn(0x7fa81015ae30 0x7fa8100cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.028+0000 7fa80ea00640 1 -- 172.21.15.6:0/3231651909 >> v1:172.21.15.6:6789/0 conn(0x7fa8100cc380 legacy=0x7fa8100cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:16.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.028+0000 7fa80ea00640 1 -- 172.21.15.6:0/3231651909 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8100cf3a0 con 0x7fa8100ce8a0 2024-03-22T13:08:16.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.030+0000 7fa80ea00640 1 --2- 172.21.15.6:0/3231651909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8100ce8a0 0x7fa81006c230 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80097a0 tx=0x7fa7f802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43c68f3332adc9dc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.034+0000 7fa80e000640 1 -- 172.21.15.6:0/3231651909 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8035020 con 0x7fa8100ce8a0 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.034+0000 7fa80e000640 1 -- 172.21.15.6:0/3231651909 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f8031830 con 0x7fa8100ce8a0 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa80e000640 1 -- 172.21.15.6:0/3231651909 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8031c00 con 0x7fa8100ce8a0 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 -- 172.21.15.6:0/3231651909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8100ce8a0 msgr2=0x7fa81006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 --2- 172.21.15.6:0/3231651909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8100ce8a0 0x7fa81006c230 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80097a0 tx=0x7fa7f802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 -- 172.21.15.6:0/3231651909 shutdown_connections 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 --2- 172.21.15.6:0/3231651909 >> v2:172.21.15.92:3300/0 conn(0x7fa81015ae30 0x7fa8100cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 --2- 172.21.15.6:0/3231651909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8100ce8a0 0x7fa81006c230 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 -- 172.21.15.6:0/3231651909 >> 172.21.15.6:0/3231651909 conn(0x7fa8100b1480 msgr2=0x7fa8100b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:16.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 -- 172.21.15.6:0/3231651909 shutdown_connections 2024-03-22T13:08:16.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.035+0000 7fa814a00640 1 -- 172.21.15.6:0/3231651909 wait complete. 2024-03-22T13:08:16.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.037+0000 7fa814a00640 1 Processor -- start 2024-03-22T13:08:16.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.037+0000 7fa814a00640 1 -- start start 2024-03-22T13:08:16.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.038+0000 7fa814a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8100cc380 0x7fa8101532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:16.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.038+0000 7fa814a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8100ce8a0 0x7fa8101e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:16.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.038+0000 7fa814a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa81015ae30 0x7fa8101570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:16.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.038+0000 7fa814a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81015a910 con 0x7fa8100cc380 2024-03-22T13:08:16.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.038+0000 7fa814a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81015a630 con 0x7fa8100ce8a0 2024-03-22T13:08:16.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.038+0000 7fa814a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8101549d0 con 0x7fa81015ae30 2024-03-22T13:08:16.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.039+0000 7fa80ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8100ce8a0 0x7fa8101e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:16.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.039+0000 7fa80f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8100cc380 0x7fa8101532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:16.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.039+0000 7fa80f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8100cc380 0x7fa8101532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40598/0 (socket says 172.21.15.6:40598) 2024-03-22T13:08:16.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.039+0000 7fa80f400640 1 -- 172.21.15.6:0/601780037 learned_addr learned my addr 172.21.15.6:0/601780037 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:16.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.040+0000 7fa80fe00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa81015ae30 0x7fa8101570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:16.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.043+0000 7fa80f400640 1 -- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa81015ae30 msgr2=0x7fa8101570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:16.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.043+0000 7fa80f400640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa81015ae30 0x7fa8101570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.043+0000 7fa80f400640 1 -- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8100ce8a0 msgr2=0x7fa8101e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:16.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.043+0000 7fa80f400640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8100ce8a0 0x7fa8101e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.043+0000 7fa80f400640 1 -- 172.21.15.6:0/601780037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8101e3650 con 0x7fa8100cc380 2024-03-22T13:08:16.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.046+0000 7fa80f400640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8100cc380 0x7fa8101532b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa80000caa0 tx=0x7fa80000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.053+0000 7fa807400640 1 -- 172.21.15.6:0/601780037 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800017070 con 0x7fa8100cc380 2024-03-22T13:08:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.053+0000 7fa807400640 1 -- 172.21.15.6:0/601780037 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80000ac00 con 0x7fa8100cc380 2024-03-22T13:08:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.053+0000 7fa807400640 1 -- 172.21.15.6:0/601780037 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800012400 con 0x7fa8100cc380 2024-03-22T13:08:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.053+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8101e3930 con 0x7fa8100cc380 2024-03-22T13:08:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.053+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8101568e0 con 0x7fa8100cc380 2024-03-22T13:08:16.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.055+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa810075650 con 0x7fa8100cc380 2024-03-22T13:08:16.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.375+0000 7fa807400640 1 -- 172.21.15.6:0/601780037 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa80001a070 con 0x7fa8100cc380 2024-03-22T13:08:16.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.377+0000 7fa807400640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa7ec06bc20 0x7fa7ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:16.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.378+0000 7fa80ea00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa7ec06bc20 0x7fa7ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:16.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.379+0000 7fa80ea00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa7ec06bc20 0x7fa7ec06e0e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8036000 tx=0x7fa7f8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.402+0000 7fa807400640 1 -- 172.21.15.6:0/601780037 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa8000570d0 con 0x7fa8100cc380 2024-03-22T13:08:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.402+0000 7fa807400640 1 -- 172.21.15.6:0/601780037 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa800021020 con 0x7fa8100cc380 2024-03-22T13:08:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.656+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa810159670 con 0x7fa8100cc380 2024-03-22T13:08:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.676+0000 7fa807400640 1 -- 172.21.15.6:0/601780037 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fa81015b260 con 0x7fa8100cc380 2024-03-22T13:08:16.676 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:16.676 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.678+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa7ec06bc20 msgr2=0x7fa7ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.678+0000 7fa814a00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa7ec06bc20 0x7fa7ec06e0e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8036000 tx=0x7fa7f8008030 comp rx=0 tx=0).stop 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.678+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8100cc380 msgr2=0x7fa8101532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.678+0000 7fa814a00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8100cc380 0x7fa8101532b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa80000caa0 tx=0x7fa80000cf60 comp rx=0 tx=0).stop 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.678+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 shutdown_connections 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.678+0000 7fa814a00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa81015ae30 0x7fa8101570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.678+0000 7fa814a00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa7ec06bc20 0x7fa7ec06e0e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.679+0000 7fa814a00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8100ce8a0 0x7fa8101e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.679+0000 7fa814a00640 1 --2- 172.21.15.6:0/601780037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8100cc380 0x7fa8101532b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.679+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 >> 172.21.15.6:0/601780037 conn(0x7fa8100b1480 msgr2=0x7fa810157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.679+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 shutdown_connections 2024-03-22T13:08:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:16.679+0000 7fa814a00640 1 -- 172.21.15.6:0/601780037 wait complete. 2024-03-22T13:08:17.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:17.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:18.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:18.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.497+0000 7f8962600640 1 Processor -- start 2024-03-22T13:08:18.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.497+0000 7f8962600640 1 -- start start 2024-03-22T13:08:18.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.498+0000 7f8962600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c0cca20 0x7f895c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:18.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.498+0000 7f8962600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895c06c440 con 0x7f895c0cc640 2024-03-22T13:08:18.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.498+0000 7f8962600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895c06c720 con 0x7f895c15b7a0 2024-03-22T13:08:18.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.498+0000 7f8962600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895c06d330 con 0x7f895c0cca20 2024-03-22T13:08:18.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.525+0000 7f8961c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f895c15b7a0 0x7f895c0ca340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60140/0 (socket says 172.21.15.6:60140) 2024-03-22T13:08:18.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.525+0000 7f8961c00640 1 -- 172.21.15.6:0/2315555056 learned_addr learned my addr 172.21.15.6:0/2315555056 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:18.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.548+0000 7f895be00640 1 --2- 172.21.15.6:0/2315555056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c0cca20 0x7f895c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:18.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.552+0000 7f895be00640 1 -- 172.21.15.6:0/2315555056 >> v1:172.21.15.92:6789/0 conn(0x7f895c15b7a0 legacy=0x7f895c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:18.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.552+0000 7f895be00640 1 -- 172.21.15.6:0/2315555056 >> v1:172.21.15.6:6789/0 conn(0x7f895c0cc640 legacy=0x7f895c06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:18.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.552+0000 7f895be00640 1 -- 172.21.15.6:0/2315555056 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c06ca00 con 0x7f895c0cca20 2024-03-22T13:08:18.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.555+0000 7f895be00640 1 --2- 172.21.15.6:0/2315555056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c0cca20 0x7f895c15e260 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f89500097d0 tx=0x7f895002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0b59c170361a619 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:18.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.562+0000 7f895b400640 1 -- 172.21.15.6:0/2315555056 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950034070 con 0x7f895c0cca20 2024-03-22T13:08:18.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.562+0000 7f895b400640 1 -- 172.21.15.6:0/2315555056 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8950030800 con 0x7f895c0cca20 2024-03-22T13:08:18.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.562+0000 7f895b400640 1 -- 172.21.15.6:0/2315555056 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950030bd0 con 0x7f895c0cca20 2024-03-22T13:08:18.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.563+0000 7f8962600640 1 -- 172.21.15.6:0/2315555056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c0cca20 msgr2=0x7f895c15e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:18.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.563+0000 7f8962600640 1 --2- 172.21.15.6:0/2315555056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c0cca20 0x7f895c15e260 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f89500097d0 tx=0x7f895002b940 comp rx=0 tx=0).stop 2024-03-22T13:08:18.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.564+0000 7f8962600640 1 -- 172.21.15.6:0/2315555056 shutdown_connections 2024-03-22T13:08:18.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.564+0000 7f8962600640 1 --2- 172.21.15.6:0/2315555056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c0cca20 0x7f895c15e260 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:18.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.564+0000 7f8962600640 1 -- 172.21.15.6:0/2315555056 >> 172.21.15.6:0/2315555056 conn(0x7f895c0cf9c0 msgr2=0x7f895c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:18.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.564+0000 7f8962600640 1 -- 172.21.15.6:0/2315555056 shutdown_connections 2024-03-22T13:08:18.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.564+0000 7f8962600640 1 -- 172.21.15.6:0/2315555056 wait complete. 2024-03-22T13:08:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.566+0000 7f8962600640 1 Processor -- start 2024-03-22T13:08:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.567+0000 7f8962600640 1 -- start start 2024-03-22T13:08:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.568+0000 7f8962600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f895c0cc640 0x7f895c1f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.568+0000 7f8962600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f895c0cca20 0x7f895c151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.568+0000 7f8962600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c15b7a0 0x7f895c1e5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.568+0000 7f8962600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895c0caa50 con 0x7f895c0cca20 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.568+0000 7f8962600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895c06cfc0 con 0x7f895c0cc640 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.568+0000 7f8962600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895c15b000 con 0x7f895c15b7a0 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.569+0000 7f895be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f895c0cca20 0x7f895c151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.569+0000 7f8961c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c15b7a0 0x7f895c1e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.569+0000 7f8961200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f895c0cc640 0x7f895c1f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.569+0000 7f8961200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f895c0cc640 0x7f895c1f0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52096/0 (socket says 172.21.15.6:52096) 2024-03-22T13:08:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.569+0000 7f8961200640 1 -- 172.21.15.6:0/1846795042 learned_addr learned my addr 172.21.15.6:0/1846795042 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.573+0000 7f8961200640 1 -- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c15b7a0 msgr2=0x7f895c1e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.573+0000 7f8961200640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c15b7a0 0x7f895c1e5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.573+0000 7f8961200640 1 -- 172.21.15.6:0/1846795042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f895c0cca20 msgr2=0x7f895c151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.573+0000 7f8961200640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f895c0cca20 0x7f895c151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.573+0000 7f8961200640 1 -- 172.21.15.6:0/1846795042 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c0c9bf0 con 0x7f895c0cc640 2024-03-22T13:08:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.575+0000 7f8961200640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f895c0cc640 0x7f895c1f0a20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f894800c440 tx=0x7f894800c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.579+0000 7f8958c00640 1 -- 172.21.15.6:0/1846795042 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948017070 con 0x7f895c0cc640 2024-03-22T13:08:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.579+0000 7f8958c00640 1 -- 172.21.15.6:0/1846795042 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89480035f0 con 0x7f895c0cc640 2024-03-22T13:08:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.579+0000 7f8958c00640 1 -- 172.21.15.6:0/1846795042 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480039e0 con 0x7f895c0cc640 2024-03-22T13:08:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.579+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895c1517c0 con 0x7f895c0cc640 2024-03-22T13:08:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.580+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895c151d80 con 0x7f895c0cc640 2024-03-22T13:08:18.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.581+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895c075650 con 0x7f895c0cc640 2024-03-22T13:08:18.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.605+0000 7f8958c00640 1 -- 172.21.15.6:0/1846795042 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f894801a070 con 0x7f895c0cc640 2024-03-22T13:08:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.606+0000 7f8958c00640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f893c06bc20 0x7f893c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:18.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.606+0000 7f895be00640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f893c06bc20 0x7f893c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:18.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.607+0000 7f895be00640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f893c06bc20 0x7f893c06e0e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8950036000 tx=0x7f895003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:18.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.864+0000 7f8958c00640 1 -- 172.21.15.6:0/1846795042 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8948056730 con 0x7f895c0cc640 2024-03-22T13:08:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:18.876+0000 7f8958c00640 1 -- 172.21.15.6:0/1846795042 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8948021020 con 0x7f895c0cc640 2024-03-22T13:08:19.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.137+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f895c0cce60 con 0x7f895c0cc640 2024-03-22T13:08:19.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.152+0000 7f8958c00640 1 -- 172.21.15.6:0/1846795042 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f89480010c0 con 0x7f895c0cc640 2024-03-22T13:08:19.152 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:19.152 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:19.152 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:19.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f893c06bc20 msgr2=0x7f893c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f893c06bc20 0x7f893c06e0e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8950036000 tx=0x7f895003b040 comp rx=0 tx=0).stop 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f895c0cc640 msgr2=0x7f895c1f0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f895c0cc640 0x7f895c1f0a20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f894800c440 tx=0x7f894800c900 comp rx=0 tx=0).stop 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 shutdown_connections 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f895c15b7a0 0x7f895c1e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f895c0cca20 0x7f895c151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f893c06bc20 0x7f893c06e0e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.154+0000 7f8962600640 1 --2- 172.21.15.6:0/1846795042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f895c0cc640 0x7f895c1f0a20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.155+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 >> 172.21.15.6:0/1846795042 conn(0x7f895c0cf9c0 msgr2=0x7f895c06d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.155+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 shutdown_connections 2024-03-22T13:08:19.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:19.155+0000 7f8962600640 1 -- 172.21.15.6:0/1846795042 wait complete. 2024-03-22T13:08:19.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:19.776 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:20.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.978+0000 7fb6d1000640 1 Processor -- start 2024-03-22T13:08:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.978+0000 7fb6d1000640 1 -- start start 2024-03-22T13:08:20.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.980+0000 7fb6d1000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb6cc152400 0x7fb6cc15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:20.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.980+0000 7fb6d1000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc15bc80 0x7fb6cc0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.980+0000 7fb6d1000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc1590b0 con 0x7fb6cc06f8c0 2024-03-22T13:08:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.980+0000 7fb6d1000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6cc0724d0 con 0x7fb6cc152400 2024-03-22T13:08:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.980+0000 7fb6d1000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0721f0 con 0x7fb6cc15bc80 2024-03-22T13:08:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.981+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc15bc80 0x7fb6cc0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.981+0000 7fb6c9e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb6cc152400 0x7fb6cc15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.981+0000 7fb6ca800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb6cc06f8c0 0x7fb6cc0c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60154/0 (socket says 172.21.15.6:60154) 2024-03-22T13:08:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.981+0000 7fb6ca800640 1 -- 172.21.15.6:0/431412672 learned_addr learned my addr 172.21.15.6:0/431412672 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.984+0000 7fb6c9400640 1 -- 172.21.15.6:0/431412672 >> v2:172.21.15.92:3300/0 conn(0x7fb6cc152400 msgr2=0x7fb6cc15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.984+0000 7fb6c9400640 1 --2- 172.21.15.6:0/431412672 >> v2:172.21.15.92:3300/0 conn(0x7fb6cc152400 0x7fb6cc15b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.984+0000 7fb6c9400640 1 -- 172.21.15.6:0/431412672 >> v1:172.21.15.92:6789/0 conn(0x7fb6cc06f8c0 legacy=0x7fb6cc0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:20.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.984+0000 7fb6c9400640 1 -- 172.21.15.6:0/431412672 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc1582f0 con 0x7fb6cc15bc80 2024-03-22T13:08:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.987+0000 7fb6c9400640 1 --2- 172.21.15.6:0/431412672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc15bc80 0x7fb6cc0716d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80097a0 tx=0x7fb6b802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20985a7ae2577ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:20.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.990+0000 7fb6c8a00640 1 -- 172.21.15.6:0/431412672 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8034070 con 0x7fb6cc15bc80 2024-03-22T13:08:20.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.992+0000 7fb6c8a00640 1 -- 172.21.15.6:0/431412672 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b8030610 con 0x7fb6cc15bc80 2024-03-22T13:08:20.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.992+0000 7fb6c8a00640 1 -- 172.21.15.6:0/431412672 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80309e0 con 0x7fb6cc15bc80 2024-03-22T13:08:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 -- 172.21.15.6:0/431412672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc15bc80 msgr2=0x7fb6cc0716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 --2- 172.21.15.6:0/431412672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc15bc80 0x7fb6cc0716d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80097a0 tx=0x7fb6b802b960 comp rx=0 tx=0).stop 2024-03-22T13:08:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 -- 172.21.15.6:0/431412672 shutdown_connections 2024-03-22T13:08:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 --2- 172.21.15.6:0/431412672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc15bc80 0x7fb6cc0716d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 --2- 172.21.15.6:0/431412672 >> v2:172.21.15.92:3300/0 conn(0x7fb6cc152400 0x7fb6cc15b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:20.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 -- 172.21.15.6:0/431412672 >> 172.21.15.6:0/431412672 conn(0x7fb6cc0cb930 msgr2=0x7fb6cc0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 -- 172.21.15.6:0/431412672 shutdown_connections 2024-03-22T13:08:20.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.993+0000 7fb6d1000640 1 -- 172.21.15.6:0/431412672 wait complete. 2024-03-22T13:08:20.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.996+0000 7fb6d1000640 1 Processor -- start 2024-03-22T13:08:20.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.996+0000 7fb6d1000640 1 -- start start 2024-03-22T13:08:20.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.997+0000 7fb6d1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6cc06f8c0 0x7fb6cc157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:20.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.997+0000 7fb6d1000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc152400 0x7fb6cc0cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:20.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.997+0000 7fb6d1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc15bc80 0x7fb6cc14dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:20.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.997+0000 7fb6d1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc158b90 con 0x7fb6cc06f8c0 2024-03-22T13:08:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.997+0000 7fb6d1000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc1588b0 con 0x7fb6cc15bc80 2024-03-22T13:08:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.997+0000 7fb6d1000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc1527e0 con 0x7fb6cc152400 2024-03-22T13:08:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.998+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc152400 0x7fb6cc0cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.998+0000 7fb6ca800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc15bc80 0x7fb6cc14dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.998+0000 7fb6c9400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc152400 0x7fb6cc0cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34982/0 (socket says 172.21.15.6:34982) 2024-03-22T13:08:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.998+0000 7fb6c9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6cc06f8c0 0x7fb6cc157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:20.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:20.998+0000 7fb6c9400640 1 -- 172.21.15.6:0/2727250263 learned_addr learned my addr 172.21.15.6:0/2727250263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.002+0000 7fb6c9400640 1 -- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc15bc80 msgr2=0x7fb6cc14dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.002+0000 7fb6c9400640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc15bc80 0x7fb6cc14dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.002+0000 7fb6c9400640 1 -- 172.21.15.6:0/2727250263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6cc06f8c0 msgr2=0x7fb6cc157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.002+0000 7fb6c9400640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6cc06f8c0 0x7fb6cc157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.002+0000 7fb6c9400640 1 -- 172.21.15.6:0/2727250263 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc156560 con 0x7fb6cc152400 2024-03-22T13:08:21.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.005+0000 7fb6c9400640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc152400 0x7fb6cc0cab20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80098c0 tx=0x7fb6b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:21.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.009+0000 7fb6c2000640 1 -- 172.21.15.6:0/2727250263 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8034040 con 0x7fb6cc152400 2024-03-22T13:08:21.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.009+0000 7fb6c2000640 1 -- 172.21.15.6:0/2727250263 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b803aaa0 con 0x7fb6cc152400 2024-03-22T13:08:21.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.009+0000 7fb6c2000640 1 -- 172.21.15.6:0/2727250263 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80443f0 con 0x7fb6cc152400 2024-03-22T13:08:21.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.009+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc14cf70 con 0x7fb6cc152400 2024-03-22T13:08:21.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.010+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc0c4790 con 0x7fb6cc152400 2024-03-22T13:08:21.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.011+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc0756c0 con 0x7fb6cc152400 2024-03-22T13:08:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.038+0000 7fb6c2000640 1 -- 172.21.15.6:0/2727250263 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb6b8037070 con 0x7fb6cc152400 2024-03-22T13:08:21.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.039+0000 7fb6c2000640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb6ac06bc20 0x7fb6ac06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:21.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.039+0000 7fb6c9e00640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb6ac06bc20 0x7fb6ac06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:21.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.040+0000 7fb6c9e00640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb6ac06bc20 0x7fb6ac06e0e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009b40 tx=0x7fb6bc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:21.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.287+0000 7fb6c2000640 1 -- 172.21.15.6:0/2727250263 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb6b8079370 con 0x7fb6cc152400 2024-03-22T13:08:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.307+0000 7fb6c2000640 1 -- 172.21.15.6:0/2727250263 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb6b803f070 con 0x7fb6cc152400 2024-03-22T13:08:21.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.561+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6cc0c6400 con 0x7fb6cc152400 2024-03-22T13:08:21.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.678+0000 7fb6c2000640 1 -- 172.21.15.6:0/2727250263 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fb6b807b2d0 con 0x7fb6cc152400 2024-03-22T13:08:21.678 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:21.678 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:21.679 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:21.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb6ac06bc20 msgr2=0x7fb6ac06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:21.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb6ac06bc20 0x7fb6ac06e0e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009b40 tx=0x7fb6bc009190 comp rx=0 tx=0).stop 2024-03-22T13:08:21.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc152400 msgr2=0x7fb6cc0cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc152400 0x7fb6cc0cab20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80098c0 tx=0x7fb6b80046c0 comp rx=0 tx=0).stop 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 shutdown_connections 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb6ac06bc20 0x7fb6ac06e0e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc15bc80 0x7fb6cc14dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc152400 0x7fb6cc0cab20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 --2- 172.21.15.6:0/2727250263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6cc06f8c0 0x7fb6cc157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.681+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 >> 172.21.15.6:0/2727250263 conn(0x7fb6cc0cb930 msgr2=0x7fb6cc1512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.682+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 shutdown_connections 2024-03-22T13:08:21.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:21.682+0000 7fb6d1000640 1 -- 172.21.15.6:0/2727250263 wait complete. 2024-03-22T13:08:22.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:22.305 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:23.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:23.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.508+0000 7f1b36200640 1 Processor -- start 2024-03-22T13:08:23.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.508+0000 7f1b36200640 1 -- start start 2024-03-22T13:08:23.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.509+0000 7f1b36200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1b300c9ab0 0x7f1b300cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:23.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.509+0000 7f1b36200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b30154980 con 0x7f1b300cc100 2024-03-22T13:08:23.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.509+0000 7f1b36200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3015afc0 con 0x7f1b3006d410 2024-03-22T13:08:23.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.509+0000 7f1b36200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b300ca360 con 0x7f1b300c9ab0 2024-03-22T13:08:23.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.510+0000 7f1b34e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1b300cc100 0x7f1b3006cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41754/0 (socket says 172.21.15.6:41754) 2024-03-22T13:08:23.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.510+0000 7f1b34e00640 1 -- 172.21.15.6:0/1033481779 learned_addr learned my addr 172.21.15.6:0/1033481779 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:23.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.511+0000 7f1b35800640 1 --2- 172.21.15.6:0/1033481779 >> v2:172.21.15.92:3300/0 conn(0x7f1b300c9ab0 0x7f1b300cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.514+0000 7f1b2fe00640 1 -- 172.21.15.6:0/1033481779 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386663808 0 0) 0x7f1b30154980 con 0x7f1b300cc100 2024-03-22T13:08:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.514+0000 7f1b2fe00640 1 -- 172.21.15.6:0/1033481779 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b14003520 con 0x7f1b300cc100 2024-03-22T13:08:23.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.515+0000 7f1b2fe00640 1 -- 172.21.15.6:0/1033481779 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315164136 0 0) 0x7f1b3015afc0 con 0x7f1b3006d410 2024-03-22T13:08:23.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.515+0000 7f1b2fe00640 1 -- 172.21.15.6:0/1033481779 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30154980 con 0x7f1b3006d410 2024-03-22T13:08:23.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.517+0000 7f1b35800640 1 -- 172.21.15.6:0/1033481779 >> v1:172.21.15.92:6789/0 conn(0x7f1b3006d410 legacy=0x7f1b30162a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:23.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.517+0000 7f1b35800640 1 -- 172.21.15.6:0/1033481779 >> v1:172.21.15.6:6789/0 conn(0x7f1b300cc100 legacy=0x7f1b3006cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:23.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.517+0000 7f1b35800640 1 -- 172.21.15.6:0/1033481779 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b300ca5b0 con 0x7f1b300c9ab0 2024-03-22T13:08:23.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.521+0000 7f1b35800640 1 --2- 172.21.15.6:0/1033481779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300c9ab0 0x7f1b300cc4e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009d90 tx=0x7f1b2802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e4e0569ef5391d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:23.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.552+0000 7f1b2fe00640 1 -- 172.21.15.6:0/1033481779 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28039070 con 0x7f1b300c9ab0 2024-03-22T13:08:23.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.552+0000 7f1b2fe00640 1 -- 172.21.15.6:0/1033481779 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2802cd00 con 0x7f1b300c9ab0 2024-03-22T13:08:23.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.552+0000 7f1b2fe00640 1 -- 172.21.15.6:0/1033481779 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28034660 con 0x7f1b300c9ab0 2024-03-22T13:08:23.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.553+0000 7f1b36200640 1 -- 172.21.15.6:0/1033481779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300c9ab0 msgr2=0x7f1b300cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:23.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.553+0000 7f1b36200640 1 --2- 172.21.15.6:0/1033481779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300c9ab0 0x7f1b300cc4e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009d90 tx=0x7f1b2802bf50 comp rx=0 tx=0).stop 2024-03-22T13:08:23.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.553+0000 7f1b36200640 1 -- 172.21.15.6:0/1033481779 shutdown_connections 2024-03-22T13:08:23.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.553+0000 7f1b36200640 1 --2- 172.21.15.6:0/1033481779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300c9ab0 0x7f1b300cc4e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:23.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.553+0000 7f1b36200640 1 -- 172.21.15.6:0/1033481779 >> 172.21.15.6:0/1033481779 conn(0x7f1b300b14f0 msgr2=0x7f1b300b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:23.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.553+0000 7f1b36200640 1 -- 172.21.15.6:0/1033481779 shutdown_connections 2024-03-22T13:08:23.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.553+0000 7f1b36200640 1 -- 172.21.15.6:0/1033481779 wait complete. 2024-03-22T13:08:23.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.556+0000 7f1b36200640 1 Processor -- start 2024-03-22T13:08:23.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.556+0000 7f1b36200640 1 -- start start 2024-03-22T13:08:23.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.557+0000 7f1b36200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b3006d410 0x7f1b3015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:23.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.557+0000 7f1b36200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b300c9ab0 0x7f1b301e5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:23.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.557+0000 7f1b36200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300cc100 0x7f1b30154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:23.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.557+0000 7f1b36200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3015ac60 con 0x7f1b3006d410 2024-03-22T13:08:23.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.557+0000 7f1b36200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3006c7e0 con 0x7f1b300cc100 2024-03-22T13:08:23.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.557+0000 7f1b36200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b300ccd50 con 0x7f1b300c9ab0 2024-03-22T13:08:23.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.558+0000 7f1b35800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300cc100 0x7f1b30154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:23.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.558+0000 7f1b35800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300cc100 0x7f1b30154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46324/0 (socket says 172.21.15.6:46324) 2024-03-22T13:08:23.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.558+0000 7f1b35800640 1 -- 172.21.15.6:0/199244577 learned_addr learned my addr 172.21.15.6:0/199244577 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:23.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.558+0000 7f1b27e00640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b300c9ab0 0x7f1b301e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:23.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.563+0000 7f1b35800640 1 -- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b300c9ab0 msgr2=0x7f1b301e5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:23.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.563+0000 7f1b35800640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b300c9ab0 0x7f1b301e5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:23.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.563+0000 7f1b35800640 1 -- 172.21.15.6:0/199244577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b3006d410 msgr2=0x7f1b3015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:23.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.563+0000 7f1b35800640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b3006d410 0x7f1b3015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:23.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.563+0000 7f1b35800640 1 -- 172.21.15.6:0/199244577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b301f1120 con 0x7f1b300cc100 2024-03-22T13:08:23.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.566+0000 7f1b35800640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300cc100 0x7f1b30154b50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009eb0 tx=0x7f1b28004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.571+0000 7f1b2d600640 1 -- 172.21.15.6:0/199244577 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28044020 con 0x7f1b300cc100 2024-03-22T13:08:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.571+0000 7f1b2d600640 1 -- 172.21.15.6:0/199244577 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b28035e50 con 0x7f1b300cc100 2024-03-22T13:08:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.571+0000 7f1b2d600640 1 -- 172.21.15.6:0/199244577 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28039040 con 0x7f1b300cc100 2024-03-22T13:08:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.571+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b301598a0 con 0x7f1b300cc100 2024-03-22T13:08:23.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.571+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b301f0360 con 0x7f1b300cc100 2024-03-22T13:08:23.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.572+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b30075650 con 0x7f1b300cc100 2024-03-22T13:08:23.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.862+0000 7f1b2d600640 1 -- 172.21.15.6:0/199244577 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b28047020 con 0x7f1b300cc100 2024-03-22T13:08:23.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.864+0000 7f1b2d600640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b0c06bc20 0x7f1b0c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:23.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.865+0000 7f1b34e00640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b0c06bc20 0x7f1b0c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:23.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.866+0000 7f1b34e00640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b0c06bc20 0x7f1b0c06e0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1b20000f60 tx=0x7f1b200078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:23.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.867+0000 7f1b2d600640 1 -- 172.21.15.6:0/199244577 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1b28079c40 con 0x7f1b300cc100 2024-03-22T13:08:23.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:23.888+0000 7f1b2d600640 1 -- 172.21.15.6:0/199244577 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b2804e030 con 0x7f1b300cc100 2024-03-22T13:08:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.150+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b300cada0 con 0x7f1b300cc100 2024-03-22T13:08:24.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.170+0000 7f1b2d600640 1 -- 172.21.15.6:0/199244577 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f1b2807c2a0 con 0x7f1b300cc100 2024-03-22T13:08:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:24.170 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:24.171 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b0c06bc20 msgr2=0x7f1b0c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b0c06bc20 0x7f1b0c06e0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1b20000f60 tx=0x7f1b200078f0 comp rx=0 tx=0).stop 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300cc100 msgr2=0x7f1b30154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300cc100 0x7f1b30154b50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009eb0 tx=0x7f1b28004a40 comp rx=0 tx=0).stop 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 shutdown_connections 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b0c06bc20 0x7f1b0c06e0e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b300cc100 0x7f1b30154b50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b300c9ab0 0x7f1b301e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 --2- 172.21.15.6:0/199244577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b3006d410 0x7f1b3015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:24.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 >> 172.21.15.6:0/199244577 conn(0x7f1b300b14f0 msgr2=0x7f1b30153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:24.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 shutdown_connections 2024-03-22T13:08:24.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:24.173+0000 7f1b36200640 1 -- 172.21.15.6:0/199244577 wait complete. 2024-03-22T13:08:24.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:24.794 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:25.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:25.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.994+0000 7f0652200640 1 Processor -- start 2024-03-22T13:08:25.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.995+0000 7f0652200640 1 -- start start 2024-03-22T13:08:25.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.996+0000 7f0652200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f064c152400 0x7f064c15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:25.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.996+0000 7f0652200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c15bc80 0x7f064c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:25.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.996+0000 7f0652200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f064c06f8c0 0x7f064c071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:25.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.996+0000 7f0652200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c0724d0 con 0x7f064c06f8c0 2024-03-22T13:08:25.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.996+0000 7f0652200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064c1590b0 con 0x7f064c152400 2024-03-22T13:08:25.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.996+0000 7f0652200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c0721f0 con 0x7f064c15bc80 2024-03-22T13:08:25.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.997+0000 7f064b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f064c152400 0x7f064c15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:25.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.997+0000 7f064be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f064c06f8c0 0x7f064c071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:25.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.997+0000 7f064aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c15bc80 0x7f064c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:25.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.997+0000 7f064b400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f064c152400 0x7f064c15b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46340/0 (socket says 172.21.15.6:46340) 2024-03-22T13:08:25.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:25.997+0000 7f064b400640 1 -- 172.21.15.6:0/3653212427 learned_addr learned my addr 172.21.15.6:0/3653212427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.000+0000 7f064be00640 1 -- 172.21.15.6:0/3653212427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c15bc80 msgr2=0x7f064c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.000+0000 7f064be00640 1 --2- 172.21.15.6:0/3653212427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c15bc80 0x7f064c0716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.000+0000 7f064be00640 1 -- 172.21.15.6:0/3653212427 >> v2:172.21.15.92:3300/0 conn(0x7f064c152400 msgr2=0x7f064c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.000+0000 7f064be00640 1 --2- 172.21.15.6:0/3653212427 >> v2:172.21.15.92:3300/0 conn(0x7f064c152400 0x7f064c15b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.000+0000 7f064be00640 1 -- 172.21.15.6:0/3653212427 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c1582f0 con 0x7f064c06f8c0 2024-03-22T13:08:26.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.003+0000 7f064be00640 1 --2- 172.21.15.6:0/3653212427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c06f8c0 0x7f064c071c10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f064c071390 tx=0x7f064402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9c1c951bc1d94d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:26.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.008+0000 7f064a000640 1 -- 172.21.15.6:0/3653212427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644035020 con 0x7f064c06f8c0 2024-03-22T13:08:26.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.008+0000 7f064a000640 1 -- 172.21.15.6:0/3653212427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644031830 con 0x7f064c06f8c0 2024-03-22T13:08:26.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.008+0000 7f064a000640 1 -- 172.21.15.6:0/3653212427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644031c00 con 0x7f064c06f8c0 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 -- 172.21.15.6:0/3653212427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c06f8c0 msgr2=0x7f064c071c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 --2- 172.21.15.6:0/3653212427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c06f8c0 0x7f064c071c10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f064c071390 tx=0x7f064402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 -- 172.21.15.6:0/3653212427 shutdown_connections 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 --2- 172.21.15.6:0/3653212427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c06f8c0 0x7f064c071c10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 --2- 172.21.15.6:0/3653212427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c15bc80 0x7f064c0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 --2- 172.21.15.6:0/3653212427 >> v2:172.21.15.92:3300/0 conn(0x7f064c152400 0x7f064c15b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 -- 172.21.15.6:0/3653212427 >> 172.21.15.6:0/3653212427 conn(0x7f064c0cb930 msgr2=0x7f064c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 -- 172.21.15.6:0/3653212427 shutdown_connections 2024-03-22T13:08:26.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.009+0000 7f0652200640 1 -- 172.21.15.6:0/3653212427 wait complete. 2024-03-22T13:08:26.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.011+0000 7f0652200640 1 Processor -- start 2024-03-22T13:08:26.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.011+0000 7f0652200640 1 -- start start 2024-03-22T13:08:26.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.012+0000 7f0652200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c06f8c0 0x7f064c0cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.012+0000 7f0652200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f064c152400 0x7f064c148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.012+0000 7f0652200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c15bc80 0x7f064c1487a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.012+0000 7f0652200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c158b90 con 0x7f064c15bc80 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.012+0000 7f0652200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1588b0 con 0x7f064c152400 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.012+0000 7f0652200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1527e0 con 0x7f064c06f8c0 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.013+0000 7f064be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c15bc80 0x7f064c1487a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.013+0000 7f064b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c06f8c0 0x7f064c0cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.013+0000 7f064aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f064c152400 0x7f064c148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.013+0000 7f064aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f064c152400 0x7f064c148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46346/0 (socket says 172.21.15.6:46346) 2024-03-22T13:08:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.013+0000 7f064aa00640 1 -- 172.21.15.6:0/3518814654 learned_addr learned my addr 172.21.15.6:0/3518814654 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:26.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.016+0000 7f064aa00640 1 -- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c06f8c0 msgr2=0x7f064c0cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:26.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.016+0000 7f064aa00640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c06f8c0 0x7f064c0cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.016+0000 7f064aa00640 1 -- 172.21.15.6:0/3518814654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c15bc80 msgr2=0x7f064c1487a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:26.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.016+0000 7f064aa00640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c15bc80 0x7f064c1487a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.016+0000 7f064aa00640 1 -- 172.21.15.6:0/3518814654 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c14cf80 con 0x7f064c152400 2024-03-22T13:08:26.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.018+0000 7f064aa00640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f064c152400 0x7f064c148260 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f063400e870 tx=0x7f063400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.023+0000 7f0643400640 1 -- 172.21.15.6:0/3518814654 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634017070 con 0x7f064c152400 2024-03-22T13:08:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.023+0000 7f0643400640 1 -- 172.21.15.6:0/3518814654 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063400c940 con 0x7f064c152400 2024-03-22T13:08:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.023+0000 7f0643400640 1 -- 172.21.15.6:0/3518814654 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063400cd10 con 0x7f064c152400 2024-03-22T13:08:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.023+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c14d200 con 0x7f064c152400 2024-03-22T13:08:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.023+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c1f0660 con 0x7f064c152400 2024-03-22T13:08:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.024+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c067ae0 con 0x7f064c152400 2024-03-22T13:08:26.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.306+0000 7f0643400640 1 -- 172.21.15.6:0/3518814654 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f063401a070 con 0x7f064c152400 2024-03-22T13:08:26.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.309+0000 7f0643400640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f062806bc20 0x7f062806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.309+0000 7f0643400640 1 -- 172.21.15.6:0/3518814654 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f063408b470 con 0x7f064c152400 2024-03-22T13:08:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.309+0000 7f064b400640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f062806bc20 0x7f062806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:26.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.310+0000 7f064b400640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f062806bc20 0x7f062806e0e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f063c011360 tx=0x7f063c00d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:26.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.329+0000 7f0643400640 1 -- 172.21.15.6:0/3518814654 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f064c067ae0 con 0x7f064c152400 2024-03-22T13:08:26.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.581+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f064c0c6400 con 0x7f064c152400 2024-03-22T13:08:26.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.596+0000 7f0643400640 1 -- 172.21.15.6:0/3518814654 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f064c14cf80 con 0x7f064c152400 2024-03-22T13:08:26.596 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:26.596 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:26.596 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.598+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f062806bc20 msgr2=0x7f062806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.598+0000 7f0652200640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f062806bc20 0x7f062806e0e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f063c011360 tx=0x7f063c00d040 comp rx=0 tx=0).stop 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.598+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f064c152400 msgr2=0x7f064c148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.598+0000 7f0652200640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f064c152400 0x7f064c148260 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f063400e870 tx=0x7f063400ed30 comp rx=0 tx=0).stop 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 shutdown_connections 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f064c15bc80 0x7f064c1487a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f062806bc20 0x7f062806e0e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f064c152400 0x7f064c148260 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 --2- 172.21.15.6:0/3518814654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f064c06f8c0 0x7f064c0cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 >> 172.21.15.6:0/3518814654 conn(0x7f064c0cb930 msgr2=0x7f064c071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 shutdown_connections 2024-03-22T13:08:26.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:26.599+0000 7f0652200640 1 -- 172.21.15.6:0/3518814654 wait complete. 2024-03-22T13:08:27.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:27.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:28.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:28.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.426+0000 7f4e7fc00640 1 Processor -- start 2024-03-22T13:08:28.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.427+0000 7f4e7fc00640 1 -- start start 2024-03-22T13:08:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.428+0000 7f4e7fc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e78152400 0x7f4e7815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.428+0000 7f4e7fc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4e7806f8c0 0x7f4e780c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.428+0000 7f4e7fc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e781590b0 con 0x7f4e7815bc80 2024-03-22T13:08:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.428+0000 7f4e7fc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e780724d0 con 0x7f4e7806f8c0 2024-03-22T13:08:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.428+0000 7f4e7fc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780721f0 con 0x7f4e78152400 2024-03-22T13:08:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.429+0000 7f4e7d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e78152400 0x7f4e7815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.429+0000 7f4e7d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e78152400 0x7f4e7815b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:42926/0 (socket says 172.21.15.6:42926) 2024-03-22T13:08:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.429+0000 7f4e7d200640 1 -- 172.21.15.6:0/897175268 learned_addr learned my addr 172.21.15.6:0/897175268 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.433+0000 7f4e7d200640 1 -- 172.21.15.6:0/897175268 >> v2:172.21.15.6:3300/0 conn(0x7f4e7806f8c0 msgr2=0x7f4e780c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.433+0000 7f4e7d200640 1 --2- 172.21.15.6:0/897175268 >> v2:172.21.15.6:3300/0 conn(0x7f4e7806f8c0 0x7f4e780c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.433+0000 7f4e7d200640 1 -- 172.21.15.6:0/897175268 >> v1:172.21.15.92:6789/0 conn(0x7f4e7815bc80 legacy=0x7f4e780716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.433+0000 7f4e7d200640 1 -- 172.21.15.6:0/897175268 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e781582f0 con 0x7f4e78152400 2024-03-22T13:08:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.436+0000 7f4e7d200640 1 --2- 172.21.15.6:0/897175268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e78152400 0x7f4e7815b740 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009870 tx=0x7f4e6c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beb5490de4cc18db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e77400640 1 -- 172.21.15.6:0/897175268 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c034070 con 0x7f4e78152400 2024-03-22T13:08:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e77400640 1 -- 172.21.15.6:0/897175268 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c030840 con 0x7f4e78152400 2024-03-22T13:08:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e77400640 1 -- 172.21.15.6:0/897175268 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c030c10 con 0x7f4e78152400 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e7fc00640 1 -- 172.21.15.6:0/897175268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e78152400 msgr2=0x7f4e7815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/897175268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e78152400 0x7f4e7815b740 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009870 tx=0x7f4e6c02b980 comp rx=0 tx=0).stop 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e7fc00640 1 -- 172.21.15.6:0/897175268 shutdown_connections 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/897175268 >> v2:172.21.15.6:3300/0 conn(0x7f4e7806f8c0 0x7f4e780c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/897175268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e78152400 0x7f4e7815b740 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.441+0000 7f4e7fc00640 1 -- 172.21.15.6:0/897175268 >> 172.21.15.6:0/897175268 conn(0x7f4e780cb930 msgr2=0x7f4e780cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.442+0000 7f4e7fc00640 1 -- 172.21.15.6:0/897175268 shutdown_connections 2024-03-22T13:08:28.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.442+0000 7f4e7fc00640 1 -- 172.21.15.6:0/897175268 wait complete. 2024-03-22T13:08:28.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.443+0000 7f4e7fc00640 1 Processor -- start 2024-03-22T13:08:28.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.443+0000 7f4e7fc00640 1 -- start start 2024-03-22T13:08:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.444+0000 7f4e7fc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e7806f8c0 0x7f4e780c1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.444+0000 7f4e7fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e78152400 0x7f4e7806b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.444+0000 7f4e7fc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7815bc80 0x7f4e7806bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:28.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.444+0000 7f4e7fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78158b90 con 0x7f4e78152400 2024-03-22T13:08:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.444+0000 7f4e7fc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781588b0 con 0x7f4e7806f8c0 2024-03-22T13:08:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.444+0000 7f4e7fc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781527e0 con 0x7f4e7815bc80 2024-03-22T13:08:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.445+0000 7f4e7dc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7815bc80 0x7f4e7806bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.446+0000 7f4e7dc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7815bc80 0x7f4e7806bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:42932/0 (socket says 172.21.15.6:42932) 2024-03-22T13:08:28.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.446+0000 7f4e7dc00640 1 -- 172.21.15.6:0/3887544823 learned_addr learned my addr 172.21.15.6:0/3887544823 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.451+0000 7f4e7dc00640 1 -- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e7806f8c0 msgr2=0x7f4e780c1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.451+0000 7f4e7dc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e7806f8c0 0x7f4e780c1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:28.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.451+0000 7f4e7dc00640 1 -- 172.21.15.6:0/3887544823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e78152400 msgr2=0x7f4e7806b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:28.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.451+0000 7f4e7dc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e78152400 0x7f4e7806b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:28.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.451+0000 7f4e7dc00640 1 -- 172.21.15.6:0/3887544823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78073260 con 0x7f4e7815bc80 2024-03-22T13:08:28.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.454+0000 7f4e7dc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7815bc80 0x7f4e7806bad0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000b970 tx=0x7f4e7000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.459+0000 7f4e74c00640 1 -- 172.21.15.6:0/3887544823 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70019070 con 0x7f4e7815bc80 2024-03-22T13:08:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.459+0000 7f4e74c00640 1 -- 172.21.15.6:0/3887544823 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7000cb60 con 0x7f4e7815bc80 2024-03-22T13:08:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.459+0000 7f4e74c00640 1 -- 172.21.15.6:0/3887544823 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70014440 con 0x7f4e7815bc80 2024-03-22T13:08:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.459+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e780c2e30 con 0x7f4e7815bc80 2024-03-22T13:08:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.459+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e781f02a0 con 0x7f4e7815bc80 2024-03-22T13:08:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.460+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e780756c0 con 0x7f4e7815bc80 2024-03-22T13:08:28.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.736+0000 7f4e74c00640 1 -- 172.21.15.6:0/3887544823 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e7001c070 con 0x7f4e7815bc80 2024-03-22T13:08:28.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.737+0000 7f4e74c00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4e5806bc50 0x7f4e5806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:28.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.738+0000 7f4e7d200640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4e5806bc50 0x7f4e5806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:28.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.738+0000 7f4e7d200640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4e5806bc50 0x7f4e5806e110 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c036000 tx=0x7f4e6c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:28.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.740+0000 7f4e74c00640 1 -- 172.21.15.6:0/3887544823 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4e70057170 con 0x7f4e7815bc80 2024-03-22T13:08:28.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:28.759+0000 7f4e74c00640 1 -- 172.21.15.6:0/3887544823 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e70023020 con 0x7f4e7815bc80 2024-03-22T13:08:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.023+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e78151c20 con 0x7f4e7815bc80 2024-03-22T13:08:29.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.046+0000 7f4e74c00640 1 -- 172.21.15.6:0/3887544823 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f4e78073260 con 0x7f4e7815bc80 2024-03-22T13:08:29.046 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:29.046 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:06:22.868708+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":34,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:06:22.868703+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:08:29.046 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.049+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4e5806bc50 msgr2=0x7f4e5806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.049+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4e5806bc50 0x7f4e5806e110 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c036000 tx=0x7f4e6c03b040 comp rx=0 tx=0).stop 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.049+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7815bc80 msgr2=0x7f4e7806bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.049+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7815bc80 0x7f4e7806bad0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000b970 tx=0x7f4e7000be30 comp rx=0 tx=0).stop 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 shutdown_connections 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7815bc80 0x7f4e7806bad0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e78152400 0x7f4e7806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4e5806bc50 0x7f4e5806e110 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 --2- 172.21.15.6:0/3887544823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e7806f8c0 0x7f4e780c1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 >> 172.21.15.6:0/3887544823 conn(0x7f4e780cb930 msgr2=0x7f4e78147ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 shutdown_connections 2024-03-22T13:08:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:29.050+0000 7f4e7fc00640 1 -- 172.21.15.6:0/3887544823 wait complete. 2024-03-22T13:08:29.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:06:22.868708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 34, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:06:22.868703+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:08:29.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:30.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:30.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.867+0000 7f5949200640 1 Processor -- start 2024-03-22T13:08:30.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.867+0000 7f5949200640 1 -- start start 2024-03-22T13:08:30.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.869+0000 7f5949200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59440c5210 0x7f5944158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:30.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.869+0000 7f5949200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594406e0f0 con 0x7f594406ede0 2024-03-22T13:08:30.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.869+0000 7f5949200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594406d350 con 0x7f59440c55f0 2024-03-22T13:08:30.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.869+0000 7f5949200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59440c6f90 con 0x7f59440c5210 2024-03-22T13:08:30.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.870+0000 7f5942a00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f59440c55f0 0x7f59440c6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:48306/0 (socket says 172.21.15.6:48306) 2024-03-22T13:08:30.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.870+0000 7f5942a00640 1 -- 172.21.15.6:0/940924146 learned_addr learned my addr 172.21.15.6:0/940924146 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:30.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.870+0000 7f5943400640 1 --2- 172.21.15.6:0/940924146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59440c5210 0x7f5944158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:30.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.872+0000 7f5942000640 1 -- 172.21.15.6:0/940924146 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939972906 0 0) 0x7f594406d350 con 0x7f59440c55f0 2024-03-22T13:08:30.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.872+0000 7f5942000640 1 -- 172.21.15.6:0/940924146 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928003520 con 0x7f59440c55f0 2024-03-22T13:08:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.873+0000 7f5943400640 1 -- 172.21.15.6:0/940924146 >> v1:172.21.15.92:6789/0 conn(0x7f59440c55f0 legacy=0x7f59440c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.874+0000 7f5943400640 1 -- 172.21.15.6:0/940924146 >> v1:172.21.15.6:6789/0 conn(0x7f594406ede0 legacy=0x7f594414e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:30.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.874+0000 7f5943400640 1 -- 172.21.15.6:0/940924146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59440c6cb0 con 0x7f59440c5210 2024-03-22T13:08:30.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.877+0000 7f5943400640 1 --2- 172.21.15.6:0/940924146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59440c5210 0x7f5944158a60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f592c0097d0 tx=0x7f592c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=518b48c9588380fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:30.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.882+0000 7f5942000640 1 -- 172.21.15.6:0/940924146 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c035020 con 0x7f59440c5210 2024-03-22T13:08:30.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.882+0000 7f5942000640 1 -- 172.21.15.6:0/940924146 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c031800 con 0x7f59440c5210 2024-03-22T13:08:30.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.882+0000 7f5942000640 1 -- 172.21.15.6:0/940924146 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c031bd0 con 0x7f59440c5210 2024-03-22T13:08:30.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.883+0000 7f5949200640 1 -- 172.21.15.6:0/940924146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59440c5210 msgr2=0x7f5944158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.883+0000 7f5949200640 1 --2- 172.21.15.6:0/940924146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59440c5210 0x7f5944158a60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f592c0097d0 tx=0x7f592c02b8a0 comp rx=0 tx=0).stop 2024-03-22T13:08:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.883+0000 7f5949200640 1 -- 172.21.15.6:0/940924146 shutdown_connections 2024-03-22T13:08:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.883+0000 7f5949200640 1 --2- 172.21.15.6:0/940924146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59440c5210 0x7f5944158a60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.883+0000 7f5949200640 1 -- 172.21.15.6:0/940924146 >> 172.21.15.6:0/940924146 conn(0x7f59440cb890 msgr2=0x7f59440cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.883+0000 7f5949200640 1 -- 172.21.15.6:0/940924146 shutdown_connections 2024-03-22T13:08:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.883+0000 7f5949200640 1 -- 172.21.15.6:0/940924146 wait complete. 2024-03-22T13:08:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.886+0000 7f5949200640 1 Processor -- start 2024-03-22T13:08:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.886+0000 7f5949200640 1 -- start start 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.887+0000 7f5949200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f594406ede0 0x7f5944154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.887+0000 7f5949200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59440c5210 0x7f59441581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.887+0000 7f5949200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59440c55f0 0x7f5944143c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.887+0000 7f5949200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594406dbd0 con 0x7f59440c5210 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.887+0000 7f5949200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594406d8f0 con 0x7f59440c55f0 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.887+0000 7f5949200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59440c69a0 con 0x7f594406ede0 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.888+0000 7f5943400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f594406ede0 0x7f5944154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.888+0000 7f5943e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59440c55f0 0x7f5944143c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.888+0000 7f5942a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59440c5210 0x7f59441581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:30.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.888+0000 7f5943e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59440c55f0 0x7f5944143c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:46368/0 (socket says 172.21.15.6:46368) 2024-03-22T13:08:30.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.888+0000 7f5943e00640 1 -- 172.21.15.6:0/2646989462 learned_addr learned my addr 172.21.15.6:0/2646989462 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.892+0000 7f5943e00640 1 -- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f594406ede0 msgr2=0x7f5944154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.892+0000 7f5943e00640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f594406ede0 0x7f5944154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.892+0000 7f5943e00640 1 -- 172.21.15.6:0/2646989462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59440c5210 msgr2=0x7f59441581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.892+0000 7f5943e00640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59440c5210 0x7f59441581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.892+0000 7f5943e00640 1 -- 172.21.15.6:0/2646989462 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59441552e0 con 0x7f59440c55f0 2024-03-22T13:08:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.895+0000 7f5943e00640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59440c55f0 0x7f5944143c00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f593c00c780 tx=0x7f593c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.899+0000 7f593b400640 1 -- 172.21.15.6:0/2646989462 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c017070 con 0x7f59440c55f0 2024-03-22T13:08:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.899+0000 7f593b400640 1 -- 172.21.15.6:0/2646989462 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c003da0 con 0x7f59440c55f0 2024-03-22T13:08:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.899+0000 7f593b400640 1 -- 172.21.15.6:0/2646989462 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c0052b0 con 0x7f59440c55f0 2024-03-22T13:08:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.899+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59441555a0 con 0x7f59440c55f0 2024-03-22T13:08:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.899+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5944153b50 con 0x7f59440c55f0 2024-03-22T13:08:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.901+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5944075650 con 0x7f59440c55f0 2024-03-22T13:08:30.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.920+0000 7f593b400640 1 -- 172.21.15.6:0/2646989462 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f593c01a070 con 0x7f59440c55f0 2024-03-22T13:08:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.921+0000 7f593b400640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f592006bc20 0x7f592006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.922+0000 7f5943400640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f592006bc20 0x7f592006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.922+0000 7f5943400640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f592006bc20 0x7f592006e0e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f592c036000 tx=0x7f592c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:30.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:30.925+0000 7f593b400640 1 -- 172.21.15.6:0/2646989462 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f593c0569c0 con 0x7f59440c55f0 2024-03-22T13:08:31.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.176+0000 7f593b400640 1 -- 172.21.15.6:0/2646989462 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f593c021020 con 0x7f59440c55f0 2024-03-22T13:08:31.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.442+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f594414c6b0 con 0x7f59440c55f0 2024-03-22T13:08:31.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.459+0000 7f593b400640 1 -- 172.21.15.6:0/2646989462 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f593c058920 con 0x7f59440c55f0 2024-03-22T13:08:31.459 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:31.459 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:31.459 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f592006bc20 msgr2=0x7f592006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f592006bc20 0x7f592006e0e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f592c036000 tx=0x7f592c03c040 comp rx=0 tx=0).stop 2024-03-22T13:08:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59440c55f0 msgr2=0x7f5944143c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59440c55f0 0x7f5944143c00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f593c00c780 tx=0x7f593c00cc40 comp rx=0 tx=0).stop 2024-03-22T13:08:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 shutdown_connections 2024-03-22T13:08:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f592006bc20 0x7f592006e0e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:31.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59440c55f0 0x7f5944143c00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:31.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59440c5210 0x7f59441581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:31.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 --2- 172.21.15.6:0/2646989462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f594406ede0 0x7f5944154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:31.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 >> 172.21.15.6:0/2646989462 conn(0x7f59440cb890 msgr2=0x7f594414a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:31.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 shutdown_connections 2024-03-22T13:08:31.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:31.463+0000 7f5949200640 1 -- 172.21.15.6:0/2646989462 wait complete. 2024-03-22T13:08:32.092 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:32.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:32.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:08:32.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:08:33.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.283+0000 7f4ad8c00640 1 Processor -- start 2024-03-22T13:08:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.283+0000 7f4ad8c00640 1 -- start start 2024-03-22T13:08:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.285+0000 7f4ad8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad406b080 0x7f4ad406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.285+0000 7f4ad8c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4ad40c2ea0 0x7f4ad40c3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.285+0000 7f4ad8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40c3d30 con 0x7f4ad406fea0 2024-03-22T13:08:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.285+0000 7f4ad8c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40c3a50 con 0x7f4ad40c2ea0 2024-03-22T13:08:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.285+0000 7f4ad8c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415a450 con 0x7f4ad406b080 2024-03-22T13:08:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.286+0000 7f4ad2800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4ad40c2ea0 0x7f4ad40c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.286+0000 7f4ad1400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4ad406fea0 0x7f4ad415d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52978/0 (socket says 172.21.15.6:52978) 2024-03-22T13:08:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.286+0000 7f4ad1400640 1 -- 172.21.15.6:0/2100152898 learned_addr learned my addr 172.21.15.6:0/2100152898 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.286+0000 7f4ad1e00640 1 --2- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad406b080 0x7f4ad406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.288+0000 7f4ad2800640 1 -- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad406b080 msgr2=0x7f4ad406b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.288+0000 7f4ad2800640 1 --2- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad406b080 0x7f4ad406b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.288+0000 7f4ad2800640 1 -- 172.21.15.6:0/2100152898 >> v1:172.21.15.6:6789/0 conn(0x7f4ad406fea0 legacy=0x7f4ad415d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.288+0000 7f4ad2800640 1 -- 172.21.15.6:0/2100152898 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad415a730 con 0x7f4ad40c2ea0 2024-03-22T13:08:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.291+0000 7f4ad2800640 1 --2- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad40c2ea0 0x7f4ad40c3300 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009d90 tx=0x7f4acc02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1024632fc9468e16 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.295+0000 7f4ad0a00640 1 -- 172.21.15.6:0/2100152898 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc039070 con 0x7f4ad40c2ea0 2024-03-22T13:08:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.295+0000 7f4ad0a00640 1 -- 172.21.15.6:0/2100152898 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc02cce0 con 0x7f4ad40c2ea0 2024-03-22T13:08:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.295+0000 7f4ad0a00640 1 -- 172.21.15.6:0/2100152898 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc034660 con 0x7f4ad40c2ea0 2024-03-22T13:08:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.295+0000 7f4ad8c00640 1 -- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad40c2ea0 msgr2=0x7f4ad40c3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.295+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad40c2ea0 0x7f4ad40c3300 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009d90 tx=0x7f4acc02bf30 comp rx=0 tx=0).stop 2024-03-22T13:08:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.296+0000 7f4ad8c00640 1 -- 172.21.15.6:0/2100152898 shutdown_connections 2024-03-22T13:08:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.296+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad40c2ea0 0x7f4ad40c3300 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.296+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/2100152898 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad406b080 0x7f4ad406b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.296+0000 7f4ad8c00640 1 -- 172.21.15.6:0/2100152898 >> 172.21.15.6:0/2100152898 conn(0x7f4ad40ccb50 msgr2=0x7f4ad40ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.296+0000 7f4ad8c00640 1 -- 172.21.15.6:0/2100152898 shutdown_connections 2024-03-22T13:08:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.296+0000 7f4ad8c00640 1 -- 172.21.15.6:0/2100152898 wait complete. 2024-03-22T13:08:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.297+0000 7f4ad8c00640 1 Processor -- start 2024-03-22T13:08:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.298+0000 7f4ad8c00640 1 -- start start 2024-03-22T13:08:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.298+0000 7f4ad8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad406b080 0x7f4ad4154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.298+0000 7f4ad8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad406fea0 0x7f4ad41f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.298+0000 7f4ad8c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad40c2ea0 0x7f4ad40c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.298+0000 7f4ad8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415e000 con 0x7f4ad406b080 2024-03-22T13:08:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.298+0000 7f4ad8c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415acf0 con 0x7f4ad406fea0 2024-03-22T13:08:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.298+0000 7f4ad8c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415b210 con 0x7f4ad40c2ea0 2024-03-22T13:08:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.299+0000 7f4ad1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad406fea0 0x7f4ad41f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.299+0000 7f4ad1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad406b080 0x7f4ad4154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.299+0000 7f4ad2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad40c2ea0 0x7f4ad40c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.299+0000 7f4ad1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad406fea0 0x7f4ad41f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:42214/0 (socket says 172.21.15.6:42214) 2024-03-22T13:08:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.299+0000 7f4ad1400640 1 -- 172.21.15.6:0/1328198924 learned_addr learned my addr 172.21.15.6:0/1328198924 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.303+0000 7f4ad1400640 1 -- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad40c2ea0 msgr2=0x7f4ad40c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.303+0000 7f4ad1400640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad40c2ea0 0x7f4ad40c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.303+0000 7f4ad1400640 1 -- 172.21.15.6:0/1328198924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad406b080 msgr2=0x7f4ad4154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.303+0000 7f4ad1400640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad406b080 0x7f4ad4154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.303+0000 7f4ad1400640 1 -- 172.21.15.6:0/1328198924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40c4cd0 con 0x7f4ad406fea0 2024-03-22T13:08:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.306+0000 7f4ad1400640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad406fea0 0x7f4ad41f1440 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c430 tx=0x7f4ac000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.311+0000 7f4aca000640 1 -- 172.21.15.6:0/1328198924 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0017070 con 0x7f4ad406fea0 2024-03-22T13:08:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.311+0000 7f4aca000640 1 -- 172.21.15.6:0/1328198924 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac0005ce0 con 0x7f4ad406fea0 2024-03-22T13:08:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.311+0000 7f4aca000640 1 -- 172.21.15.6:0/1328198924 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00072f0 con 0x7f4ad406fea0 2024-03-22T13:08:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.311+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad41f0a50 con 0x7f4ad406fea0 2024-03-22T13:08:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.311+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad415c8e0 con 0x7f4ad406fea0 2024-03-22T13:08:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.313+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad40756c0 con 0x7f4ad406fea0 2024-03-22T13:08:33.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.595+0000 7f4aca000640 1 -- 172.21.15.6:0/1328198924 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4ac001a070 con 0x7f4ad406fea0 2024-03-22T13:08:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.598+0000 7f4aca000640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4ab406bc50 0x7f4ab406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.598+0000 7f4aca000640 1 -- 172.21.15.6:0/1328198924 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4ac008a5a0 con 0x7f4ad406fea0 2024-03-22T13:08:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.598+0000 7f4ad1e00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4ab406bc50 0x7f4ab406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.599+0000 7f4ad1e00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4ab406bc50 0x7f4ab406e110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009b40 tx=0x7f4ac4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:33.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.609+0000 7f4aca000640 1 -- 172.21.15.6:0/1328198924 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4ac0021020 con 0x7f4ad406fea0 2024-03-22T13:08:33.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.874+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ad40c3790 con 0x7f4ad406fea0 2024-03-22T13:08:33.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.961+0000 7f4aca000640 1 -- 172.21.15.6:0/1328198924 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f4ac0058720 con 0x7f4ad406fea0 2024-03-22T13:08:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:33.962 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:33.962 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4ab406bc50 msgr2=0x7f4ab406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4ab406bc50 0x7f4ab406e110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009b40 tx=0x7f4ac4009190 comp rx=0 tx=0).stop 2024-03-22T13:08:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad406fea0 msgr2=0x7f4ad41f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad406fea0 0x7f4ad41f1440 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c430 tx=0x7f4ac000c8f0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 shutdown_connections 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ad40c2ea0 0x7f4ad40c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4ab406bc50 0x7f4ab406e110 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ad406fea0 0x7f4ad41f1440 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 --2- 172.21.15.6:0/1328198924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ad406b080 0x7f4ad4154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 >> 172.21.15.6:0/1328198924 conn(0x7f4ad40ccb50 msgr2=0x7f4ad40cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 shutdown_connections 2024-03-22T13:08:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:33.964+0000 7f4ad8c00640 1 -- 172.21.15.6:0/1328198924 wait complete. 2024-03-22T13:08:34.587 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:34.587 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:35.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.784+0000 7fcb5ee00640 1 Processor -- start 2024-03-22T13:08:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.784+0000 7fcb5ee00640 1 -- start start 2024-03-22T13:08:35.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.785+0000 7fcb5ee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb58152400 0x7fcb5815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:35.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.785+0000 7fcb5ee00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb5815bc80 0x7fcb580716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.785+0000 7fcb5ee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb5806f8c0 0x7fcb58071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.785+0000 7fcb5ee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb580724d0 con 0x7fcb5806f8c0 2024-03-22T13:08:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.785+0000 7fcb5ee00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb581590b0 con 0x7fcb5815bc80 2024-03-22T13:08:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.785+0000 7fcb5ee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580721f0 con 0x7fcb58152400 2024-03-22T13:08:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.786+0000 7fcb57400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb5815bc80 0x7fcb580716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.786+0000 7fcb57e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb58152400 0x7fcb5815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.787+0000 7fcb5ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb5806f8c0 0x7fcb58071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.787+0000 7fcb57400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb5815bc80 0x7fcb580716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:42224/0 (socket says 172.21.15.6:42224) 2024-03-22T13:08:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.787+0000 7fcb57400640 1 -- 172.21.15.6:0/387353814 learned_addr learned my addr 172.21.15.6:0/387353814 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.790+0000 7fcb57400640 1 -- 172.21.15.6:0/387353814 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb58152400 msgr2=0x7fcb5815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.790+0000 7fcb57400640 1 --2- 172.21.15.6:0/387353814 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb58152400 0x7fcb5815b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.790+0000 7fcb57400640 1 -- 172.21.15.6:0/387353814 >> v2:172.21.15.6:3300/0 conn(0x7fcb5806f8c0 msgr2=0x7fcb58071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.790+0000 7fcb57400640 1 --2- 172.21.15.6:0/387353814 >> v2:172.21.15.6:3300/0 conn(0x7fcb5806f8c0 0x7fcb58071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.790+0000 7fcb57400640 1 -- 172.21.15.6:0/387353814 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb581582f0 con 0x7fcb5815bc80 2024-03-22T13:08:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.790+0000 7fcb5ce00640 1 --2- 172.21.15.6:0/387353814 >> v2:172.21.15.6:3300/0 conn(0x7fcb5806f8c0 0x7fcb58071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:08:35.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.793+0000 7fcb57400640 1 --2- 172.21.15.6:0/387353814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb5815bc80 0x7fcb580716d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcb580713c0 tx=0x7fcb4402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93ae9f6b0c1038f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.799+0000 7fcb56a00640 1 -- 172.21.15.6:0/387353814 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44034070 con 0x7fcb5815bc80 2024-03-22T13:08:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.799+0000 7fcb56a00640 1 -- 172.21.15.6:0/387353814 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb44030860 con 0x7fcb5815bc80 2024-03-22T13:08:35.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.799+0000 7fcb56a00640 1 -- 172.21.15.6:0/387353814 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44030c30 con 0x7fcb5815bc80 2024-03-22T13:08:35.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.799+0000 7fcb5ee00640 1 -- 172.21.15.6:0/387353814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb5815bc80 msgr2=0x7fcb580716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:35.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.799+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/387353814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb5815bc80 0x7fcb580716d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcb580713c0 tx=0x7fcb4402b9a0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.800+0000 7fcb5ee00640 1 -- 172.21.15.6:0/387353814 shutdown_connections 2024-03-22T13:08:35.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.800+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/387353814 >> v2:172.21.15.6:3300/0 conn(0x7fcb5806f8c0 0x7fcb58071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.800+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/387353814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb5815bc80 0x7fcb580716d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.800+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/387353814 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb58152400 0x7fcb5815b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.800+0000 7fcb5ee00640 1 -- 172.21.15.6:0/387353814 >> 172.21.15.6:0/387353814 conn(0x7fcb580cb930 msgr2=0x7fcb580cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:35.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.800+0000 7fcb5ee00640 1 -- 172.21.15.6:0/387353814 shutdown_connections 2024-03-22T13:08:35.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.800+0000 7fcb5ee00640 1 -- 172.21.15.6:0/387353814 wait complete. 2024-03-22T13:08:35.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.802+0000 7fcb5ee00640 1 Processor -- start 2024-03-22T13:08:35.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.802+0000 7fcb5ee00640 1 -- start start 2024-03-22T13:08:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.803+0000 7fcb5ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb5806f8c0 0x7fcb58148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.803+0000 7fcb5ee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb58152400 0x7fcb581f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.803+0000 7fcb5ee00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb5815bc80 0x7fcb580ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.803+0000 7fcb5ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb58158b90 con 0x7fcb5806f8c0 2024-03-22T13:08:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.803+0000 7fcb5ee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb581588b0 con 0x7fcb58152400 2024-03-22T13:08:35.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.803+0000 7fcb5ee00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb581527e0 con 0x7fcb5815bc80 2024-03-22T13:08:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.804+0000 7fcb5ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb5815bc80 0x7fcb580ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.804+0000 7fcb57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb5806f8c0 0x7fcb58148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.804+0000 7fcb57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb58152400 0x7fcb581f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.804+0000 7fcb5ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb5815bc80 0x7fcb580ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44978/0 (socket says 172.21.15.6:44978) 2024-03-22T13:08:35.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.804+0000 7fcb5ce00640 1 -- 172.21.15.6:0/3501465812 learned_addr learned my addr 172.21.15.6:0/3501465812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.808+0000 7fcb5ce00640 1 -- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb58152400 msgr2=0x7fcb581f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.808+0000 7fcb5ce00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb58152400 0x7fcb581f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.808+0000 7fcb5ce00640 1 -- 172.21.15.6:0/3501465812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb5806f8c0 msgr2=0x7fcb58148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.808+0000 7fcb5ce00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb5806f8c0 0x7fcb58148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:35.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.808+0000 7fcb5ce00640 1 -- 172.21.15.6:0/3501465812 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb58148950 con 0x7fcb5815bc80 2024-03-22T13:08:35.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.811+0000 7fcb5ce00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb5815bc80 0x7fcb580ca8b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000ef00 tx=0x7fcb5000b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.813+0000 7fcb4fe00640 1 -- 172.21.15.6:0/3501465812 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50018030 con 0x7fcb5815bc80 2024-03-22T13:08:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.813+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb58155e90 con 0x7fcb5815bc80 2024-03-22T13:08:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.813+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb581f3f80 con 0x7fcb5815bc80 2024-03-22T13:08:35.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:35.814+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb580756c0 con 0x7fcb5815bc80 2024-03-22T13:08:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.103+0000 7fcb4fe00640 1 -- 172.21.15.6:0/3501465812 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5000bd20 con 0x7fcb5815bc80 2024-03-22T13:08:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.103+0000 7fcb4fe00640 1 -- 172.21.15.6:0/3501465812 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50003bb0 con 0x7fcb5815bc80 2024-03-22T13:08:36.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.107+0000 7fcb4fe00640 1 -- 172.21.15.6:0/3501465812 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb50005d10 con 0x7fcb5815bc80 2024-03-22T13:08:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.109+0000 7fcb4fe00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcb3806bbd0 0x7fcb3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.109+0000 7fcb4fe00640 1 -- 172.21.15.6:0/3501465812 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fcb5008bae0 con 0x7fcb5815bc80 2024-03-22T13:08:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.110+0000 7fcb57e00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcb3806bbd0 0x7fcb3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:36.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.111+0000 7fcb57e00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcb3806bbd0 0x7fcb3806e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009b40 tx=0x7fcb48009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:36.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.120+0000 7fcb4fe00640 1 -- 172.21.15.6:0/3501465812 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcb50059c60 con 0x7fcb5815bc80 2024-03-22T13:08:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.371+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb580c6400 con 0x7fcb5815bc80 2024-03-22T13:08:36.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.392+0000 7fcb4fe00640 1 -- 172.21.15.6:0/3501465812 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fcb5005dd00 con 0x7fcb5815bc80 2024-03-22T13:08:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:36.392 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:36.392 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:36.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcb3806bbd0 msgr2=0x7fcb3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcb3806bbd0 0x7fcb3806e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009b40 tx=0x7fcb48009190 comp rx=0 tx=0).stop 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb5815bc80 msgr2=0x7fcb580ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb5815bc80 0x7fcb580ca8b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000ef00 tx=0x7fcb5000b480 comp rx=0 tx=0).stop 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 shutdown_connections 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb5815bc80 0x7fcb580ca8b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fcb3806bbd0 0x7fcb3806e090 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb58152400 0x7fcb581f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 --2- 172.21.15.6:0/3501465812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb5806f8c0 0x7fcb58148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.394+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 >> 172.21.15.6:0/3501465812 conn(0x7fcb580cb930 msgr2=0x7fcb5814f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.395+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 shutdown_connections 2024-03-22T13:08:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:36.395+0000 7fcb5ee00640 1 -- 172.21.15.6:0/3501465812 wait complete. 2024-03-22T13:08:37.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:37.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:38.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.219+0000 7f88c1a00640 1 Processor -- start 2024-03-22T13:08:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.220+0000 7f88c1a00640 1 -- start start 2024-03-22T13:08:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.221+0000 7f88c1a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f88bc0cc640 0x7f88bc06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.221+0000 7f88c1a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.221+0000 7f88c1a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc06dd00 con 0x7f88bc0cca20 2024-03-22T13:08:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.221+0000 7f88c1a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88bc06d330 con 0x7f88bc0cc640 2024-03-22T13:08:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.221+0000 7f88c1a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc06c440 con 0x7f88bc15b7a0 2024-03-22T13:08:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.222+0000 7f88c1000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.222+0000 7f88c1000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc15e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44984/0 (socket says 172.21.15.6:44984) 2024-03-22T13:08:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.222+0000 7f88c1000640 1 -- 172.21.15.6:0/1592828259 learned_addr learned my addr 172.21.15.6:0/1592828259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.224+0000 7f88c1000640 1 -- 172.21.15.6:0/1592828259 >> v2:172.21.15.92:3300/0 conn(0x7f88bc0cc640 msgr2=0x7f88bc06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.224+0000 7f88c1000640 1 --2- 172.21.15.6:0/1592828259 >> v2:172.21.15.92:3300/0 conn(0x7f88bc0cc640 0x7f88bc06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.224+0000 7f88c1000640 1 -- 172.21.15.6:0/1592828259 >> v1:172.21.15.92:6789/0 conn(0x7f88bc0cca20 legacy=0x7f88bc0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.224+0000 7f88c1000640 1 -- 172.21.15.6:0/1592828259 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc06c720 con 0x7f88bc15b7a0 2024-03-22T13:08:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.226+0000 7f88c1000640 1 --2- 172.21.15.6:0/1592828259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc15e260 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009e50 tx=0x7f88b402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd9489910f75235 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.229+0000 7f88baa00640 1 -- 172.21.15.6:0/1592828259 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b402cb00 con 0x7f88bc15b7a0 2024-03-22T13:08:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.231+0000 7f88baa00640 1 -- 172.21.15.6:0/1592828259 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b402cca0 con 0x7f88bc15b7a0 2024-03-22T13:08:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.231+0000 7f88baa00640 1 -- 172.21.15.6:0/1592828259 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4034610 con 0x7f88bc15b7a0 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 -- 172.21.15.6:0/1592828259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 msgr2=0x7f88bc15e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 --2- 172.21.15.6:0/1592828259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc15e260 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009e50 tx=0x7f88b402bf60 comp rx=0 tx=0).stop 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 -- 172.21.15.6:0/1592828259 shutdown_connections 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 --2- 172.21.15.6:0/1592828259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc15e260 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 --2- 172.21.15.6:0/1592828259 >> v2:172.21.15.92:3300/0 conn(0x7f88bc0cc640 0x7f88bc06d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 -- 172.21.15.6:0/1592828259 >> 172.21.15.6:0/1592828259 conn(0x7f88bc0cf9c0 msgr2=0x7f88bc0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 -- 172.21.15.6:0/1592828259 shutdown_connections 2024-03-22T13:08:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.232+0000 7f88c1a00640 1 -- 172.21.15.6:0/1592828259 wait complete. 2024-03-22T13:08:38.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.235+0000 7f88c1a00640 1 Processor -- start 2024-03-22T13:08:38.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.235+0000 7f88c1a00640 1 -- start start 2024-03-22T13:08:38.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.236+0000 7f88c1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88bc0cc640 0x7f88bc1f0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:38.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.236+0000 7f88c1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88bc0cca20 0x7f88bc0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:38.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.237+0000 7f88c1a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:38.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.237+0000 7f88c1a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc06cfc0 con 0x7f88bc0cca20 2024-03-22T13:08:38.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.237+0000 7f88c1a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc06cce0 con 0x7f88bc0cc640 2024-03-22T13:08:38.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.237+0000 7f88c1a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc0caa50 con 0x7f88bc15b7a0 2024-03-22T13:08:38.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.238+0000 7f88c1000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:38.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.238+0000 7f88c1000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc0b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:44998/0 (socket says 172.21.15.6:44998) 2024-03-22T13:08:38.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.238+0000 7f88c1000640 1 -- 172.21.15.6:0/3384934308 learned_addr learned my addr 172.21.15.6:0/3384934308 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:38.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.241+0000 7f88c1000640 1 -- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88bc0cc640 msgr2=0x7f88bc1f0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:38.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.241+0000 7f88c1000640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88bc0cc640 0x7f88bc1f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.241+0000 7f88c1000640 1 -- 172.21.15.6:0/3384934308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88bc0cca20 msgr2=0x7f88bc0b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:38.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.241+0000 7f88c1000640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88bc0cca20 0x7f88bc0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.241+0000 7f88c1000640 1 -- 172.21.15.6:0/3384934308 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc0c3610 con 0x7f88bc15b7a0 2024-03-22T13:08:38.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.244+0000 7f88c1000640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc0b1bc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f88b4036040 tx=0x7f88b4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.322+0000 7f88b3e00640 1 -- 172.21.15.6:0/3384934308 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4044020 con 0x7f88bc15b7a0 2024-03-22T13:08:38.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.323+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc0af6c0 con 0x7f88bc15b7a0 2024-03-22T13:08:38.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.323+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc0afc80 con 0x7f88bc15b7a0 2024-03-22T13:08:38.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.324+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc152920 con 0x7f88bc15b7a0 2024-03-22T13:08:38.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.329+0000 7f88b3e00640 1 -- 172.21.15.6:0/3384934308 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b402ce50 con 0x7f88bc15b7a0 2024-03-22T13:08:38.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.330+0000 7f88b3e00640 1 -- 172.21.15.6:0/3384934308 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4040670 con 0x7f88bc15b7a0 2024-03-22T13:08:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.603+0000 7f88b3e00640 1 -- 172.21.15.6:0/3384934308 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f88b4047020 con 0x7f88bc15b7a0 2024-03-22T13:08:38.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.605+0000 7f88b3e00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f889c06bc00 0x7f889c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:38.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.605+0000 7f88bbe00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f889c06bc00 0x7f889c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:38.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.606+0000 7f88bbe00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f889c06bc00 0x7f889c06e0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009670 tx=0x7f88ac006bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:38.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.610+0000 7f88b3e00640 1 -- 172.21.15.6:0/3384934308 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f88b4079e30 con 0x7f88bc15b7a0 2024-03-22T13:08:38.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.628+0000 7f88b3e00640 1 -- 172.21.15.6:0/3384934308 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f88bc152920 con 0x7f88bc15b7a0 2024-03-22T13:08:38.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.878+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88bc075650 con 0x7f88bc15b7a0 2024-03-22T13:08:38.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.893+0000 7f88b3e00640 1 -- 172.21.15.6:0/3384934308 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f88b404e030 con 0x7f88bc15b7a0 2024-03-22T13:08:38.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:38.893 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.895+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f889c06bc00 msgr2=0x7f889c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.895+0000 7f88c1a00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f889c06bc00 0x7f889c06e0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009670 tx=0x7f88ac006bf0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.895+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 msgr2=0x7f88bc0b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc0b1bc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f88b4036040 tx=0x7f88b4004a40 comp rx=0 tx=0).stop 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 shutdown_connections 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f88bc15b7a0 0x7f88bc0b1bc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88bc0cca20 0x7f88bc0b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f889c06bc00 0x7f889c06e0c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 --2- 172.21.15.6:0/3384934308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88bc0cc640 0x7f88bc1f0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 >> 172.21.15.6:0/3384934308 conn(0x7f88bc0cf9c0 msgr2=0x7f88bc159460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 shutdown_connections 2024-03-22T13:08:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:38.896+0000 7f88c1a00640 1 -- 172.21.15.6:0/3384934308 wait complete. 2024-03-22T13:08:39.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:39.518 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:40.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:40.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.717+0000 7f6a29400640 1 Processor -- start 2024-03-22T13:08:40.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.718+0000 7f6a29400640 1 -- start start 2024-03-22T13:08:40.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.719+0000 7f6a29400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c9c30 0x7f6a240c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:40.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.719+0000 7f6a29400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2415a3c0 con 0x7f6a241545a0 2024-03-22T13:08:40.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.719+0000 7f6a29400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240c4130 con 0x7f6a240c96e0 2024-03-22T13:08:40.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.719+0000 7f6a29400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240c4410 con 0x7f6a240c9c30 2024-03-22T13:08:40.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.720+0000 7f6a23200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6a240c96e0 0x7f6a24164bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:55462/0 (socket says 172.21.15.6:55462) 2024-03-22T13:08:40.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.720+0000 7f6a23200640 1 -- 172.21.15.6:0/1805564000 learned_addr learned my addr 172.21.15.6:0/1805564000 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:40.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.720+0000 7f6a21e00640 1 --2- 172.21.15.6:0/1805564000 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c9c30 0x7f6a240c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.723+0000 7f6a21400640 1 -- 172.21.15.6:0/1805564000 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233407330 0 0) 0x7f6a240c4130 con 0x7f6a240c96e0 2024-03-22T13:08:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.723+0000 7f6a21400640 1 -- 172.21.15.6:0/1805564000 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a08003520 con 0x7f6a240c96e0 2024-03-22T13:08:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.723+0000 7f6a21400640 1 -- 172.21.15.6:0/1805564000 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316188253 0 0) 0x7f6a2415a3c0 con 0x7f6a241545a0 2024-03-22T13:08:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.723+0000 7f6a21400640 1 -- 172.21.15.6:0/1805564000 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a240c4130 con 0x7f6a241545a0 2024-03-22T13:08:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.724+0000 7f6a21e00640 1 -- 172.21.15.6:0/1805564000 >> v1:172.21.15.92:6789/0 conn(0x7f6a240c96e0 legacy=0x7f6a24164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.724+0000 7f6a21e00640 1 -- 172.21.15.6:0/1805564000 >> v1:172.21.15.6:6789/0 conn(0x7f6a241545a0 legacy=0x7f6a240ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.724+0000 7f6a21e00640 1 -- 172.21.15.6:0/1805564000 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2415a660 con 0x7f6a240c9c30 2024-03-22T13:08:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.727+0000 7f6a21e00640 1 --2- 172.21.15.6:0/1805564000 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c9c30 0x7f6a240c3af0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6a100097d0 tx=0x7f6a1002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d7152a6a00dc98a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:40.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.732+0000 7f6a21400640 1 -- 172.21.15.6:0/1805564000 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a10034070 con 0x7f6a240c9c30 2024-03-22T13:08:40.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.732+0000 7f6a21400640 1 -- 172.21.15.6:0/1805564000 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a10030880 con 0x7f6a240c9c30 2024-03-22T13:08:40.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.732+0000 7f6a21400640 1 -- 172.21.15.6:0/1805564000 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a10030c50 con 0x7f6a240c9c30 2024-03-22T13:08:40.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.733+0000 7f6a29400640 1 -- 172.21.15.6:0/1805564000 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c9c30 msgr2=0x7f6a240c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.733+0000 7f6a29400640 1 --2- 172.21.15.6:0/1805564000 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c9c30 0x7f6a240c3af0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6a100097d0 tx=0x7f6a1002b9c0 comp rx=0 tx=0).stop 2024-03-22T13:08:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.733+0000 7f6a29400640 1 -- 172.21.15.6:0/1805564000 shutdown_connections 2024-03-22T13:08:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.733+0000 7f6a29400640 1 --2- 172.21.15.6:0/1805564000 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c9c30 0x7f6a240c3af0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.733+0000 7f6a29400640 1 -- 172.21.15.6:0/1805564000 >> 172.21.15.6:0/1805564000 conn(0x7f6a240714f0 msgr2=0x7f6a24071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.733+0000 7f6a29400640 1 -- 172.21.15.6:0/1805564000 shutdown_connections 2024-03-22T13:08:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.733+0000 7f6a29400640 1 -- 172.21.15.6:0/1805564000 wait complete. 2024-03-22T13:08:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.735+0000 7f6a29400640 1 Processor -- start 2024-03-22T13:08:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.735+0000 7f6a29400640 1 -- start start 2024-03-22T13:08:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.736+0000 7f6a29400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c96e0 0x7f6a241f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.736+0000 7f6a29400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a240c9c30 0x7f6a240c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.736+0000 7f6a29400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a241545a0 0x7f6a24159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.736+0000 7f6a29400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24154980 con 0x7f6a240c9c30 2024-03-22T13:08:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.736+0000 7f6a29400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240c46f0 con 0x7f6a241545a0 2024-03-22T13:08:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.736+0000 7f6a29400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240cadc0 con 0x7f6a240c96e0 2024-03-22T13:08:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.737+0000 7f6a23200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a241545a0 0x7f6a24159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.737+0000 7f6a22800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c96e0 0x7f6a241f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.737+0000 7f6a21e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a240c9c30 0x7f6a240c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.737+0000 7f6a23200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a241545a0 0x7f6a24159730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:42260/0 (socket says 172.21.15.6:42260) 2024-03-22T13:08:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.737+0000 7f6a23200640 1 -- 172.21.15.6:0/3071161560 learned_addr learned my addr 172.21.15.6:0/3071161560 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.741+0000 7f6a23200640 1 -- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c96e0 msgr2=0x7f6a241f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.741+0000 7f6a23200640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c96e0 0x7f6a241f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.741+0000 7f6a23200640 1 -- 172.21.15.6:0/3071161560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a240c9c30 msgr2=0x7f6a240c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.741+0000 7f6a23200640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a240c9c30 0x7f6a240c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.741+0000 7f6a23200640 1 -- 172.21.15.6:0/3071161560 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a240c83e0 con 0x7f6a241545a0 2024-03-22T13:08:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.744+0000 7f6a23200640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a241545a0 0x7f6a24159730 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00cad0 tx=0x7f6a1c00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:40.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.748+0000 7f6a1aa00640 1 -- 172.21.15.6:0/3071161560 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c017070 con 0x7f6a241545a0 2024-03-22T13:08:40.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.748+0000 7f6a1aa00640 1 -- 172.21.15.6:0/3071161560 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c00ac50 con 0x7f6a241545a0 2024-03-22T13:08:40.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.748+0000 7f6a1aa00640 1 -- 172.21.15.6:0/3071161560 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c012400 con 0x7f6a241545a0 2024-03-22T13:08:40.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.748+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a240c86c0 con 0x7f6a241545a0 2024-03-22T13:08:40.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.749+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a241f33d0 con 0x7f6a241545a0 2024-03-22T13:08:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.750+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a240756c0 con 0x7f6a241545a0 2024-03-22T13:08:40.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.773+0000 7f6a1aa00640 1 -- 172.21.15.6:0/3071161560 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6a1c01a070 con 0x7f6a241545a0 2024-03-22T13:08:40.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.774+0000 7f6a1aa00640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6a0406bc20 0x7f6a0406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:40.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.775+0000 7f6a22800640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6a0406bc20 0x7f6a0406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:40.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:40.776+0000 7f6a22800640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6a0406bc20 0x7f6a0406e0e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6a14007fd0 tx=0x7f6a14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:41.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.021+0000 7f6a1aa00640 1 -- 172.21.15.6:0/3071161560 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6a1c0570f0 con 0x7f6a241545a0 2024-03-22T13:08:41.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.033+0000 7f6a1aa00640 1 -- 172.21.15.6:0/3071161560 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6a1c021020 con 0x7f6a241545a0 2024-03-22T13:08:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.289+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a240c9410 con 0x7f6a241545a0 2024-03-22T13:08:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.359+0000 7f6a1aa00640 1 -- 172.21.15.6:0/3071161560 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f6a1c059050 con 0x7f6a241545a0 2024-03-22T13:08:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:41.359 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6a0406bc20 msgr2=0x7f6a0406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6a0406bc20 0x7f6a0406e0e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6a14007fd0 tx=0x7f6a14009040 comp rx=0 tx=0).stop 2024-03-22T13:08:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a241545a0 msgr2=0x7f6a24159730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a241545a0 0x7f6a24159730 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00cad0 tx=0x7f6a1c00cf90 comp rx=0 tx=0).stop 2024-03-22T13:08:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 shutdown_connections 2024-03-22T13:08:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6a0406bc20 0x7f6a0406e0e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a241545a0 0x7f6a24159730 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a240c9c30 0x7f6a240c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 --2- 172.21.15.6:0/3071161560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a240c96e0 0x7f6a241f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 >> 172.21.15.6:0/3071161560 conn(0x7f6a240714f0 msgr2=0x7f6a240718d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 shutdown_connections 2024-03-22T13:08:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:41.362+0000 7f6a29400640 1 -- 172.21.15.6:0/3071161560 wait complete. 2024-03-22T13:08:41.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:41.986 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:42.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:43.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.181+0000 7f1cdba00640 1 Processor -- start 2024-03-22T13:08:43.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.181+0000 7f1cdba00640 1 -- start start 2024-03-22T13:08:43.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.183+0000 7f1cdba00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1cd4154240 0x7f1cd415ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:43.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.183+0000 7f1cdba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1cd40c7ee0 0x7f1cd40c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:43.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.183+0000 7f1cdba00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd4154620 con 0x7f1cd40c68c0 2024-03-22T13:08:43.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.183+0000 7f1cdba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd415e130 con 0x7f1cd40c7ee0 2024-03-22T13:08:43.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.183+0000 7f1cdba00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40c6e00 con 0x7f1cd4154240 2024-03-22T13:08:43.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.184+0000 7f1cd3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1cd40c7ee0 0x7f1cd40c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:43.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.184+0000 7f1cd3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1cd40c7ee0 0x7f1cd40c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44394/0 (socket says 172.21.15.6:44394) 2024-03-22T13:08:43.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.184+0000 7f1cd3e00640 1 -- 172.21.15.6:0/2384004907 learned_addr learned my addr 172.21.15.6:0/2384004907 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:43.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.184+0000 7f1cd9000640 1 --2- 172.21.15.6:0/2384004907 >> v2:172.21.15.92:3300/0 conn(0x7f1cd4154240 0x7f1cd415ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:43.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.187+0000 7f1cd3e00640 1 -- 172.21.15.6:0/2384004907 >> v2:172.21.15.92:3300/0 conn(0x7f1cd4154240 msgr2=0x7f1cd415ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:43.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.187+0000 7f1cd3e00640 1 --2- 172.21.15.6:0/2384004907 >> v2:172.21.15.92:3300/0 conn(0x7f1cd4154240 0x7f1cd415ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.187+0000 7f1cd3e00640 1 -- 172.21.15.6:0/2384004907 >> v1:172.21.15.92:6789/0 conn(0x7f1cd40c68c0 legacy=0x7f1cd406b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:43.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.187+0000 7f1cd3e00640 1 -- 172.21.15.6:0/2384004907 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40c70e0 con 0x7f1cd40c7ee0 2024-03-22T13:08:43.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.190+0000 7f1cd3e00640 1 --2- 172.21.15.6:0/2384004907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 0x7f1cd40c4030 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4071390 tx=0x7f1cc002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828d1601ef0d7263 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.195+0000 7f1cd3400640 1 -- 172.21.15.6:0/2384004907 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc002f850 con 0x7f1cd40c7ee0 2024-03-22T13:08:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.195+0000 7f1cd3400640 1 -- 172.21.15.6:0/2384004907 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc002f9f0 con 0x7f1cd40c7ee0 2024-03-22T13:08:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.195+0000 7f1cd3400640 1 -- 172.21.15.6:0/2384004907 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc002fdc0 con 0x7f1cd40c7ee0 2024-03-22T13:08:43.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.196+0000 7f1cdba00640 1 -- 172.21.15.6:0/2384004907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 msgr2=0x7f1cd40c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:43.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.196+0000 7f1cdba00640 1 --2- 172.21.15.6:0/2384004907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 0x7f1cd40c4030 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4071390 tx=0x7f1cc002ba30 comp rx=0 tx=0).stop 2024-03-22T13:08:43.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.196+0000 7f1cdba00640 1 -- 172.21.15.6:0/2384004907 shutdown_connections 2024-03-22T13:08:43.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.196+0000 7f1cdba00640 1 --2- 172.21.15.6:0/2384004907 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 0x7f1cd40c4030 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.196+0000 7f1cdba00640 1 --2- 172.21.15.6:0/2384004907 >> v2:172.21.15.92:3300/0 conn(0x7f1cd4154240 0x7f1cd415ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.196+0000 7f1cdba00640 1 -- 172.21.15.6:0/2384004907 >> 172.21.15.6:0/2384004907 conn(0x7f1cd40cb930 msgr2=0x7f1cd40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.196+0000 7f1cdba00640 1 -- 172.21.15.6:0/2384004907 shutdown_connections 2024-03-22T13:08:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.197+0000 7f1cdba00640 1 -- 172.21.15.6:0/2384004907 wait complete. 2024-03-22T13:08:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.198+0000 7f1cdba00640 1 Processor -- start 2024-03-22T13:08:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.198+0000 7f1cdba00640 1 -- start start 2024-03-22T13:08:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.199+0000 7f1cdba00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cd40c68c0 0x7f1cd40c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.199+0000 7f1cdba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 0x7f1cd4152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.199+0000 7f1cdba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cd4154240 0x7f1cd4151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.199+0000 7f1cdba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd415ddc0 con 0x7f1cd40c7ee0 2024-03-22T13:08:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.199+0000 7f1cdba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd415dae0 con 0x7f1cd4154240 2024-03-22T13:08:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.199+0000 7f1cdba00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd415e5c0 con 0x7f1cd40c68c0 2024-03-22T13:08:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.200+0000 7f1cd3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 0x7f1cd4152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.200+0000 7f1cd9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cd4154240 0x7f1cd4151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.200+0000 7f1cd9000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cd40c68c0 0x7f1cd40c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.200+0000 7f1cd9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cd4154240 0x7f1cd4151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48554/0 (socket says 172.21.15.6:48554) 2024-03-22T13:08:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.200+0000 7f1cd9a00640 1 -- 172.21.15.6:0/670491647 learned_addr learned my addr 172.21.15.6:0/670491647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:43.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.204+0000 7f1cd9a00640 1 -- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cd40c68c0 msgr2=0x7f1cd40c9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:43.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.204+0000 7f1cd9a00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cd40c68c0 0x7f1cd40c9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.204+0000 7f1cd9a00640 1 -- 172.21.15.6:0/670491647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 msgr2=0x7f1cd4152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:43.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.204+0000 7f1cd9a00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 0x7f1cd4152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.204+0000 7f1cd9a00640 1 -- 172.21.15.6:0/670491647 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd41f3ad0 con 0x7f1cd4154240 2024-03-22T13:08:43.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.206+0000 7f1cd9a00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cd4154240 0x7f1cd4151fd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c7a0 tx=0x7f1ccc00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:43.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.211+0000 7f1cd0c00640 1 -- 172.21.15.6:0/670491647 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc003b70 con 0x7f1cd4154240 2024-03-22T13:08:43.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.211+0000 7f1cd0c00640 1 -- 172.21.15.6:0/670491647 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc003d30 con 0x7f1cd4154240 2024-03-22T13:08:43.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.211+0000 7f1cd0c00640 1 -- 172.21.15.6:0/670491647 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00aa90 con 0x7f1cd4154240 2024-03-22T13:08:43.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.211+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd41f3d70 con 0x7f1cd4154240 2024-03-22T13:08:43.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.211+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd41f4330 con 0x7f1cd4154240 2024-03-22T13:08:43.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.212+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd40756c0 con 0x7f1cd4154240 2024-03-22T13:08:43.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.237+0000 7f1cd0c00640 1 -- 172.21.15.6:0/670491647 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1ccc00ac30 con 0x7f1cd4154240 2024-03-22T13:08:43.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.237+0000 7f1cd0c00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cb406bc50 0x7f1cb406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:43.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.238+0000 7f1cd9000640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cb406bc50 0x7f1cb406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:43.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.239+0000 7f1cd9000640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cb406bc50 0x7f1cb406e110 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009aa0 tx=0x7f1cc8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:43.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.510+0000 7f1cd0c00640 1 -- 172.21.15.6:0/670491647 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1ccc057830 con 0x7f1cd4154240 2024-03-22T13:08:43.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.522+0000 7f1cd0c00640 1 -- 172.21.15.6:0/670491647 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1ccc059790 con 0x7f1cd4154240 2024-03-22T13:08:43.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.804+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cd4074d50 con 0x7f1cd4154240 2024-03-22T13:08:43.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.820+0000 7f1cd0c00640 1 -- 172.21.15.6:0/670491647 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f1ccc05d830 con 0x7f1cd4154240 2024-03-22T13:08:43.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:43.820 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:43.820 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:43.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cb406bc50 msgr2=0x7f1cb406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cb406bc50 0x7f1cb406e110 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009aa0 tx=0x7f1cc8009190 comp rx=0 tx=0).stop 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cd4154240 msgr2=0x7f1cd4151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cd4154240 0x7f1cd4151fd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c7a0 tx=0x7f1ccc00cc60 comp rx=0 tx=0).stop 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 shutdown_connections 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cb406bc50 0x7f1cb406e110 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cd4154240 0x7f1cd4151fd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cd40c7ee0 0x7f1cd4152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 --2- 172.21.15.6:0/670491647 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cd40c68c0 0x7f1cd40c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.822+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 >> 172.21.15.6:0/670491647 conn(0x7f1cd40cb930 msgr2=0x7f1cd40c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.823+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 shutdown_connections 2024-03-22T13:08:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:43.823+0000 7f1cdba00640 1 -- 172.21.15.6:0/670491647 wait complete. 2024-03-22T13:08:44.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:44.443 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:45.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:45.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.632+0000 7f577ca00640 1 Processor -- start 2024-03-22T13:08:45.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.632+0000 7f577ca00640 1 -- start start 2024-03-22T13:08:45.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.634+0000 7f577ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5778152400 0x7f577815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:45.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.634+0000 7f577ca00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f577806f8c0 0x7f57780c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:45.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.634+0000 7f577ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57781590b0 con 0x7f577815bc80 2024-03-22T13:08:45.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.634+0000 7f577ca00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57780724d0 con 0x7f577806f8c0 2024-03-22T13:08:45.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.634+0000 7f577ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57780721f0 con 0x7f5778152400 2024-03-22T13:08:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.635+0000 7f5775e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5778152400 0x7f577815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.635+0000 7f5775400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f577815bc80 0x7f57780716d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36496/0 (socket says 172.21.15.6:36496) 2024-03-22T13:08:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.635+0000 7f5775400640 1 -- 172.21.15.6:0/1167470250 learned_addr learned my addr 172.21.15.6:0/1167470250 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.635+0000 7f5776800640 1 --2- 172.21.15.6:0/1167470250 >> v2:172.21.15.92:3300/0 conn(0x7f577806f8c0 0x7f57780c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.638+0000 7f5775e00640 1 -- 172.21.15.6:0/1167470250 >> v2:172.21.15.92:3300/0 conn(0x7f577806f8c0 msgr2=0x7f57780c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.638+0000 7f5775e00640 1 --2- 172.21.15.6:0/1167470250 >> v2:172.21.15.92:3300/0 conn(0x7f577806f8c0 0x7f57780c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.638+0000 7f5775e00640 1 -- 172.21.15.6:0/1167470250 >> v1:172.21.15.6:6789/0 conn(0x7f577815bc80 legacy=0x7f57780716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.638+0000 7f5775e00640 1 -- 172.21.15.6:0/1167470250 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57781582f0 con 0x7f5778152400 2024-03-22T13:08:45.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.640+0000 7f5775e00640 1 --2- 172.21.15.6:0/1167470250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5778152400 0x7f577815b740 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f57680097a0 tx=0x7f576802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e65116c6921be743 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:45.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.644+0000 7f5774a00640 1 -- 172.21.15.6:0/1167470250 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768034070 con 0x7f5778152400 2024-03-22T13:08:45.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.644+0000 7f5774a00640 1 -- 172.21.15.6:0/1167470250 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5768030610 con 0x7f5778152400 2024-03-22T13:08:45.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.644+0000 7f5774a00640 1 -- 172.21.15.6:0/1167470250 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680309e0 con 0x7f5778152400 2024-03-22T13:08:45.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.645+0000 7f577ca00640 1 -- 172.21.15.6:0/1167470250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5778152400 msgr2=0x7f577815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:45.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.645+0000 7f577ca00640 1 --2- 172.21.15.6:0/1167470250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5778152400 0x7f577815b740 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f57680097a0 tx=0x7f576802b960 comp rx=0 tx=0).stop 2024-03-22T13:08:45.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.645+0000 7f577ca00640 1 -- 172.21.15.6:0/1167470250 shutdown_connections 2024-03-22T13:08:45.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.645+0000 7f577ca00640 1 --2- 172.21.15.6:0/1167470250 >> v2:172.21.15.92:3300/0 conn(0x7f577806f8c0 0x7f57780c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:45.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.645+0000 7f577ca00640 1 --2- 172.21.15.6:0/1167470250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5778152400 0x7f577815b740 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:45.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.646+0000 7f577ca00640 1 -- 172.21.15.6:0/1167470250 >> 172.21.15.6:0/1167470250 conn(0x7f57780cb930 msgr2=0x7f57780cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:45.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.646+0000 7f577ca00640 1 -- 172.21.15.6:0/1167470250 shutdown_connections 2024-03-22T13:08:45.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.646+0000 7f577ca00640 1 -- 172.21.15.6:0/1167470250 wait complete. 2024-03-22T13:08:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.648+0000 7f577ca00640 1 Processor -- start 2024-03-22T13:08:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.649+0000 7f577ca00640 1 -- start start 2024-03-22T13:08:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.650+0000 7f577ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577806f8c0 0x7f5778145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.650+0000 7f577ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5778152400 0x7f5778146760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.650+0000 7f577ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f577815bc80 0x7f5778146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.650+0000 7f577ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778158b90 con 0x7f5778152400 2024-03-22T13:08:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.650+0000 7f577ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57781588b0 con 0x7f577806f8c0 2024-03-22T13:08:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.650+0000 7f577ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57781527e0 con 0x7f577815bc80 2024-03-22T13:08:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.651+0000 7f5776800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f577815bc80 0x7f5778146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.651+0000 7f5775400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5778152400 0x7f5778146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.651+0000 7f5776800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f577815bc80 0x7f5778146ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51536/0 (socket says 172.21.15.6:51536) 2024-03-22T13:08:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.651+0000 7f5776800640 1 -- 172.21.15.6:0/3492537236 learned_addr learned my addr 172.21.15.6:0/3492537236 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:45.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.657+0000 7f5776800640 1 -- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577806f8c0 msgr2=0x7f5778145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:45.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.657+0000 7f5776800640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577806f8c0 0x7f5778145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.657+0000 7f5776800640 1 -- 172.21.15.6:0/3492537236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5778152400 msgr2=0x7f5778146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.657+0000 7f5776800640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5778152400 0x7f5778146760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.657+0000 7f5776800640 1 -- 172.21.15.6:0/3492537236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57781ec050 con 0x7f577815bc80 2024-03-22T13:08:45.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.660+0000 7f5776800640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f577815bc80 0x7f5778146ca0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5770009dd0 tx=0x7f577000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.664+0000 7f576e000640 1 -- 172.21.15.6:0/3492537236 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770018020 con 0x7f577815bc80 2024-03-22T13:08:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.664+0000 7f576e000640 1 -- 172.21.15.6:0/3492537236 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5770014480 con 0x7f577815bc80 2024-03-22T13:08:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.664+0000 7f576e000640 1 -- 172.21.15.6:0/3492537236 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770014870 con 0x7f577815bc80 2024-03-22T13:08:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.664+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57781ec2d0 con 0x7f577815bc80 2024-03-22T13:08:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.664+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57781ec890 con 0x7f577815bc80 2024-03-22T13:08:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.665+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57780756c0 con 0x7f577815bc80 2024-03-22T13:08:45.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.698+0000 7f576e000640 1 -- 172.21.15.6:0/3492537236 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5770005d10 con 0x7f577815bc80 2024-03-22T13:08:45.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.699+0000 7f576e000640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f575806bc20 0x7f575806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:45.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.700+0000 7f5775e00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f575806bc20 0x7f575806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.701+0000 7f5775e00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f575806bc20 0x7f575806e0e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5768036000 tx=0x7f5768030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:45.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.702+0000 7f576e000640 1 -- 172.21.15.6:0/3492537236 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f57700581e0 con 0x7f577815bc80 2024-03-22T13:08:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:45.961+0000 7f576e000640 1 -- 172.21.15.6:0/3492537236 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5770022030 con 0x7f577815bc80 2024-03-22T13:08:46.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.227+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5778148cb0 con 0x7f577815bc80 2024-03-22T13:08:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.248+0000 7f576e000640 1 -- 172.21.15.6:0/3492537236 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f5770060030 con 0x7f577815bc80 2024-03-22T13:08:46.248 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:46.248 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f575806bc20 msgr2=0x7f575806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f575806bc20 0x7f575806e0e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5768036000 tx=0x7f5768030000 comp rx=0 tx=0).stop 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f577815bc80 msgr2=0x7f5778146ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f577815bc80 0x7f5778146ca0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5770009dd0 tx=0x7f577000c490 comp rx=0 tx=0).stop 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 shutdown_connections 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f577815bc80 0x7f5778146ca0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5778152400 0x7f5778146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f575806bc20 0x7f575806e0e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 --2- 172.21.15.6:0/3492537236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577806f8c0 0x7f5778145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 >> 172.21.15.6:0/3492537236 conn(0x7f57780cb930 msgr2=0x7f57780716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 shutdown_connections 2024-03-22T13:08:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:46.251+0000 7f577ca00640 1 -- 172.21.15.6:0/3492537236 wait complete. 2024-03-22T13:08:46.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:46.873 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:47.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:48.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.059+0000 7f2587000640 1 Processor -- start 2024-03-22T13:08:48.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.060+0000 7f2587000640 1 -- start start 2024-03-22T13:08:48.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.061+0000 7f2587000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f25800cc380 0x7f258006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:48.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.061+0000 7f2587000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f258015ae30 0x7f25800cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:48.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.061+0000 7f2587000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258015b240 con 0x7f25800ce8a0 2024-03-22T13:08:48.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.061+0000 7f2587000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25800cee00 con 0x7f25800cc380 2024-03-22T13:08:48.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.061+0000 7f2587000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25800cf0c0 con 0x7f258015ae30 2024-03-22T13:08:48.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.062+0000 7f2586600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f258015ae30 0x7f25800cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:48.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.062+0000 7f2586600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f258015ae30 0x7f25800cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48586/0 (socket says 172.21.15.6:48586) 2024-03-22T13:08:48.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.062+0000 7f2586600640 1 -- 172.21.15.6:0/2352302969 learned_addr learned my addr 172.21.15.6:0/2352302969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:48.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.063+0000 7f2585c00640 1 --2- 172.21.15.6:0/2352302969 >> v2:172.21.15.6:3300/0 conn(0x7f25800cc380 0x7f258006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:48.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.064+0000 7f2586600640 1 -- 172.21.15.6:0/2352302969 >> v2:172.21.15.6:3300/0 conn(0x7f25800cc380 msgr2=0x7f258006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:48.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.064+0000 7f2586600640 1 --2- 172.21.15.6:0/2352302969 >> v2:172.21.15.6:3300/0 conn(0x7f25800cc380 0x7f258006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.064+0000 7f2586600640 1 -- 172.21.15.6:0/2352302969 >> v1:172.21.15.6:6789/0 conn(0x7f25800ce8a0 legacy=0x7f25800cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:48.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.064+0000 7f2586600640 1 -- 172.21.15.6:0/2352302969 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25800cf3a0 con 0x7f258015ae30 2024-03-22T13:08:48.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.066+0000 7f2586600640 1 --2- 172.21.15.6:0/2352302969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f258015ae30 0x7f25800cc760 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f25800c9f50 tx=0x7f257802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f749bd123714eea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:48.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.070+0000 7f257fe00640 1 -- 172.21.15.6:0/2352302969 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257802cb20 con 0x7f258015ae30 2024-03-22T13:08:48.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.071+0000 7f257fe00640 1 -- 172.21.15.6:0/2352302969 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257802ccc0 con 0x7f258015ae30 2024-03-22T13:08:48.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.071+0000 7f257fe00640 1 -- 172.21.15.6:0/2352302969 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578034610 con 0x7f258015ae30 2024-03-22T13:08:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.071+0000 7f2587000640 1 -- 172.21.15.6:0/2352302969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f258015ae30 msgr2=0x7f25800cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.071+0000 7f2587000640 1 --2- 172.21.15.6:0/2352302969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f258015ae30 0x7f25800cc760 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f25800c9f50 tx=0x7f257802bf80 comp rx=0 tx=0).stop 2024-03-22T13:08:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.072+0000 7f2587000640 1 -- 172.21.15.6:0/2352302969 shutdown_connections 2024-03-22T13:08:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.072+0000 7f2587000640 1 --2- 172.21.15.6:0/2352302969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f258015ae30 0x7f25800cc760 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.072+0000 7f2587000640 1 --2- 172.21.15.6:0/2352302969 >> v2:172.21.15.6:3300/0 conn(0x7f25800cc380 0x7f258006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.072+0000 7f2587000640 1 -- 172.21.15.6:0/2352302969 >> 172.21.15.6:0/2352302969 conn(0x7f25800b1480 msgr2=0x7f25800b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.072+0000 7f2587000640 1 -- 172.21.15.6:0/2352302969 shutdown_connections 2024-03-22T13:08:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.072+0000 7f2587000640 1 -- 172.21.15.6:0/2352302969 wait complete. 2024-03-22T13:08:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.074+0000 7f2587000640 1 Processor -- start 2024-03-22T13:08:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.074+0000 7f2587000640 1 -- start start 2024-03-22T13:08:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.075+0000 7f2587000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25800cc380 0x7f258015def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.075+0000 7f2587000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25800ce8a0 0x7f25800b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.075+0000 7f2587000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f258015ae30 0x7f258015b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.075+0000 7f2587000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258015a910 con 0x7f258015ae30 2024-03-22T13:08:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.075+0000 7f2587000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258015a630 con 0x7f25800cc380 2024-03-22T13:08:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.075+0000 7f2587000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25801549d0 con 0x7f25800ce8a0 2024-03-22T13:08:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.075+0000 7f2585c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25800cc380 0x7f258015def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.076+0000 7f2585c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25800cc380 0x7f258015def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48588/0 (socket says 172.21.15.6:48588) 2024-03-22T13:08:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.076+0000 7f2585c00640 1 -- 172.21.15.6:0/505786169 learned_addr learned my addr 172.21.15.6:0/505786169 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.076+0000 7f2586600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f258015ae30 0x7f258015b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.076+0000 7f2585200640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25800ce8a0 0x7f25800b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.081+0000 7f2585c00640 1 -- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25800ce8a0 msgr2=0x7f25800b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.081+0000 7f2585c00640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25800ce8a0 0x7f25800b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.081+0000 7f2585c00640 1 -- 172.21.15.6:0/505786169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f258015ae30 msgr2=0x7f258015b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.081+0000 7f2585c00640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f258015ae30 0x7f258015b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.081+0000 7f2585c00640 1 -- 172.21.15.6:0/505786169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258015c4d0 con 0x7f25800cc380 2024-03-22T13:08:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.084+0000 7f2585c00640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25800cc380 0x7f258015def0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f257400d5f0 tx=0x7f257400dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.089+0000 7f257d600640 1 -- 172.21.15.6:0/505786169 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25740125f0 con 0x7f25800cc380 2024-03-22T13:08:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.089+0000 7f257d600640 1 -- 172.21.15.6:0/505786169 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2574019070 con 0x7f25800cc380 2024-03-22T13:08:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.089+0000 7f257d600640 1 -- 172.21.15.6:0/505786169 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25740129d0 con 0x7f25800cc380 2024-03-22T13:08:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.089+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258015c7b0 con 0x7f25800cc380 2024-03-22T13:08:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.089+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258015cd70 con 0x7f25800cc380 2024-03-22T13:08:48.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.090+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2580075650 con 0x7f25800cc380 2024-03-22T13:08:48.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.122+0000 7f257d600640 1 -- 172.21.15.6:0/505786169 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2574012b70 con 0x7f25800cc380 2024-03-22T13:08:48.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.124+0000 7f257d600640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f256006bc50 0x7f256006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:48.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.124+0000 7f2585200640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f256006bc50 0x7f256006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:48.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.125+0000 7f2585200640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f256006bc50 0x7f256006e110 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f258015c150 tx=0x7f256c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:48.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.127+0000 7f257d600640 1 -- 172.21.15.6:0/505786169 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f2574012e60 con 0x7f25800cc380 2024-03-22T13:08:48.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.383+0000 7f257d600640 1 -- 172.21.15.6:0/505786169 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2574059c20 con 0x7f25800cc380 2024-03-22T13:08:48.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.655+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2580151c80 con 0x7f25800cc380 2024-03-22T13:08:48.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.675+0000 7f257d600640 1 -- 172.21.15.6:0/505786169 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f257405dcc0 con 0x7f25800cc380 2024-03-22T13:08:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:48.676 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:48.676 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f256006bc50 msgr2=0x7f256006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f256006bc50 0x7f256006e110 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f258015c150 tx=0x7f256c009040 comp rx=0 tx=0).stop 2024-03-22T13:08:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25800cc380 msgr2=0x7f258015def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25800cc380 0x7f258015def0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f257400d5f0 tx=0x7f257400dab0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 shutdown_connections 2024-03-22T13:08:48.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f258015ae30 0x7f258015b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25800ce8a0 0x7f25800b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f256006bc50 0x7f256006e110 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 --2- 172.21.15.6:0/505786169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25800cc380 0x7f258015def0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:48.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 >> 172.21.15.6:0/505786169 conn(0x7f25800b1480 msgr2=0x7f25800cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:48.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.678+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 shutdown_connections 2024-03-22T13:08:48.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:48.679+0000 7f2587000640 1 -- 172.21.15.6:0/505786169 wait complete. 2024-03-22T13:08:49.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:49.299 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:50.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:50.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.498+0000 7f7ddf200640 1 Processor -- start 2024-03-22T13:08:50.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.498+0000 7f7ddf200640 1 -- start start 2024-03-22T13:08:50.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.500+0000 7f7ddf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7de014da20 0x7f7de006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:50.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.500+0000 7f7ddf200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7de006e2e0 0x7f7de00c4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:50.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.500+0000 7f7ddf200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de00c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:50.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.500+0000 7f7ddf200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de00c3f50 con 0x7f7de014da20 2024-03-22T13:08:50.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.500+0000 7f7ddf200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de006f560 con 0x7f7de006e2e0 2024-03-22T13:08:50.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.500+0000 7f7ddf200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00c62c0 con 0x7f7de00c4970 2024-03-22T13:08:50.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.501+0000 7f7dde800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de00c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:50.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.501+0000 7f7ddd400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7de006e2e0 0x7f7de00c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:50.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.501+0000 7f7dde800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de00c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51560/0 (socket says 172.21.15.6:51560) 2024-03-22T13:08:50.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.501+0000 7f7dde800640 1 -- 172.21.15.6:0/2213022922 learned_addr learned my addr 172.21.15.6:0/2213022922 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:50.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.501+0000 7f7ddd400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7de006e2e0 0x7f7de00c4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48600/0 (socket says 172.21.15.6:48600) 2024-03-22T13:08:50.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.503+0000 7f7dde800640 1 -- 172.21.15.6:0/2213022922 >> v2:172.21.15.92:3300/0 conn(0x7f7de006e2e0 msgr2=0x7f7de00c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:50.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.503+0000 7f7dde800640 1 --2- 172.21.15.6:0/2213022922 >> v2:172.21.15.92:3300/0 conn(0x7f7de006e2e0 0x7f7de00c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:50.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.503+0000 7f7dde800640 1 -- 172.21.15.6:0/2213022922 >> v2:172.21.15.6:3300/0 conn(0x7f7de014da20 msgr2=0x7f7de006ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:50.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.503+0000 7f7dde800640 1 --2- 172.21.15.6:0/2213022922 >> v2:172.21.15.6:3300/0 conn(0x7f7de014da20 0x7f7de006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:50.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.503+0000 7f7dde800640 1 -- 172.21.15.6:0/2213022922 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de00c65a0 con 0x7f7de00c4970 2024-03-22T13:08:50.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.505+0000 7f7dde800640 1 --2- 172.21.15.6:0/2213022922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de00c9410 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7de0072350 tx=0x7f7dd802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9218b2e15eaf7b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:50.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.509+0000 7f7ddca00640 1 -- 172.21.15.6:0/2213022922 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd802cb00 con 0x7f7de00c4970 2024-03-22T13:08:50.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddca00640 1 -- 172.21.15.6:0/2213022922 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd802cca0 con 0x7f7de00c4970 2024-03-22T13:08:50.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddca00640 1 -- 172.21.15.6:0/2213022922 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8034610 con 0x7f7de00c4970 2024-03-22T13:08:50.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 -- 172.21.15.6:0/2213022922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 msgr2=0x7f7de00c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:50.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 --2- 172.21.15.6:0/2213022922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de00c9410 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7de0072350 tx=0x7f7dd802bf60 comp rx=0 tx=0).stop 2024-03-22T13:08:50.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 -- 172.21.15.6:0/2213022922 shutdown_connections 2024-03-22T13:08:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 --2- 172.21.15.6:0/2213022922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de00c9410 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 --2- 172.21.15.6:0/2213022922 >> v2:172.21.15.92:3300/0 conn(0x7f7de006e2e0 0x7f7de00c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 --2- 172.21.15.6:0/2213022922 >> v2:172.21.15.6:3300/0 conn(0x7f7de014da20 0x7f7de006ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 -- 172.21.15.6:0/2213022922 >> 172.21.15.6:0/2213022922 conn(0x7f7de006c190 msgr2=0x7f7de006c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.510+0000 7f7ddf200640 1 -- 172.21.15.6:0/2213022922 shutdown_connections 2024-03-22T13:08:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.511+0000 7f7ddf200640 1 -- 172.21.15.6:0/2213022922 wait complete. 2024-03-22T13:08:50.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.513+0000 7f7ddf200640 1 Processor -- start 2024-03-22T13:08:50.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.513+0000 7f7ddf200640 1 -- start start 2024-03-22T13:08:50.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.515+0000 7f7ddf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7de006e2e0 0x7f7de0150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:50.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.515+0000 7f7ddf200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de01587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:50.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.515+0000 7f7ddf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7de014da20 0x7f7de0158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:50.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.515+0000 7f7ddf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00c6e40 con 0x7f7de006e2e0 2024-03-22T13:08:50.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.515+0000 7f7ddf200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00c6b60 con 0x7f7de014da20 2024-03-22T13:08:50.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.515+0000 7f7ddf200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de014de00 con 0x7f7de00c4970 2024-03-22T13:08:50.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.516+0000 7f7ddd400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de01587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:50.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.516+0000 7f7dde800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7de014da20 0x7f7de0158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:50.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.516+0000 7f7ddd400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de01587f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:51568/0 (socket says 172.21.15.6:51568) 2024-03-22T13:08:50.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.516+0000 7f7dde800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7de014da20 0x7f7de0158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48608/0 (socket says 172.21.15.6:48608) 2024-03-22T13:08:50.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.516+0000 7f7ddd400640 1 -- 172.21.15.6:0/959389760 learned_addr learned my addr 172.21.15.6:0/959389760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:50.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.519+0000 7f7ddd400640 1 -- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7de014da20 msgr2=0x7f7de0158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:50.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.519+0000 7f7ddd400640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7de014da20 0x7f7de0158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:50.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.519+0000 7f7ddd400640 1 -- 172.21.15.6:0/959389760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7de006e2e0 msgr2=0x7f7de0150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:50.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.520+0000 7f7ddd400640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7de006e2e0 0x7f7de0150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:50.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.520+0000 7f7ddd400640 1 -- 172.21.15.6:0/959389760 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de0149d90 con 0x7f7de00c4970 2024-03-22T13:08:50.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.522+0000 7f7ddd400640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de01587f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00e8a0 tx=0x7f7dcc00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:50.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.526+0000 7f7dd6000640 1 -- 172.21.15.6:0/959389760 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc017070 con 0x7f7de00c4970 2024-03-22T13:08:50.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.526+0000 7f7dd6000640 1 -- 172.21.15.6:0/959389760 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc00c940 con 0x7f7de00c4970 2024-03-22T13:08:50.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.526+0000 7f7dd6000640 1 -- 172.21.15.6:0/959389760 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc00cd30 con 0x7f7de00c4970 2024-03-22T13:08:50.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.526+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de014a070 con 0x7f7de00c4970 2024-03-22T13:08:50.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.526+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de014a630 con 0x7f7de00c4970 2024-03-22T13:08:50.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.527+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de014a830 con 0x7f7de00c4970 2024-03-22T13:08:50.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.557+0000 7f7dd6000640 1 -- 172.21.15.6:0/959389760 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7dcc01a070 con 0x7f7de00c4970 2024-03-22T13:08:50.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.558+0000 7f7dd6000640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7dc006bc50 0x7f7dc006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:50.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.559+0000 7f7ddde00640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7dc006bc50 0x7f7dc006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.559+0000 7f7ddde00640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7dc006bc50 0x7f7dc006e110 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009710 tx=0x7f7dd0006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:50.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.835+0000 7f7dd6000640 1 -- 172.21.15.6:0/959389760 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f7dcc057520 con 0x7f7de00c4970 2024-03-22T13:08:50.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:50.854+0000 7f7dd6000640 1 -- 172.21.15.6:0/959389760 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7dcc059480 con 0x7f7de00c4970 2024-03-22T13:08:51.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.099+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7de0075650 con 0x7f7de00c4970 2024-03-22T13:08:51.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.112+0000 7f7dd6000640 1 -- 172.21.15.6:0/959389760 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f7dcc05d520 con 0x7f7de00c4970 2024-03-22T13:08:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:51.112 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:51.112 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7dc006bc50 msgr2=0x7f7dc006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7dc006bc50 0x7f7dc006e110 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009710 tx=0x7f7dd0006c90 comp rx=0 tx=0).stop 2024-03-22T13:08:51.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 msgr2=0x7f7de01587f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de01587f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00e8a0 tx=0x7f7dcc00ed60 comp rx=0 tx=0).stop 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 shutdown_connections 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f7dc006bc50 0x7f7dc006e110 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7de014da20 0x7f7de0158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7de00c4970 0x7f7de01587f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 --2- 172.21.15.6:0/959389760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7de006e2e0 0x7f7de0150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 >> 172.21.15.6:0/959389760 conn(0x7f7de006c190 msgr2=0x7f7de00c7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 shutdown_connections 2024-03-22T13:08:51.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:51.114+0000 7f7ddf200640 1 -- 172.21.15.6:0/959389760 wait complete. 2024-03-22T13:08:51.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:51.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:52.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.943+0000 7f9f0b600640 1 Processor -- start 2024-03-22T13:08:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.943+0000 7f9f0b600640 1 -- start start 2024-03-22T13:08:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.945+0000 7f9f0b600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f0406f8c0 0x7f9f040716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.945+0000 7f9f0b600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f04071c10 0x7f9f040c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.945+0000 7f9f0b600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f041590b0 con 0x7f9f04152400 2024-03-22T13:08:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.945+0000 7f9f0b600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f040724d0 con 0x7f9f04071c10 2024-03-22T13:08:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.945+0000 7f9f0b600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f040721f0 con 0x7f9f0406f8c0 2024-03-22T13:08:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.946+0000 7f9f09600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f04071c10 0x7f9f040c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.946+0000 7f9f03e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9f0406f8c0 0x7f9f040716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.946+0000 7f9f08c00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9f04152400 0x7f9f0415b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35932/0 (socket says 172.21.15.6:35932) 2024-03-22T13:08:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.946+0000 7f9f08c00640 1 -- 172.21.15.6:0/1382489852 learned_addr learned my addr 172.21.15.6:0/1382489852 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:52.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.949+0000 7f9f03e00640 1 -- 172.21.15.6:0/1382489852 >> v2:172.21.15.6:3300/0 conn(0x7f9f04071c10 msgr2=0x7f9f040c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:52.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.949+0000 7f9f03e00640 1 --2- 172.21.15.6:0/1382489852 >> v2:172.21.15.6:3300/0 conn(0x7f9f04071c10 0x7f9f040c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:52.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.949+0000 7f9f03e00640 1 -- 172.21.15.6:0/1382489852 >> v1:172.21.15.92:6789/0 conn(0x7f9f04152400 legacy=0x7f9f0415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:52.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.949+0000 7f9f03e00640 1 -- 172.21.15.6:0/1382489852 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f041582f0 con 0x7f9f0406f8c0 2024-03-22T13:08:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.955+0000 7f9f03e00640 1 --2- 172.21.15.6:0/1382489852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f040716d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9f04155d20 tx=0x7f9ef002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89d3212f61053f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.960+0000 7f9f03400640 1 -- 172.21.15.6:0/1382489852 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0034070 con 0x7f9f0406f8c0 2024-03-22T13:08:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.960+0000 7f9f03400640 1 -- 172.21.15.6:0/1382489852 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef0030610 con 0x7f9f0406f8c0 2024-03-22T13:08:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.960+0000 7f9f03400640 1 -- 172.21.15.6:0/1382489852 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00309e0 con 0x7f9f0406f8c0 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 -- 172.21.15.6:0/1382489852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 msgr2=0x7f9f040716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1382489852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f040716d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9f04155d20 tx=0x7f9ef002b960 comp rx=0 tx=0).stop 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 -- 172.21.15.6:0/1382489852 shutdown_connections 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1382489852 >> v2:172.21.15.6:3300/0 conn(0x7f9f04071c10 0x7f9f040c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1382489852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f040716d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 -- 172.21.15.6:0/1382489852 >> 172.21.15.6:0/1382489852 conn(0x7f9f040cb930 msgr2=0x7f9f040cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 -- 172.21.15.6:0/1382489852 shutdown_connections 2024-03-22T13:08:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.961+0000 7f9f0b600640 1 -- 172.21.15.6:0/1382489852 wait complete. 2024-03-22T13:08:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.963+0000 7f9f0b600640 1 Processor -- start 2024-03-22T13:08:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.963+0000 7f9f0b600640 1 -- start start 2024-03-22T13:08:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.964+0000 7f9f0b600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f041f4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.964+0000 7f9f0b600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f04071c10 0x7f9f040c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.964+0000 7f9f0b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04152400 0x7f9f040c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.964+0000 7f9f0b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04158b90 con 0x7f9f04152400 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.964+0000 7f9f0b600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f041588b0 con 0x7f9f0406f8c0 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.964+0000 7f9f0b600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f041527e0 con 0x7f9f04071c10 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.965+0000 7f9f08c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f041f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.965+0000 7f9f09600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04152400 0x7f9f040c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.965+0000 7f9f08c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f041f4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48470/0 (socket says 172.21.15.6:48470) 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.965+0000 7f9f08c00640 1 -- 172.21.15.6:0/1637642144 learned_addr learned my addr 172.21.15.6:0/1637642144 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.965+0000 7f9f03e00640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f04071c10 0x7f9f040c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:52.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.969+0000 7f9f09600640 1 -- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f04071c10 msgr2=0x7f9f040c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:52.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.969+0000 7f9f09600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f04071c10 0x7f9f040c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:52.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.969+0000 7f9f09600640 1 -- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 msgr2=0x7f9f041f4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:52.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.969+0000 7f9f09600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f041f4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:52.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.969+0000 7f9f09600640 1 -- 172.21.15.6:0/1637642144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f041f05d0 con 0x7f9f04152400 2024-03-22T13:08:52.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.972+0000 7f9f09600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04152400 0x7f9f040c8870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00efc0 tx=0x7f9efc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:52.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.978+0000 7f9f00c00640 1 -- 172.21.15.6:0/1637642144 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc018020 con 0x7f9f04152400 2024-03-22T13:08:52.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.978+0000 7f9f00c00640 1 -- 172.21.15.6:0/1637642144 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc014480 con 0x7f9f04152400 2024-03-22T13:08:52.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.978+0000 7f9f00c00640 1 -- 172.21.15.6:0/1637642144 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc014870 con 0x7f9f04152400 2024-03-22T13:08:52.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.978+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0414e160 con 0x7f9f04152400 2024-03-22T13:08:52.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.978+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f041f0890 con 0x7f9f04152400 2024-03-22T13:08:52.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:52.979+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f040756c0 con 0x7f9f04152400 2024-03-22T13:08:53.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.017+0000 7f9f00c00640 1 -- 172.21.15.6:0/1637642144 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9efc005d10 con 0x7f9f04152400 2024-03-22T13:08:53.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.019+0000 7f9f00c00640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ee406bc20 0x7f9ee406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.019+0000 7f9f08c00640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ee406bc20 0x7f9ee406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.020+0000 7f9f08c00640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ee406bc20 0x7f9ee406e0e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8004e40 tx=0x7f9ef8007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.023+0000 7f9f00c00640 1 -- 172.21.15.6:0/1637642144 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9efc058220 con 0x7f9f04152400 2024-03-22T13:08:53.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.323+0000 7f9f00c00640 1 -- 172.21.15.6:0/1637642144 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9efc022030 con 0x7f9f04152400 2024-03-22T13:08:53.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.588+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f04156120 con 0x7f9f04152400 2024-03-22T13:08:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.642+0000 7f9f00c00640 1 -- 172.21.15.6:0/1637642144 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f9efc010040 con 0x7f9f04152400 2024-03-22T13:08:53.642 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:53.642 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ee406bc20 msgr2=0x7f9ee406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ee406bc20 0x7f9ee406e0e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8004e40 tx=0x7f9ef8007f10 comp rx=0 tx=0).stop 2024-03-22T13:08:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04152400 msgr2=0x7f9f040c8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04152400 0x7f9f040c8870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00efc0 tx=0x7f9efc00c490 comp rx=0 tx=0).stop 2024-03-22T13:08:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 shutdown_connections 2024-03-22T13:08:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04152400 0x7f9f040c8870 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9f04071c10 0x7f9f040c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9ee406bc20 0x7f9ee406e0e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 --2- 172.21.15.6:0/1637642144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f0406f8c0 0x7f9f041f4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.645+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 >> 172.21.15.6:0/1637642144 conn(0x7f9f040cb930 msgr2=0x7f9f04150ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.646+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 shutdown_connections 2024-03-22T13:08:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:53.646+0000 7f9f0b600640 1 -- 172.21.15.6:0/1637642144 wait complete. 2024-03-22T13:08:54.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:54.265 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:55.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:55.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.463+0000 7ff885a00640 1 Processor -- start 2024-03-22T13:08:55.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.464+0000 7ff885a00640 1 -- start start 2024-03-22T13:08:55.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.465+0000 7ff885a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8800cc3b0 0x7ff8800cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:55.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.465+0000 7ff885a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8800ce230 con 0x7ff8800ccea0 2024-03-22T13:08:55.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.465+0000 7ff885a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8800ce510 con 0x7ff8800cf770 2024-03-22T13:08:55.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.465+0000 7ff885a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8800ce7f0 con 0x7ff8800cc3b0 2024-03-22T13:08:55.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.466+0000 7ff87ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8800cc3b0 0x7ff8800cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:55.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.466+0000 7ff87f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff8800ccea0 0x7ff88015ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49456/0 (socket says 172.21.15.6:49456) 2024-03-22T13:08:55.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.466+0000 7ff87f400640 1 -- 172.21.15.6:0/3343168113 learned_addr learned my addr 172.21.15.6:0/3343168113 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:55.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.470+0000 7ff87d600640 1 -- 172.21.15.6:0/3343168113 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728398387 0 0) 0x7ff8800ce230 con 0x7ff8800ccea0 2024-03-22T13:08:55.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.470+0000 7ff87d600640 1 -- 172.21.15.6:0/3343168113 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff864003520 con 0x7ff8800ccea0 2024-03-22T13:08:55.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.471+0000 7ff87ea00640 1 -- 172.21.15.6:0/3343168113 >> v1:172.21.15.92:6789/0 conn(0x7ff8800cf770 legacy=0x7ff8800cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:55.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.471+0000 7ff87ea00640 1 -- 172.21.15.6:0/3343168113 >> v1:172.21.15.6:6789/0 conn(0x7ff8800ccea0 legacy=0x7ff88015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:55.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.471+0000 7ff87ea00640 1 -- 172.21.15.6:0/3343168113 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8800cead0 con 0x7ff8800cc3b0 2024-03-22T13:08:55.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.478+0000 7ff87ea00640 1 --2- 172.21.15.6:0/3343168113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800cc3b0 0x7ff8800cf230 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff8700097a0 tx=0x7ff87002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8456ab271aa4c190 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.484+0000 7ff87d600640 1 -- 172.21.15.6:0/3343168113 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff870035020 con 0x7ff8800cc3b0 2024-03-22T13:08:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.484+0000 7ff87d600640 1 -- 172.21.15.6:0/3343168113 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff870031830 con 0x7ff8800cc3b0 2024-03-22T13:08:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.484+0000 7ff87d600640 1 -- 172.21.15.6:0/3343168113 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff870031c00 con 0x7ff8800cc3b0 2024-03-22T13:08:55.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.484+0000 7ff885a00640 1 -- 172.21.15.6:0/3343168113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800cc3b0 msgr2=0x7ff8800cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:55.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.484+0000 7ff885a00640 1 --2- 172.21.15.6:0/3343168113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800cc3b0 0x7ff8800cf230 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff8700097a0 tx=0x7ff87002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:08:55.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.484+0000 7ff885a00640 1 -- 172.21.15.6:0/3343168113 shutdown_connections 2024-03-22T13:08:55.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.485+0000 7ff885a00640 1 --2- 172.21.15.6:0/3343168113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800cc3b0 0x7ff8800cf230 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.485+0000 7ff885a00640 1 -- 172.21.15.6:0/3343168113 >> 172.21.15.6:0/3343168113 conn(0x7ff8800b16d0 msgr2=0x7ff8800b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.485+0000 7ff885a00640 1 -- 172.21.15.6:0/3343168113 shutdown_connections 2024-03-22T13:08:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.485+0000 7ff885a00640 1 -- 172.21.15.6:0/3343168113 wait complete. 2024-03-22T13:08:55.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.487+0000 7ff885a00640 1 Processor -- start 2024-03-22T13:08:55.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.488+0000 7ff885a00640 1 -- start start 2024-03-22T13:08:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.489+0000 7ff885a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8800cc3b0 0x7ff8801f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.489+0000 7ff885a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800ccea0 0x7ff880154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.489+0000 7ff885a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8800cf770 0x7ff8800b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.489+0000 7ff885a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88015a5d0 con 0x7ff8800cf770 2024-03-22T13:08:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.489+0000 7ff885a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88006dc10 con 0x7ff8800ccea0 2024-03-22T13:08:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.489+0000 7ff885a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff880154bb0 con 0x7ff8800cc3b0 2024-03-22T13:08:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.489+0000 7ff87e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800ccea0 0x7ff880154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.490+0000 7ff87ea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8800cc3b0 0x7ff8801f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.490+0000 7ff87e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800ccea0 0x7ff880154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48484/0 (socket says 172.21.15.6:48484) 2024-03-22T13:08:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.490+0000 7ff87e000640 1 -- 172.21.15.6:0/4050774789 learned_addr learned my addr 172.21.15.6:0/4050774789 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.490+0000 7ff87f400640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8800cf770 0x7ff8800b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.494+0000 7ff87e000640 1 -- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8800cc3b0 msgr2=0x7ff8801f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:55.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.494+0000 7ff87e000640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8800cc3b0 0x7ff8801f0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:55.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.494+0000 7ff87e000640 1 -- 172.21.15.6:0/4050774789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8800cf770 msgr2=0x7ff8800b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:55.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.494+0000 7ff87e000640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8800cf770 0x7ff8800b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:55.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.494+0000 7ff87e000640 1 -- 172.21.15.6:0/4050774789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8801efdb0 con 0x7ff8800ccea0 2024-03-22T13:08:55.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.496+0000 7ff87e000640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800ccea0 0x7ff880154f40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c3f0 tx=0x7ff86c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.500+0000 7ff876a00640 1 -- 172.21.15.6:0/4050774789 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c017070 con 0x7ff8800ccea0 2024-03-22T13:08:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.500+0000 7ff876a00640 1 -- 172.21.15.6:0/4050774789 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c0052f0 con 0x7ff8800ccea0 2024-03-22T13:08:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.500+0000 7ff876a00640 1 -- 172.21.15.6:0/4050774789 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c003830 con 0x7ff8800ccea0 2024-03-22T13:08:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.500+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8800c8dd0 con 0x7ff8800ccea0 2024-03-22T13:08:55.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.500+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8800b30d0 con 0x7ff8800ccea0 2024-03-22T13:08:55.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.502+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8800756c0 con 0x7ff8800ccea0 2024-03-22T13:08:55.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.522+0000 7ff876a00640 1 -- 172.21.15.6:0/4050774789 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff86c01a070 con 0x7ff8800ccea0 2024-03-22T13:08:55.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.523+0000 7ff876a00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff86006bbd0 0x7ff86006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:55.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.524+0000 7ff87ea00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff86006bbd0 0x7ff86006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:55.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.525+0000 7ff87ea00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff86006bbd0 0x7ff86006e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff870036000 tx=0x7ff870008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:55.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.765+0000 7ff876a00640 1 -- 172.21.15.6:0/4050774789 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff86c056470 con 0x7ff8800ccea0 2024-03-22T13:08:55.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:55.784+0000 7ff876a00640 1 -- 172.21.15.6:0/4050774789 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff86c021080 con 0x7ff8800ccea0 2024-03-22T13:08:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.043+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff88006de80 con 0x7ff8800ccea0 2024-03-22T13:08:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.056+0000 7ff876a00640 1 -- 172.21.15.6:0/4050774789 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7ff86c0583d0 con 0x7ff8800ccea0 2024-03-22T13:08:56.056 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:56.056 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:56.057 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff86006bbd0 msgr2=0x7ff86006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff86006bbd0 0x7ff86006e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff870036000 tx=0x7ff870008030 comp rx=0 tx=0).stop 2024-03-22T13:08:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800ccea0 msgr2=0x7ff880154f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800ccea0 0x7ff880154f40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c3f0 tx=0x7ff86c00c8b0 comp rx=0 tx=0).stop 2024-03-22T13:08:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 shutdown_connections 2024-03-22T13:08:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8800cf770 0x7ff8800b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff86006bbd0 0x7ff86006e090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8800ccea0 0x7ff880154f40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 --2- 172.21.15.6:0/4050774789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8800cc3b0 0x7ff8801f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 >> 172.21.15.6:0/4050774789 conn(0x7ff8800b16d0 msgr2=0x7ff880159740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 shutdown_connections 2024-03-22T13:08:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:56.059+0000 7ff885a00640 1 -- 172.21.15.6:0/4050774789 wait complete. 2024-03-22T13:08:56.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:56.681 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:08:57.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:08:57.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.877+0000 7f5fccc00640 1 Processor -- start 2024-03-22T13:08:57.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.877+0000 7f5fccc00640 1 -- start start 2024-03-22T13:08:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.878+0000 7f5fccc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc806ede0 0x7f5fc815abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.878+0000 7f5fccc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc8070890 con 0x7f5fc8154540 2024-03-22T13:08:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.878+0000 7f5fccc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc8070b70 con 0x7f5fc80c91e0 2024-03-22T13:08:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.878+0000 7f5fccc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80c79e0 con 0x7f5fc806ede0 2024-03-22T13:08:57.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.879+0000 7f5fc6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc806ede0 0x7f5fc815abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:57.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.879+0000 7f5fc7e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5fc80c91e0 0x7f5fc8164c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35956/0 (socket says 172.21.15.6:35956) 2024-03-22T13:08:57.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.879+0000 7f5fc7e00640 1 -- 172.21.15.6:0/3281301736 learned_addr learned my addr 172.21.15.6:0/3281301736 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:08:57.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.882+0000 7f5fc6000640 1 -- 172.21.15.6:0/3281301736 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622476891 0 0) 0x7f5fc8070b70 con 0x7f5fc80c91e0 2024-03-22T13:08:57.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.882+0000 7f5fc6000640 1 -- 172.21.15.6:0/3281301736 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac003520 con 0x7f5fc80c91e0 2024-03-22T13:08:57.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.883+0000 7f5fc6a00640 1 -- 172.21.15.6:0/3281301736 >> v1:172.21.15.92:6789/0 conn(0x7f5fc80c91e0 legacy=0x7f5fc8164c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:57.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.883+0000 7f5fc6a00640 1 -- 172.21.15.6:0/3281301736 >> v1:172.21.15.6:6789/0 conn(0x7f5fc8154540 legacy=0x7f5fc815e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:08:57.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.883+0000 7f5fc6a00640 1 -- 172.21.15.6:0/3281301736 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc815a3c0 con 0x7f5fc806ede0 2024-03-22T13:08:57.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.886+0000 7f5fc6a00640 1 --2- 172.21.15.6:0/3281301736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc806ede0 0x7f5fc815abc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40097d0 tx=0x7f5fb402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93e070cf989d2db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:57.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.892+0000 7f5fc6000640 1 -- 172.21.15.6:0/3281301736 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4035020 con 0x7f5fc806ede0 2024-03-22T13:08:57.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.892+0000 7f5fc6000640 1 -- 172.21.15.6:0/3281301736 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4031890 con 0x7f5fc806ede0 2024-03-22T13:08:57.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.892+0000 7f5fc6000640 1 -- 172.21.15.6:0/3281301736 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4031c60 con 0x7f5fc806ede0 2024-03-22T13:08:57.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.892+0000 7f5fccc00640 1 -- 172.21.15.6:0/3281301736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc806ede0 msgr2=0x7f5fc815abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:57.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.892+0000 7f5fccc00640 1 --2- 172.21.15.6:0/3281301736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc806ede0 0x7f5fc815abc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40097d0 tx=0x7f5fb402b930 comp rx=0 tx=0).stop 2024-03-22T13:08:57.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.893+0000 7f5fccc00640 1 -- 172.21.15.6:0/3281301736 shutdown_connections 2024-03-22T13:08:57.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.893+0000 7f5fccc00640 1 --2- 172.21.15.6:0/3281301736 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc806ede0 0x7f5fc815abc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:57.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.893+0000 7f5fccc00640 1 -- 172.21.15.6:0/3281301736 >> 172.21.15.6:0/3281301736 conn(0x7f5fc806c190 msgr2=0x7f5fc806c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:57.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.893+0000 7f5fccc00640 1 -- 172.21.15.6:0/3281301736 shutdown_connections 2024-03-22T13:08:57.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.893+0000 7f5fccc00640 1 -- 172.21.15.6:0/3281301736 wait complete. 2024-03-22T13:08:57.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.895+0000 7f5fccc00640 1 Processor -- start 2024-03-22T13:08:57.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.895+0000 7f5fccc00640 1 -- start start 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.896+0000 7f5fccc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc806ede0 0x7f5fc8150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.896+0000 7f5fccc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc80c91e0 0x7f5fc8155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.896+0000 7f5fccc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc8154540 0x7f5fc8149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.896+0000 7f5fccc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8070e50 con 0x7f5fc806ede0 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.896+0000 7f5fccc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc815a730 con 0x7f5fc8154540 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.896+0000 7f5fccc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc80c7bb0 con 0x7f5fc80c91e0 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.897+0000 7f5fc7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc806ede0 0x7f5fc8150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.897+0000 7f5fc6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc80c91e0 0x7f5fc8155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.897+0000 7f5fc7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc8154540 0x7f5fc8149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.897+0000 7f5fc7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc806ede0 0x7f5fc8150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39128/0 (socket says 172.21.15.6:39128) 2024-03-22T13:08:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.897+0000 7f5fc7400640 1 -- 172.21.15.6:0/916933798 learned_addr learned my addr 172.21.15.6:0/916933798 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:08:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.901+0000 7f5fc7400640 1 -- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc80c91e0 msgr2=0x7f5fc8155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.901+0000 7f5fc7400640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc80c91e0 0x7f5fc8155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.901+0000 7f5fc7400640 1 -- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc8154540 msgr2=0x7f5fc8149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.901+0000 7f5fc7400640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc8154540 0x7f5fc8149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:57.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.901+0000 7f5fc7400640 1 -- 172.21.15.6:0/916933798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc8148ac0 con 0x7f5fc806ede0 2024-03-22T13:08:57.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.904+0000 7f5fc7400640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc806ede0 0x7f5fc8150920 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c470 tx=0x7f5fb800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:57.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.906+0000 7f5fbf400640 1 -- 172.21.15.6:0/916933798 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8017070 con 0x7f5fc806ede0 2024-03-22T13:08:57.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.907+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc8148da0 con 0x7f5fc806ede0 2024-03-22T13:08:57.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.907+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc8149360 con 0x7f5fc806ede0 2024-03-22T13:08:57.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.908+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc8075650 con 0x7f5fc806ede0 2024-03-22T13:08:57.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.909+0000 7f5fbf400640 1 -- 172.21.15.6:0/916933798 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb80035f0 con 0x7f5fc806ede0 2024-03-22T13:08:57.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:57.909+0000 7f5fbf400640 1 -- 172.21.15.6:0/916933798 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8021060 con 0x7f5fc806ede0 2024-03-22T13:08:58.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.232+0000 7f5fbf400640 1 -- 172.21.15.6:0/916933798 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5fb801a070 con 0x7f5fc806ede0 2024-03-22T13:08:58.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.234+0000 7f5fbf400640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fa806bc00 0x7f5fa806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:08:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.234+0000 7f5fc6a00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fa806bc00 0x7f5fa806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:08:58.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.235+0000 7f5fc6a00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fa806bc00 0x7f5fa806e0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4036000 tx=0x7f5fb403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:08:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.275+0000 7f5fbf400640 1 -- 172.21.15.6:0/916933798 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5fb8056ad0 con 0x7f5fc806ede0 2024-03-22T13:08:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.276+0000 7f5fbf400640 1 -- 172.21.15.6:0/916933798 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5fb80010c0 con 0x7f5fc806ede0 2024-03-22T13:08:58.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.545+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fc8157cd0 con 0x7f5fc806ede0 2024-03-22T13:08:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.568+0000 7f5fbf400640 1 -- 172.21.15.6:0/916933798 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f5fb8058a30 con 0x7f5fc806ede0 2024-03-22T13:08:58.568 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:08:58.568 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:08:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.571+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fa806bc00 msgr2=0x7f5fa806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.571+0000 7f5fccc00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fa806bc00 0x7f5fa806e0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4036000 tx=0x7f5fb403c040 comp rx=0 tx=0).stop 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc806ede0 msgr2=0x7f5fc8150920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc806ede0 0x7f5fc8150920 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c470 tx=0x7f5fb800c930 comp rx=0 tx=0).stop 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 shutdown_connections 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fa806bc00 0x7f5fa806e0c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc8154540 0x7f5fc8149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc80c91e0 0x7f5fc8155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 --2- 172.21.15.6:0/916933798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fc806ede0 0x7f5fc8150920 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 >> 172.21.15.6:0/916933798 conn(0x7f5fc806c190 msgr2=0x7f5fc80c5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 shutdown_connections 2024-03-22T13:08:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:08:58.572+0000 7f5fccc00640 1 -- 172.21.15.6:0/916933798 wait complete. 2024-03-22T13:08:59.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:08:59.195 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:00.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:00.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.397+0000 7ff03fe00640 1 Processor -- start 2024-03-22T13:09:00.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.397+0000 7ff03fe00640 1 -- start start 2024-03-22T13:09:00.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.398+0000 7ff03fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff04006e2e0 0x7ff0400c4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:00.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.399+0000 7ff03fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff0400c4970 0x7ff0400c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:00.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.399+0000 7ff03fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04006f190 con 0x7ff04014da20 2024-03-22T13:09:00.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.399+0000 7ff03fe00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400c3f50 con 0x7ff0400c4970 2024-03-22T13:09:00.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.399+0000 7ff03fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04006f560 con 0x7ff04006e2e0 2024-03-22T13:09:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.399+0000 7ff03f400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff0400c4970 0x7ff0400c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:00.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.400+0000 7ff03e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff04006e2e0 0x7ff0400c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:00.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.400+0000 7ff03e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff04006e2e0 0x7ff0400c4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:52552/0 (socket says 172.21.15.6:52552) 2024-03-22T13:09:00.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.400+0000 7ff03e000640 1 -- 172.21.15.6:0/817720355 learned_addr learned my addr 172.21.15.6:0/817720355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:00.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.403+0000 7ff03e000640 1 -- 172.21.15.6:0/817720355 >> v2:172.21.15.92:3300/0 conn(0x7ff0400c4970 msgr2=0x7ff0400c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:00.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.403+0000 7ff03e000640 1 --2- 172.21.15.6:0/817720355 >> v2:172.21.15.92:3300/0 conn(0x7ff0400c4970 0x7ff0400c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:00.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.403+0000 7ff03e000640 1 -- 172.21.15.6:0/817720355 >> v1:172.21.15.6:6789/0 conn(0x7ff04014da20 legacy=0x7ff04006ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:00.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.403+0000 7ff03e000640 1 -- 172.21.15.6:0/817720355 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400c62c0 con 0x7ff04006e2e0 2024-03-22T13:09:00.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.405+0000 7ff03e000640 1 --2- 172.21.15.6:0/817720355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff04006e2e0 0x7ff0400c4430 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d19a83f82835752d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.409+0000 7ff03d600640 1 -- 172.21.15.6:0/817720355 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c034070 con 0x7ff04006e2e0 2024-03-22T13:09:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03d600640 1 -- 172.21.15.6:0/817720355 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c030840 con 0x7ff04006e2e0 2024-03-22T13:09:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03d600640 1 -- 172.21.15.6:0/817720355 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c030c10 con 0x7ff04006e2e0 2024-03-22T13:09:00.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03fe00640 1 -- 172.21.15.6:0/817720355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff04006e2e0 msgr2=0x7ff0400c4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:00.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03fe00640 1 --2- 172.21.15.6:0/817720355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff04006e2e0 0x7ff0400c4430 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c02b980 comp rx=0 tx=0).stop 2024-03-22T13:09:00.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03fe00640 1 -- 172.21.15.6:0/817720355 shutdown_connections 2024-03-22T13:09:00.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03fe00640 1 --2- 172.21.15.6:0/817720355 >> v2:172.21.15.92:3300/0 conn(0x7ff0400c4970 0x7ff0400c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:00.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03fe00640 1 --2- 172.21.15.6:0/817720355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff04006e2e0 0x7ff0400c4430 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:00.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.410+0000 7ff03fe00640 1 -- 172.21.15.6:0/817720355 >> 172.21.15.6:0/817720355 conn(0x7ff04006c190 msgr2=0x7ff04006c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:00.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.411+0000 7ff03fe00640 1 -- 172.21.15.6:0/817720355 shutdown_connections 2024-03-22T13:09:00.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.411+0000 7ff03fe00640 1 -- 172.21.15.6:0/817720355 wait complete. 2024-03-22T13:09:00.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.413+0000 7ff03fe00640 1 Processor -- start 2024-03-22T13:09:00.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.413+0000 7ff03fe00640 1 -- start start 2024-03-22T13:09:00.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.415+0000 7ff03fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff04006e2e0 0x7ff0401f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:00.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.415+0000 7ff03fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0400c4970 0x7ff04014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:00.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.415+0000 7ff03fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff04014da20 0x7ff04014f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:00.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.415+0000 7ff03fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400c6b60 con 0x7ff04014da20 2024-03-22T13:09:00.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.415+0000 7ff03fe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400c6880 con 0x7ff04006e2e0 2024-03-22T13:09:00.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.415+0000 7ff03fe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400c6e40 con 0x7ff0400c4970 2024-03-22T13:09:00.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.415+0000 7ff03ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff04006e2e0 0x7ff0401f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.416+0000 7ff03e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0400c4970 0x7ff04014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.416+0000 7ff03e000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0400c4970 0x7ff04014ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:52566/0 (socket says 172.21.15.6:52566) 2024-03-22T13:09:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.416+0000 7ff03e000640 1 -- 172.21.15.6:0/1560656768 learned_addr learned my addr 172.21.15.6:0/1560656768 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.420+0000 7ff03e000640 1 -- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff04006e2e0 msgr2=0x7ff0401f1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.420+0000 7ff03e000640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff04006e2e0 0x7ff0401f1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.420+0000 7ff03e000640 1 -- 172.21.15.6:0/1560656768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff04014da20 msgr2=0x7ff04014f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.420+0000 7ff03e000640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff04014da20 0x7ff04014f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.420+0000 7ff03e000640 1 -- 172.21.15.6:0/1560656768 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0401f01e0 con 0x7ff0400c4970 2024-03-22T13:09:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.422+0000 7ff03e000640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0400c4970 0x7ff04014ef60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.426+0000 7ff036a00640 1 -- 172.21.15.6:0/1560656768 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c034040 con 0x7ff0400c4970 2024-03-22T13:09:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.426+0000 7ff036a00640 1 -- 172.21.15.6:0/1560656768 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c039d10 con 0x7ff0400c4970 2024-03-22T13:09:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.426+0000 7ff036a00640 1 -- 172.21.15.6:0/1560656768 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c043630 con 0x7ff0400c4970 2024-03-22T13:09:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.426+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0401511d0 con 0x7ff0400c4970 2024-03-22T13:09:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.426+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0401f0760 con 0x7ff0400c4970 2024-03-22T13:09:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.428+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff040075650 con 0x7ff0400c4970 2024-03-22T13:09:00.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.466+0000 7ff036a00640 1 -- 172.21.15.6:0/1560656768 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff02c04a060 con 0x7ff0400c4970 2024-03-22T13:09:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.467+0000 7ff036a00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff02006bbd0 0x7ff02006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:00.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.467+0000 7ff03ea00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff02006bbd0 0x7ff02006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:00.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.468+0000 7ff03ea00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff02006bbd0 0x7ff02006e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff0300017f0 tx=0x7ff030009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.727+0000 7ff036a00640 1 -- 172.21.15.6:0/1560656768 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff02c079210 con 0x7ff0400c4970 2024-03-22T13:09:00.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:00.793+0000 7ff036a00640 1 -- 172.21.15.6:0/1560656768 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff02c03e070 con 0x7ff0400c4970 2024-03-22T13:09:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.044+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0400c4430 con 0x7ff0400c4970 2024-03-22T13:09:01.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.060+0000 7ff036a00640 1 -- 172.21.15.6:0/1560656768 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7ff04006f190 con 0x7ff0400c4970 2024-03-22T13:09:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:01.060 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:01.060 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff02006bbd0 msgr2=0x7ff02006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff02006bbd0 0x7ff02006e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff0300017f0 tx=0x7ff030009040 comp rx=0 tx=0).stop 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0400c4970 msgr2=0x7ff04014ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0400c4970 0x7ff04014ef60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009870 tx=0x7ff02c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 shutdown_connections 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff04014da20 0x7ff04014f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0400c4970 0x7ff04014ef60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:01.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff02006bbd0 0x7ff02006e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:01.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 --2- 172.21.15.6:0/1560656768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff04006e2e0 0x7ff0401f1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:01.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 >> 172.21.15.6:0/1560656768 conn(0x7ff04006c190 msgr2=0x7ff0400c8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:01.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 shutdown_connections 2024-03-22T13:09:01.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:01.063+0000 7ff03fe00640 1 -- 172.21.15.6:0/1560656768 wait complete. 2024-03-22T13:09:01.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:01.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:02.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:02.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:09:02.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:09:02.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.888+0000 7fab3d400640 1 Processor -- start 2024-03-22T13:09:02.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.888+0000 7fab3d400640 1 -- start start 2024-03-22T13:09:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.890+0000 7fab3d400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fab3806f8c0 0x7fab380716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.890+0000 7fab3d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab381590b0 con 0x7fab38152400 2024-03-22T13:09:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.890+0000 7fab3d400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab380721f0 con 0x7fab38071c10 2024-03-22T13:09:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.890+0000 7fab3d400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab380724d0 con 0x7fab3806f8c0 2024-03-22T13:09:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.891+0000 7fab35e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fab3806f8c0 0x7fab380716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.891+0000 7fab36800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fab38152400 0x7fab3815b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49470/0 (socket says 172.21.15.6:49470) 2024-03-22T13:09:02.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.891+0000 7fab36800640 1 -- 172.21.15.6:0/917391948 learned_addr learned my addr 172.21.15.6:0/917391948 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:02.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.895+0000 7fab35e00640 1 -- 172.21.15.6:0/917391948 >> v1:172.21.15.92:6789/0 conn(0x7fab38071c10 legacy=0x7fab380c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:02.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.895+0000 7fab35e00640 1 -- 172.21.15.6:0/917391948 >> v1:172.21.15.6:6789/0 conn(0x7fab38152400 legacy=0x7fab3815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:02.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.895+0000 7fab35e00640 1 -- 172.21.15.6:0/917391948 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab381582f0 con 0x7fab3806f8c0 2024-03-22T13:09:02.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.898+0000 7fab35e00640 1 --2- 172.21.15.6:0/917391948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab3806f8c0 0x7fab380716d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fab240097a0 tx=0x7fab2402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9456781120ac01d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.902+0000 7fab35400640 1 -- 172.21.15.6:0/917391948 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2402f810 con 0x7fab3806f8c0 2024-03-22T13:09:02.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.905+0000 7fab35400640 1 -- 172.21.15.6:0/917391948 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2402f9b0 con 0x7fab3806f8c0 2024-03-22T13:09:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.905+0000 7fab35400640 1 -- 172.21.15.6:0/917391948 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2402fd80 con 0x7fab3806f8c0 2024-03-22T13:09:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.905+0000 7fab3d400640 1 -- 172.21.15.6:0/917391948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab3806f8c0 msgr2=0x7fab380716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.905+0000 7fab3d400640 1 --2- 172.21.15.6:0/917391948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab3806f8c0 0x7fab380716d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fab240097a0 tx=0x7fab2402b9f0 comp rx=0 tx=0).stop 2024-03-22T13:09:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.906+0000 7fab3d400640 1 -- 172.21.15.6:0/917391948 shutdown_connections 2024-03-22T13:09:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.906+0000 7fab3d400640 1 --2- 172.21.15.6:0/917391948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab3806f8c0 0x7fab380716d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.906+0000 7fab3d400640 1 -- 172.21.15.6:0/917391948 >> 172.21.15.6:0/917391948 conn(0x7fab380cb930 msgr2=0x7fab380cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.906+0000 7fab3d400640 1 -- 172.21.15.6:0/917391948 shutdown_connections 2024-03-22T13:09:02.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.906+0000 7fab3d400640 1 -- 172.21.15.6:0/917391948 wait complete. 2024-03-22T13:09:02.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.908+0000 7fab3d400640 1 Processor -- start 2024-03-22T13:09:02.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.909+0000 7fab3d400640 1 -- start start 2024-03-22T13:09:02.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.910+0000 7fab3d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab3806f8c0 0x7fab381ec730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:02.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.910+0000 7fab3d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab38071c10 0x7fab3813e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:02.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.910+0000 7fab3d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab38152400 0x7fab38138390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:02.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.910+0000 7fab3d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab38158b90 con 0x7fab38071c10 2024-03-22T13:09:02.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.910+0000 7fab3d400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab381588b0 con 0x7fab38152400 2024-03-22T13:09:02.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.910+0000 7fab3d400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab381527e0 con 0x7fab3806f8c0 2024-03-22T13:09:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.911+0000 7fab35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab38071c10 0x7fab3813e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.911+0000 7fab36800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab3806f8c0 0x7fab381ec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.911+0000 7fab37200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab38152400 0x7fab38138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.911+0000 7fab35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab38071c10 0x7fab3813e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50288/0 (socket says 172.21.15.6:50288) 2024-03-22T13:09:02.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.911+0000 7fab35e00640 1 -- 172.21.15.6:0/4158513029 learned_addr learned my addr 172.21.15.6:0/4158513029 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:02.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.915+0000 7fab36800640 1 -- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab38152400 msgr2=0x7fab38138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:02.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.915+0000 7fab36800640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab38152400 0x7fab38138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:02.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.915+0000 7fab36800640 1 -- 172.21.15.6:0/4158513029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab38071c10 msgr2=0x7fab3813e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:02.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.915+0000 7fab36800640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab38071c10 0x7fab3813e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:02.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.915+0000 7fab36800640 1 -- 172.21.15.6:0/4158513029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab381ef4c0 con 0x7fab3806f8c0 2024-03-22T13:09:02.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.917+0000 7fab36800640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab3806f8c0 0x7fab381ec730 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fab2800c470 tx=0x7fab2800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:02.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.922+0000 7fab2ea00640 1 -- 172.21.15.6:0/4158513029 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28017070 con 0x7fab3806f8c0 2024-03-22T13:09:02.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.922+0000 7fab2ea00640 1 -- 172.21.15.6:0/4158513029 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab28005ce0 con 0x7fab3806f8c0 2024-03-22T13:09:02.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.922+0000 7fab2ea00640 1 -- 172.21.15.6:0/4158513029 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280072d0 con 0x7fab3806f8c0 2024-03-22T13:09:02.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.922+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab38156150 con 0x7fab3806f8c0 2024-03-22T13:09:02.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.922+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab38156710 con 0x7fab3806f8c0 2024-03-22T13:09:02.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:02.923+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab38155f70 con 0x7fab3806f8c0 2024-03-22T13:09:03.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.225+0000 7fab2ea00640 1 -- 172.21.15.6:0/4158513029 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fab2801a070 con 0x7fab3806f8c0 2024-03-22T13:09:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.227+0000 7fab2ea00640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab1806bc20 0x7fab1806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.227+0000 7fab2ea00640 1 -- 172.21.15.6:0/4158513029 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fab28056a00 con 0x7fab3806f8c0 2024-03-22T13:09:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.227+0000 7fab35e00640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab1806bc20 0x7fab1806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:03.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.229+0000 7fab35e00640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab1806bc20 0x7fab1806e0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fab3806fdf0 tx=0x7fab24008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.239+0000 7fab2ea00640 1 -- 172.21.15.6:0/4158513029 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fab38155f70 con 0x7fab3806f8c0 2024-03-22T13:09:03.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.545+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab380756c0 con 0x7fab3806f8c0 2024-03-22T13:09:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.569+0000 7fab2ea00640 1 -- 172.21.15.6:0/4158513029 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fab28058960 con 0x7fab3806f8c0 2024-03-22T13:09:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:03.569 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab1806bc20 msgr2=0x7fab1806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab1806bc20 0x7fab1806e0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fab3806fdf0 tx=0x7fab24008030 comp rx=0 tx=0).stop 2024-03-22T13:09:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab3806f8c0 msgr2=0x7fab381ec730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab3806f8c0 0x7fab381ec730 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fab2800c470 tx=0x7fab2800c930 comp rx=0 tx=0).stop 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 shutdown_connections 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fab1806bc20 0x7fab1806e0e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab38152400 0x7fab38138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab38071c10 0x7fab3813e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 --2- 172.21.15.6:0/4158513029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab3806f8c0 0x7fab381ec730 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 >> 172.21.15.6:0/4158513029 conn(0x7fab380cb930 msgr2=0x7fab380744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 shutdown_connections 2024-03-22T13:09:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:03.572+0000 7fab3d400640 1 -- 172.21.15.6:0/4158513029 wait complete. 2024-03-22T13:09:04.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:04.194 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:05.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:05.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.394+0000 7f68a3200640 1 Processor -- start 2024-03-22T13:09:05.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.394+0000 7f68a3200640 1 -- start start 2024-03-22T13:09:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.395+0000 7f68a3200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a40ce8a0 0x7f68a406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.396+0000 7f68a3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a415b260 con 0x7f68a415ae30 2024-03-22T13:09:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.396+0000 7f68a3200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a40cee00 con 0x7f68a40cc380 2024-03-22T13:09:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.396+0000 7f68a3200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a40cf0c0 con 0x7f68a40ce8a0 2024-03-22T13:09:05.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.396+0000 7f68a2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f68a415ae30 0x7f68a4164b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56444/0 (socket says 172.21.15.6:56444) 2024-03-22T13:09:05.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.396+0000 7f68a2800640 1 -- 172.21.15.6:0/1346395376 learned_addr learned my addr 172.21.15.6:0/1346395376 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:05.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.397+0000 7f68a1400640 1 --2- 172.21.15.6:0/1346395376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a40ce8a0 0x7f68a406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.400+0000 7f68a0a00640 1 -- 172.21.15.6:0/1346395376 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244701566 0 0) 0x7f68a40cee00 con 0x7f68a40cc380 2024-03-22T13:09:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.400+0000 7f68a0a00640 1 -- 172.21.15.6:0/1346395376 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6888003520 con 0x7f68a40cc380 2024-03-22T13:09:05.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.400+0000 7f68a0a00640 1 -- 172.21.15.6:0/1346395376 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314131549 0 0) 0x7f689c001490 con 0x7f68a415ae30 2024-03-22T13:09:05.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.400+0000 7f68a0a00640 1 -- 172.21.15.6:0/1346395376 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a40cee00 con 0x7f68a415ae30 2024-03-22T13:09:05.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.400+0000 7f68a1400640 1 -- 172.21.15.6:0/1346395376 >> v1:172.21.15.92:6789/0 conn(0x7f68a40cc380 legacy=0x7f68a40cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:05.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.400+0000 7f68a1400640 1 -- 172.21.15.6:0/1346395376 >> v1:172.21.15.6:6789/0 conn(0x7f68a415ae30 legacy=0x7f68a4164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:05.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.400+0000 7f68a1400640 1 -- 172.21.15.6:0/1346395376 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a40cf3a0 con 0x7f68a40ce8a0 2024-03-22T13:09:05.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.403+0000 7f68a1400640 1 --2- 172.21.15.6:0/1346395376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a40ce8a0 0x7f68a406c230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f68900097d0 tx=0x7f689002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2248b77b9ee9e2aa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.409+0000 7f68a0a00640 1 -- 172.21.15.6:0/1346395376 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890035020 con 0x7f68a40ce8a0 2024-03-22T13:09:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.409+0000 7f68a0a00640 1 -- 172.21.15.6:0/1346395376 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6890031890 con 0x7f68a40ce8a0 2024-03-22T13:09:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.409+0000 7f68a0a00640 1 -- 172.21.15.6:0/1346395376 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890031c60 con 0x7f68a40ce8a0 2024-03-22T13:09:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.410+0000 7f68a3200640 1 -- 172.21.15.6:0/1346395376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a40ce8a0 msgr2=0x7f68a406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.410+0000 7f68a3200640 1 --2- 172.21.15.6:0/1346395376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a40ce8a0 0x7f68a406c230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f68900097d0 tx=0x7f689002b930 comp rx=0 tx=0).stop 2024-03-22T13:09:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.410+0000 7f68a3200640 1 -- 172.21.15.6:0/1346395376 shutdown_connections 2024-03-22T13:09:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.410+0000 7f68a3200640 1 --2- 172.21.15.6:0/1346395376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a40ce8a0 0x7f68a406c230 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.410+0000 7f68a3200640 1 -- 172.21.15.6:0/1346395376 >> 172.21.15.6:0/1346395376 conn(0x7f68a40b1480 msgr2=0x7f68a40b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.410+0000 7f68a3200640 1 -- 172.21.15.6:0/1346395376 shutdown_connections 2024-03-22T13:09:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.410+0000 7f68a3200640 1 -- 172.21.15.6:0/1346395376 wait complete. 2024-03-22T13:09:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.412+0000 7f68a3200640 1 Processor -- start 2024-03-22T13:09:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.412+0000 7f68a3200640 1 -- start start 2024-03-22T13:09:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.413+0000 7f68a3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68a40cc380 0x7f68a415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.413+0000 7f68a3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a40ce8a0 0x7f68a4154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.413+0000 7f68a3200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a415ae30 0x7f68a4150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.413+0000 7f68a3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a415a910 con 0x7f68a40cc380 2024-03-22T13:09:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.413+0000 7f68a3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a415a630 con 0x7f68a40ce8a0 2024-03-22T13:09:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.413+0000 7f68a3200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a41549d0 con 0x7f68a415ae30 2024-03-22T13:09:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.414+0000 7f68a1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a40ce8a0 0x7f68a4154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.414+0000 7f68a1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68a40cc380 0x7f68a415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.414+0000 7f68a2800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a415ae30 0x7f68a4150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.414+0000 7f68a1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a40ce8a0 0x7f68a4154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:34374/0 (socket says 172.21.15.6:34374) 2024-03-22T13:09:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.414+0000 7f68a1400640 1 -- 172.21.15.6:0/1229916211 learned_addr learned my addr 172.21.15.6:0/1229916211 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.418+0000 7f68a1400640 1 -- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a415ae30 msgr2=0x7f68a4150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.418+0000 7f68a1400640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a415ae30 0x7f68a4150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.418+0000 7f68a1400640 1 -- 172.21.15.6:0/1229916211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68a40cc380 msgr2=0x7f68a415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.418+0000 7f68a1400640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68a40cc380 0x7f68a415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.418+0000 7f68a1400640 1 -- 172.21.15.6:0/1229916211 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a41e6210 con 0x7f68a40ce8a0 2024-03-22T13:09:05.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.420+0000 7f68a1400640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a40ce8a0 0x7f68a4154bb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f68900098f0 tx=0x7f68900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:05.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.424+0000 7f689a000640 1 -- 172.21.15.6:0/1229916211 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6890035040 con 0x7f68a40ce8a0 2024-03-22T13:09:05.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.424+0000 7f689a000640 1 -- 172.21.15.6:0/1229916211 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6890030ca0 con 0x7f68a40ce8a0 2024-03-22T13:09:05.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.424+0000 7f689a000640 1 -- 172.21.15.6:0/1229916211 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68900466c0 con 0x7f68a40ce8a0 2024-03-22T13:09:05.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.424+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a415b5e0 con 0x7f68a40ce8a0 2024-03-22T13:09:05.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.425+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a415c890 con 0x7f68a40ce8a0 2024-03-22T13:09:05.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.426+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a4075650 con 0x7f68a40ce8a0 2024-03-22T13:09:05.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.447+0000 7f689a000640 1 -- 172.21.15.6:0/1229916211 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6890038020 con 0x7f68a40ce8a0 2024-03-22T13:09:05.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.448+0000 7f689a000640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f688406bc20 0x7f688406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:05.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.449+0000 7f68a1e00640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f688406bc20 0x7f688406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:05.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.449+0000 7f68a1e00640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f688406bc20 0x7f688406e0e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f68940011e0 tx=0x7f6894009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:05.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.688+0000 7f689a000640 1 -- 172.21.15.6:0/1229916211 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f68900797d0 con 0x7f68a40ce8a0 2024-03-22T13:09:05.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.699+0000 7f689a000640 1 -- 172.21.15.6:0/1229916211 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f68a41e6210 con 0x7f68a40ce8a0 2024-03-22T13:09:05.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.963+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68a406de60 con 0x7f68a40ce8a0 2024-03-22T13:09:05.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.986+0000 7f689a000640 1 -- 172.21.15.6:0/1229916211 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f689002d040 con 0x7f68a40ce8a0 2024-03-22T13:09:05.986 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:05.986 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:05.987 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f688406bc20 msgr2=0x7f688406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f688406bc20 0x7f688406e0e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f68940011e0 tx=0x7f6894009040 comp rx=0 tx=0).stop 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a40ce8a0 msgr2=0x7f68a4154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a40ce8a0 0x7f68a4154bb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f68900098f0 tx=0x7f68900046c0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 shutdown_connections 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68a415ae30 0x7f68a4150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f688406bc20 0x7f688406e0e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68a40ce8a0 0x7f68a4154bb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 --2- 172.21.15.6:0/1229916211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68a40cc380 0x7f68a415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:05.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 >> 172.21.15.6:0/1229916211 conn(0x7f68a40b1480 msgr2=0x7f68a4158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:05.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 shutdown_connections 2024-03-22T13:09:05.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:05.989+0000 7f68a3200640 1 -- 172.21.15.6:0/1229916211 wait complete. 2024-03-22T13:09:06.609 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:06.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:07.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:07.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.812+0000 7f48ad800640 1 Processor -- start 2024-03-22T13:09:07.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.812+0000 7f48ad800640 1 -- start start 2024-03-22T13:09:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.813+0000 7f48ad800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48a80cc640 0x7f48a806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.813+0000 7f48ad800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f48a80cca20 0x7f48a80ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.813+0000 7f48ad800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.813+0000 7f48ad800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a806d330 con 0x7f48a80cc640 2024-03-22T13:09:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.813+0000 7f48ad800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a806dd00 con 0x7f48a80cca20 2024-03-22T13:09:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.813+0000 7f48ad800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a806c440 con 0x7f48a815b7a0 2024-03-22T13:09:07.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.827+0000 7f48a7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48a80cc640 0x7f48a806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:07.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.827+0000 7f48a7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48a80cc640 0x7f48a806d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50304/0 (socket says 172.21.15.6:50304) 2024-03-22T13:09:07.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.827+0000 7f48a7e00640 1 -- 172.21.15.6:0/3422438160 learned_addr learned my addr 172.21.15.6:0/3422438160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:07.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.830+0000 7f48a7e00640 1 -- 172.21.15.6:0/3422438160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 msgr2=0x7f48a815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:07.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.830+0000 7f48a7e00640 1 --2- 172.21.15.6:0/3422438160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.830+0000 7f48a7e00640 1 -- 172.21.15.6:0/3422438160 >> v2:172.21.15.92:3300/0 conn(0x7f48a80cca20 msgr2=0x7f48a80ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:07.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.830+0000 7f48a7e00640 1 --2- 172.21.15.6:0/3422438160 >> v2:172.21.15.92:3300/0 conn(0x7f48a80cca20 0x7f48a80ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.830+0000 7f48a7e00640 1 -- 172.21.15.6:0/3422438160 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a806c720 con 0x7f48a80cc640 2024-03-22T13:09:07.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.833+0000 7f48a7e00640 1 --2- 172.21.15.6:0/3422438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 0x7f48a806d7c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48a80cc2f0 tx=0x7f489802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ab1bb7d2a689a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.863+0000 7f48a6a00640 1 -- 172.21.15.6:0/3422438160 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898034070 con 0x7f48a80cc640 2024-03-22T13:09:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.863+0000 7f48a6a00640 1 -- 172.21.15.6:0/3422438160 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4898030880 con 0x7f48a80cc640 2024-03-22T13:09:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.863+0000 7f48a6a00640 1 -- 172.21.15.6:0/3422438160 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898030c50 con 0x7f48a80cc640 2024-03-22T13:09:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 -- 172.21.15.6:0/3422438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 msgr2=0x7f48a806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 --2- 172.21.15.6:0/3422438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 0x7f48a806d7c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48a80cc2f0 tx=0x7f489802b9c0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 -- 172.21.15.6:0/3422438160 shutdown_connections 2024-03-22T13:09:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 --2- 172.21.15.6:0/3422438160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a815d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 --2- 172.21.15.6:0/3422438160 >> v2:172.21.15.92:3300/0 conn(0x7f48a80cca20 0x7f48a80ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 --2- 172.21.15.6:0/3422438160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 0x7f48a806d7c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 -- 172.21.15.6:0/3422438160 >> 172.21.15.6:0/3422438160 conn(0x7f48a80cf9c0 msgr2=0x7f48a80cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 -- 172.21.15.6:0/3422438160 shutdown_connections 2024-03-22T13:09:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.864+0000 7f48ad800640 1 -- 172.21.15.6:0/3422438160 wait complete. 2024-03-22T13:09:07.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.866+0000 7f48ad800640 1 Processor -- start 2024-03-22T13:09:07.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.867+0000 7f48ad800640 1 -- start start 2024-03-22T13:09:07.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.867+0000 7f48ad800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 0x7f48a80b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:07.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.867+0000 7f48ad800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48a80cca20 0x7f48a80cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:07.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.868+0000 7f48ad800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a81f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:07.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.868+0000 7f48ad800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a806cfc0 con 0x7f48a80cc640 2024-03-22T13:09:07.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.868+0000 7f48ad800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a806cce0 con 0x7f48a80cca20 2024-03-22T13:09:07.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.868+0000 7f48ad800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a815b000 con 0x7f48a815b7a0 2024-03-22T13:09:07.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.868+0000 7f48ace00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a81f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:07.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.868+0000 7f48a7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48a80cca20 0x7f48a80cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:07.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.869+0000 7f48ace00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a81f0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:42974/0 (socket says 172.21.15.6:42974) 2024-03-22T13:09:07.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.869+0000 7f48ace00640 1 -- 172.21.15.6:0/1937405 learned_addr learned my addr 172.21.15.6:0/1937405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:07.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.873+0000 7f48ace00640 1 -- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48a80cca20 msgr2=0x7f48a80cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:07.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.873+0000 7f48ace00640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48a80cca20 0x7f48a80cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.873+0000 7f48ace00640 1 -- 172.21.15.6:0/1937405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 msgr2=0x7f48a80b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:07.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.873+0000 7f48ace00640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 0x7f48a80b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:07.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.873+0000 7f48ace00640 1 -- 172.21.15.6:0/1937405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a80af3d0 con 0x7f48a815b7a0 2024-03-22T13:09:07.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.878+0000 7f48ace00640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a81f0bf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f48a000b9e0 tx=0x7f48a000bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.883+0000 7f489fe00640 1 -- 172.21.15.6:0/1937405 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0019070 con 0x7f48a815b7a0 2024-03-22T13:09:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.883+0000 7f489fe00640 1 -- 172.21.15.6:0/1937405 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a000ca40 con 0x7f48a815b7a0 2024-03-22T13:09:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.883+0000 7f489fe00640 1 -- 172.21.15.6:0/1937405 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0014440 con 0x7f48a815b7a0 2024-03-22T13:09:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.883+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a81f0410 con 0x7f48a815b7a0 2024-03-22T13:09:07.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.884+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a80b0f50 con 0x7f48a815b7a0 2024-03-22T13:09:07.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:07.885+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a8075650 con 0x7f48a815b7a0 2024-03-22T13:09:08.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.180+0000 7f489fe00640 1 -- 172.21.15.6:0/1937405 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f48a001c070 con 0x7f48a815b7a0 2024-03-22T13:09:08.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.182+0000 7f489fe00640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f488806bc20 0x7f488806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:08.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.182+0000 7f489fe00640 1 -- 172.21.15.6:0/1937405 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f48a008aee0 con 0x7f48a815b7a0 2024-03-22T13:09:08.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.183+0000 7f48a7e00640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f488806bc20 0x7f488806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:08.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.184+0000 7f48a7e00640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f488806bc20 0x7f488806e0e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4898036000 tx=0x7f489803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:08.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.200+0000 7f489fe00640 1 -- 172.21.15.6:0/1937405 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f48a0023020 con 0x7f48a815b7a0 2024-03-22T13:09:08.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.450+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a80af100 con 0x7f48a815b7a0 2024-03-22T13:09:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.466+0000 7f489fe00640 1 -- 172.21.15.6:0/1937405 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f48a0059620 con 0x7f48a815b7a0 2024-03-22T13:09:08.466 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:08.466 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.469+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f488806bc20 msgr2=0x7f488806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.469+0000 7f48ad800640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f488806bc20 0x7f488806e0e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4898036000 tx=0x7f489803b040 comp rx=0 tx=0).stop 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.469+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 msgr2=0x7f48a81f0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.469+0000 7f48ad800640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a81f0bf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f48a000b9e0 tx=0x7f48a000bde0 comp rx=0 tx=0).stop 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 shutdown_connections 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48a815b7a0 0x7f48a81f0bf0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f488806bc20 0x7f488806e0e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48a80cca20 0x7f48a80cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 --2- 172.21.15.6:0/1937405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a80cc640 0x7f48a80b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 >> 172.21.15.6:0/1937405 conn(0x7f48a80cf9c0 msgr2=0x7f48a815a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 shutdown_connections 2024-03-22T13:09:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:08.470+0000 7f48ad800640 1 -- 172.21.15.6:0/1937405 wait complete. 2024-03-22T13:09:09.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:09.091 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:10.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:10.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.278+0000 7fd48f400640 1 Processor -- start 2024-03-22T13:09:10.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.278+0000 7fd48f400640 1 -- start start 2024-03-22T13:09:10.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.279+0000 7fd48f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd488152400 0x7fd48815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:10.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.279+0000 7fd48f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48806f8c0 0x7fd4880c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:10.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.279+0000 7fd48f400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4881590b0 con 0x7fd48815bc80 2024-03-22T13:09:10.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.279+0000 7fd48f400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4880724d0 con 0x7fd488152400 2024-03-22T13:09:10.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.279+0000 7fd48f400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880721f0 con 0x7fd48806f8c0 2024-03-22T13:09:10.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.280+0000 7fd48ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd488152400 0x7fd48815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:10.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.280+0000 7fd48ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd488152400 0x7fd48815b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50320/0 (socket says 172.21.15.6:50320) 2024-03-22T13:09:10.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.280+0000 7fd48ca00640 1 -- 172.21.15.6:0/2864173211 learned_addr learned my addr 172.21.15.6:0/2864173211 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:10.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.281+0000 7fd48d400640 1 --2- 172.21.15.6:0/2864173211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48806f8c0 0x7fd4880c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.283+0000 7fd48ca00640 1 -- 172.21.15.6:0/2864173211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48806f8c0 msgr2=0x7fd4880c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.283+0000 7fd48ca00640 1 --2- 172.21.15.6:0/2864173211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48806f8c0 0x7fd4880c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.283+0000 7fd48ca00640 1 -- 172.21.15.6:0/2864173211 >> v1:172.21.15.6:6789/0 conn(0x7fd48815bc80 legacy=0x7fd4880716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.283+0000 7fd48ca00640 1 -- 172.21.15.6:0/2864173211 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4881582f0 con 0x7fd488152400 2024-03-22T13:09:10.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.286+0000 7fd48ca00640 1 --2- 172.21.15.6:0/2864173211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd488152400 0x7fd48815b740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009870 tx=0x7fd47c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e328b5697eb335f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd487400640 1 -- 172.21.15.6:0/2864173211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c034070 con 0x7fd488152400 2024-03-22T13:09:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd487400640 1 -- 172.21.15.6:0/2864173211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c030840 con 0x7fd488152400 2024-03-22T13:09:10.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd487400640 1 -- 172.21.15.6:0/2864173211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c030c10 con 0x7fd488152400 2024-03-22T13:09:10.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd48f400640 1 -- 172.21.15.6:0/2864173211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd488152400 msgr2=0x7fd48815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:10.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd48f400640 1 --2- 172.21.15.6:0/2864173211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd488152400 0x7fd48815b740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009870 tx=0x7fd47c02b980 comp rx=0 tx=0).stop 2024-03-22T13:09:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd48f400640 1 -- 172.21.15.6:0/2864173211 shutdown_connections 2024-03-22T13:09:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd48f400640 1 --2- 172.21.15.6:0/2864173211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48806f8c0 0x7fd4880c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.291+0000 7fd48f400640 1 --2- 172.21.15.6:0/2864173211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd488152400 0x7fd48815b740 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.292+0000 7fd48f400640 1 -- 172.21.15.6:0/2864173211 >> 172.21.15.6:0/2864173211 conn(0x7fd4880cb930 msgr2=0x7fd4880cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.292+0000 7fd48f400640 1 -- 172.21.15.6:0/2864173211 shutdown_connections 2024-03-22T13:09:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.292+0000 7fd48f400640 1 -- 172.21.15.6:0/2864173211 wait complete. 2024-03-22T13:09:10.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.294+0000 7fd48f400640 1 Processor -- start 2024-03-22T13:09:10.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.295+0000 7fd48f400640 1 -- start start 2024-03-22T13:09:10.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.296+0000 7fd48f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd48806f8c0 0x7fd488155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:10.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.296+0000 7fd48f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488152400 0x7fd488159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:10.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.296+0000 7fd48f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48815bc80 0x7fd4881f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:10.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.296+0000 7fd48f400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd488158b90 con 0x7fd48806f8c0 2024-03-22T13:09:10.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.296+0000 7fd48f400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881588b0 con 0x7fd488152400 2024-03-22T13:09:10.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.296+0000 7fd48f400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881527e0 con 0x7fd48815bc80 2024-03-22T13:09:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.297+0000 7fd48ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd48806f8c0 0x7fd488155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.297+0000 7fd487e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488152400 0x7fd488159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.297+0000 7fd48d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48815bc80 0x7fd4881f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.297+0000 7fd48ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd48806f8c0 0x7fd488155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50328/0 (socket says 172.21.15.6:50328) 2024-03-22T13:09:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.297+0000 7fd48ca00640 1 -- 172.21.15.6:0/783920418 learned_addr learned my addr 172.21.15.6:0/783920418 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:10.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.301+0000 7fd48ca00640 1 -- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48815bc80 msgr2=0x7fd4881f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:10.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.301+0000 7fd48ca00640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48815bc80 0x7fd4881f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.301+0000 7fd48ca00640 1 -- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488152400 msgr2=0x7fd488159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:10.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.301+0000 7fd48ca00640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488152400 0x7fd488159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.301+0000 7fd48ca00640 1 -- 172.21.15.6:0/783920418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4881461b0 con 0x7fd48806f8c0 2024-03-22T13:09:10.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.301+0000 7fd487e00640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488152400 0x7fd488159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:09:10.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.304+0000 7fd48ca00640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd48806f8c0 0x7fd488155fe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd47c000c00 tx=0x7fd47c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.308+0000 7fd484c00640 1 -- 172.21.15.6:0/783920418 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c048070 con 0x7fd48806f8c0 2024-03-22T13:09:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.308+0000 7fd484c00640 1 -- 172.21.15.6:0/783920418 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c030640 con 0x7fd48806f8c0 2024-03-22T13:09:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.308+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488146490 con 0x7fd48806f8c0 2024-03-22T13:09:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.308+0000 7fd484c00640 1 -- 172.21.15.6:0/783920418 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c034040 con 0x7fd48806f8c0 2024-03-22T13:09:10.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.308+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488146a50 con 0x7fd48806f8c0 2024-03-22T13:09:10.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.309+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd488072d10 con 0x7fd48806f8c0 2024-03-22T13:09:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.340+0000 7fd484c00640 1 -- 172.21.15.6:0/783920418 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd47c04b070 con 0x7fd48806f8c0 2024-03-22T13:09:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.342+0000 7fd484c00640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd46806bbd0 0x7fd46806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.342+0000 7fd487e00640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd46806bbd0 0x7fd46806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.343+0000 7fd487e00640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd46806bbd0 0x7fd46806e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd488074e20 tx=0x7fd474009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.347+0000 7fd484c00640 1 -- 172.21.15.6:0/783920418 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd47c0307f0 con 0x7fd48806f8c0 2024-03-22T13:09:10.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.638+0000 7fd484c00640 1 -- 172.21.15.6:0/783920418 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd488072d10 con 0x7fd48806f8c0 2024-03-22T13:09:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.887+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4880756c0 con 0x7fd48806f8c0 2024-03-22T13:09:10.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.909+0000 7fd484c00640 1 -- 172.21.15.6:0/783920418 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fd47c052020 con 0x7fd48806f8c0 2024-03-22T13:09:10.909 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:10.909 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:10.909 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd46806bbd0 msgr2=0x7fd46806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd46806bbd0 0x7fd46806e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd488074e20 tx=0x7fd474009060 comp rx=0 tx=0).stop 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd48806f8c0 msgr2=0x7fd488155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd48806f8c0 0x7fd488155fe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd47c000c00 tx=0x7fd47c004730 comp rx=0 tx=0).stop 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 shutdown_connections 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd48815bc80 0x7fd4881f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd46806bbd0 0x7fd46806e090 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488152400 0x7fd488159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 --2- 172.21.15.6:0/783920418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd48806f8c0 0x7fd488155fe0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 >> 172.21.15.6:0/783920418 conn(0x7fd4880cb930 msgr2=0x7fd4880716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:10.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.912+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 shutdown_connections 2024-03-22T13:09:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:10.913+0000 7fd48f400640 1 -- 172.21.15.6:0/783920418 wait complete. 2024-03-22T13:09:11.539 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:11.540 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:12.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.738+0000 7f651f800640 1 Processor -- start 2024-03-22T13:09:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.739+0000 7f651f800640 1 -- start start 2024-03-22T13:09:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.740+0000 7f651f800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f65180ccad0 0x7f651806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.740+0000 7f651f800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651806c7a0 con 0x7f651815b0e0 2024-03-22T13:09:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.740+0000 7f651f800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651806ca80 con 0x7f65180ca520 2024-03-22T13:09:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.740+0000 7f651f800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65180cd000 con 0x7f65180ccad0 2024-03-22T13:09:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.741+0000 7f651d800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f651815b0e0 0x7f6518164db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56474/0 (socket says 172.21.15.6:56474) 2024-03-22T13:09:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.741+0000 7f651d800640 1 -- 172.21.15.6:0/3906242814 learned_addr learned my addr 172.21.15.6:0/3906242814 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.741+0000 7f6517e00640 1 --2- 172.21.15.6:0/3906242814 >> v2:172.21.15.92:3300/0 conn(0x7f65180ccad0 0x7f651806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.744+0000 7f6517400640 1 -- 172.21.15.6:0/3906242814 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296582180 0 0) 0x7f651806c7a0 con 0x7f651815b0e0 2024-03-22T13:09:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.744+0000 7f6517400640 1 -- 172.21.15.6:0/3906242814 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6500003520 con 0x7f651815b0e0 2024-03-22T13:09:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.745+0000 7f6517e00640 1 -- 172.21.15.6:0/3906242814 >> v1:172.21.15.92:6789/0 conn(0x7f65180ca520 legacy=0x7f65180cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.746+0000 7f6517e00640 1 -- 172.21.15.6:0/3906242814 >> v1:172.21.15.6:6789/0 conn(0x7f651815b0e0 legacy=0x7f6518164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.746+0000 7f6517e00640 1 -- 172.21.15.6:0/3906242814 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651806cd60 con 0x7f65180ccad0 2024-03-22T13:09:12.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.751+0000 7f6517e00640 1 --2- 172.21.15.6:0/3906242814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ccad0 0x7f651806d840 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f65040097a0 tx=0x7f650402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63b4cdc8d792c3f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.756+0000 7f6517400640 1 -- 172.21.15.6:0/3906242814 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6504034070 con 0x7f65180ccad0 2024-03-22T13:09:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.756+0000 7f6517400640 1 -- 172.21.15.6:0/3906242814 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6504030610 con 0x7f65180ccad0 2024-03-22T13:09:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.756+0000 7f6517400640 1 -- 172.21.15.6:0/3906242814 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65040309e0 con 0x7f65180ccad0 2024-03-22T13:09:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.757+0000 7f651f800640 1 -- 172.21.15.6:0/3906242814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ccad0 msgr2=0x7f651806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.757+0000 7f651f800640 1 --2- 172.21.15.6:0/3906242814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ccad0 0x7f651806d840 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f65040097a0 tx=0x7f650402b960 comp rx=0 tx=0).stop 2024-03-22T13:09:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.757+0000 7f651f800640 1 -- 172.21.15.6:0/3906242814 shutdown_connections 2024-03-22T13:09:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.757+0000 7f651f800640 1 --2- 172.21.15.6:0/3906242814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ccad0 0x7f651806d840 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.757+0000 7f651f800640 1 -- 172.21.15.6:0/3906242814 >> 172.21.15.6:0/3906242814 conn(0x7f65180b16b0 msgr2=0x7f65180b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.757+0000 7f651f800640 1 -- 172.21.15.6:0/3906242814 shutdown_connections 2024-03-22T13:09:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.757+0000 7f651f800640 1 -- 172.21.15.6:0/3906242814 wait complete. 2024-03-22T13:09:12.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.759+0000 7f651f800640 1 Processor -- start 2024-03-22T13:09:12.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.759+0000 7f651f800640 1 -- start start 2024-03-22T13:09:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.760+0000 7f651f800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ca520 0x7f651815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.760+0000 7f651f800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65180ccad0 0x7f6518154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.760+0000 7f651f800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651815b0e0 0x7f65181553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.760+0000 7f651f800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6518154c20 con 0x7f651815b0e0 2024-03-22T13:09:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.760+0000 7f651f800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651806d320 con 0x7f65180ca520 2024-03-22T13:09:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.760+0000 7f651f800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651815b980 con 0x7f65180ccad0 2024-03-22T13:09:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.761+0000 7f651ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ca520 0x7f651815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.761+0000 7f651d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651815b0e0 0x7f65181553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.761+0000 7f6517e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65180ccad0 0x7f6518154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.761+0000 7f651ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ca520 0x7f651815c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:34444/0 (socket says 172.21.15.6:34444) 2024-03-22T13:09:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.761+0000 7f651ce00640 1 -- 172.21.15.6:0/3721255685 learned_addr learned my addr 172.21.15.6:0/3721255685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.765+0000 7f651ce00640 1 -- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65180ccad0 msgr2=0x7f6518154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.765+0000 7f651ce00640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65180ccad0 0x7f6518154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.765+0000 7f651ce00640 1 -- 172.21.15.6:0/3721255685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651815b0e0 msgr2=0x7f65181553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.765+0000 7f651ce00640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651815b0e0 0x7f65181553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.765+0000 7f651ce00640 1 -- 172.21.15.6:0/3721255685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65181f39b0 con 0x7f65180ca520 2024-03-22T13:09:12.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.768+0000 7f651ce00640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ca520 0x7f651815c020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c450 tx=0x7f650c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.770+0000 7f6514c00640 1 -- 172.21.15.6:0/3721255685 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c0052d0 con 0x7f65180ca520 2024-03-22T13:09:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.771+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65180cef30 con 0x7f65180ca520 2024-03-22T13:09:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.771+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6518155d00 con 0x7f65180ca520 2024-03-22T13:09:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.772+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65180756c0 con 0x7f65180ca520 2024-03-22T13:09:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.774+0000 7f6514c00640 1 -- 172.21.15.6:0/3721255685 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c005470 con 0x7f65180ca520 2024-03-22T13:09:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:12.774+0000 7f6514c00640 1 -- 172.21.15.6:0/3721255685 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c007250 con 0x7f65180ca520 2024-03-22T13:09:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.076+0000 7f6514c00640 1 -- 172.21.15.6:0/3721255685 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f650c0073f0 con 0x7f65180ca520 2024-03-22T13:09:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.079+0000 7f6514c00640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64f806bbd0 0x7f64f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.079+0000 7f6517e00640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64f806bbd0 0x7f64f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:13.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.080+0000 7f6517e00640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64f806bbd0 0x7f64f806e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6504036000 tx=0x7f650403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:13.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.081+0000 7f6514c00640 1 -- 172.21.15.6:0/3721255685 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f650c057570 con 0x7f65180ca520 2024-03-22T13:09:13.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.100+0000 7f6514c00640 1 -- 172.21.15.6:0/3721255685 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f650c0594d0 con 0x7f65180ca520 2024-03-22T13:09:13.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.355+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65180cb080 con 0x7f65180ca520 2024-03-22T13:09:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.377+0000 7f6514c00640 1 -- 172.21.15.6:0/3721255685 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f650c05d570 con 0x7f65180ca520 2024-03-22T13:09:13.377 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:13.378 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64f806bbd0 msgr2=0x7f64f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64f806bbd0 0x7f64f806e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6504036000 tx=0x7f650403b040 comp rx=0 tx=0).stop 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ca520 msgr2=0x7f651815c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ca520 0x7f651815c020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c450 tx=0x7f650c00c910 comp rx=0 tx=0).stop 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 shutdown_connections 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651815b0e0 0x7f65181553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65180ccad0 0x7f6518154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f64f806bbd0 0x7f64f806e090 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 --2- 172.21.15.6:0/3721255685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65180ca520 0x7f651815c020 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.380+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 >> 172.21.15.6:0/3721255685 conn(0x7f65180b16b0 msgr2=0x7f65180cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.381+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 shutdown_connections 2024-03-22T13:09:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:13.381+0000 7f651f800640 1 -- 172.21.15.6:0/3721255685 wait complete. 2024-03-22T13:09:14.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:14.001 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:15.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.203+0000 7f9a43e00640 1 Processor -- start 2024-03-22T13:09:15.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.203+0000 7f9a43e00640 1 -- start start 2024-03-22T13:09:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.205+0000 7f9a43e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c0cc790 0x7f9a3c0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.205+0000 7f9a43e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9a3c15ab40 0x7f9a3c15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.205+0000 7f9a43e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c15b510 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.205+0000 7f9a43e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a3c0cd0f0 con 0x7f9a3c15ab40 2024-03-22T13:09:15.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.205+0000 7f9a43e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0ce230 con 0x7f9a3c0cc790 2024-03-22T13:09:15.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.206+0000 7f9a40a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c0cc790 0x7f9a3c0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:15.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.206+0000 7f9a41e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9a3c15ab40 0x7f9a3c15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:15.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.206+0000 7f9a40a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c0cc790 0x7f9a3c0ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60826/0 (socket says 172.21.15.6:60826) 2024-03-22T13:09:15.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.206+0000 7f9a40a00640 1 -- 172.21.15.6:0/4159188744 learned_addr learned my addr 172.21.15.6:0/4159188744 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.209+0000 7f9a40a00640 1 -- 172.21.15.6:0/4159188744 >> v2:172.21.15.92:3300/0 conn(0x7f9a3c15ab40 msgr2=0x7f9a3c15afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.209+0000 7f9a40a00640 1 --2- 172.21.15.6:0/4159188744 >> v2:172.21.15.92:3300/0 conn(0x7f9a3c15ab40 0x7f9a3c15afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.209+0000 7f9a40a00640 1 -- 172.21.15.6:0/4159188744 >> v1:172.21.15.92:6789/0 conn(0x7f9a3c0cc3b0 legacy=0x7f9a3c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.210+0000 7f9a40a00640 1 -- 172.21.15.6:0/4159188744 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c0ce510 con 0x7f9a3c0cc790 2024-03-22T13:09:15.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.212+0000 7f9a40a00640 1 --2- 172.21.15.6:0/4159188744 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c0cc790 0x7f9a3c0ccbb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009870 tx=0x7f9a2802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d8fc81f9ef8993a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:15.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.217+0000 7f9a3be00640 1 -- 172.21.15.6:0/4159188744 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a28035020 con 0x7f9a3c0cc790 2024-03-22T13:09:15.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.217+0000 7f9a3be00640 1 -- 172.21.15.6:0/4159188744 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a28031640 con 0x7f9a3c0cc790 2024-03-22T13:09:15.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.217+0000 7f9a3be00640 1 -- 172.21.15.6:0/4159188744 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2802d040 con 0x7f9a3c0cc790 2024-03-22T13:09:15.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.217+0000 7f9a43e00640 1 -- 172.21.15.6:0/4159188744 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c0cc790 msgr2=0x7f9a3c0ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:15.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.217+0000 7f9a43e00640 1 --2- 172.21.15.6:0/4159188744 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c0cc790 0x7f9a3c0ccbb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009870 tx=0x7f9a2802b8f0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.218+0000 7f9a43e00640 1 -- 172.21.15.6:0/4159188744 shutdown_connections 2024-03-22T13:09:15.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.218+0000 7f9a43e00640 1 --2- 172.21.15.6:0/4159188744 >> v2:172.21.15.92:3300/0 conn(0x7f9a3c15ab40 0x7f9a3c15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.218+0000 7f9a43e00640 1 --2- 172.21.15.6:0/4159188744 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c0cc790 0x7f9a3c0ccbb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.218+0000 7f9a43e00640 1 -- 172.21.15.6:0/4159188744 >> 172.21.15.6:0/4159188744 conn(0x7f9a3c0b16d0 msgr2=0x7f9a3c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:15.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.218+0000 7f9a43e00640 1 -- 172.21.15.6:0/4159188744 shutdown_connections 2024-03-22T13:09:15.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.218+0000 7f9a43e00640 1 -- 172.21.15.6:0/4159188744 wait complete. 2024-03-22T13:09:15.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.219+0000 7f9a43e00640 1 Processor -- start 2024-03-22T13:09:15.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.219+0000 7f9a43e00640 1 -- start start 2024-03-22T13:09:15.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.220+0000 7f9a43e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a3c0cc3b0 0x7f9a3c15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:15.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.220+0000 7f9a43e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0cc790 0x7f9a3c15c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:15.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.220+0000 7f9a43e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c15ab40 0x7f9a3c1586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:15.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.220+0000 7f9a43e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0cedb0 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.220+0000 7f9a43e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0cead0 con 0x7f9a3c0cc790 2024-03-22T13:09:15.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.220+0000 7f9a43e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c06dc10 con 0x7f9a3c15ab40 2024-03-22T13:09:15.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.221+0000 7f9a41e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c15ab40 0x7f9a3c1586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:15.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.221+0000 7f9a40a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0cc790 0x7f9a3c15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:15.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.221+0000 7f9a41400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a3c0cc3b0 0x7f9a3c15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:15.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.221+0000 7f9a41e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c15ab40 0x7f9a3c1586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:60842/0 (socket says 172.21.15.6:60842) 2024-03-22T13:09:15.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.221+0000 7f9a41e00640 1 -- 172.21.15.6:0/1387265285 learned_addr learned my addr 172.21.15.6:0/1387265285 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:15.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.226+0000 7f9a41400640 1 -- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c15ab40 msgr2=0x7f9a3c1586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:15.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.226+0000 7f9a41400640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c15ab40 0x7f9a3c1586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.226+0000 7f9a41400640 1 -- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0cc790 msgr2=0x7f9a3c15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:15.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.226+0000 7f9a41400640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0cc790 0x7f9a3c15c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.226+0000 7f9a41400640 1 -- 172.21.15.6:0/1387265285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c1f05b0 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.228+0000 7f9a41400640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a3c0cc3b0 0x7f9a3c15c1e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000c340 tx=0x7f9a3000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:15.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.233+0000 7f9a39600640 1 -- 172.21.15.6:0/1387265285 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300052d0 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.233+0000 7f9a39600640 1 -- 172.21.15.6:0/1387265285 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a30005490 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.233+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c1f0870 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.233+0000 7f9a39600640 1 -- 172.21.15.6:0/1387265285 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30007260 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.233+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c1f0e30 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.235+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c0756c0 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.267+0000 7f9a39600640 1 -- 172.21.15.6:0/1387265285 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a30007400 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.269+0000 7f9a39600640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9a1c06bc20 0x7f9a1c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:15.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.270+0000 7f9a40a00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9a1c06bc20 0x7f9a1c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.271+0000 7f9a40a00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9a1c06bc20 0x7f9a1c06e0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9a28036000 tx=0x7f9a2803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:15.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.274+0000 7f9a39600640 1 -- 172.21.15.6:0/1387265285 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9a300568a0 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.565+0000 7f9a39600640 1 -- 172.21.15.6:0/1387265285 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a30058800 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.825+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a3c15afb0 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.846+0000 7f9a39600640 1 -- 172.21.15.6:0/1387265285 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f9a3005c8a0 con 0x7f9a3c0cc3b0 2024-03-22T13:09:15.846 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:15.846 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:15.846 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:15.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9a1c06bc20 msgr2=0x7f9a1c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9a1c06bc20 0x7f9a1c06e0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9a28036000 tx=0x7f9a2803c040 comp rx=0 tx=0).stop 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a3c0cc3b0 msgr2=0x7f9a3c15c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a3c0cc3b0 0x7f9a3c15c1e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000c340 tx=0x7f9a3000c800 comp rx=0 tx=0).stop 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 shutdown_connections 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c15ab40 0x7f9a3c1586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f9a1c06bc20 0x7f9a1c06e0e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0cc790 0x7f9a3c15c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 --2- 172.21.15.6:0/1387265285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a3c0cc3b0 0x7f9a3c15c1e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.848+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 >> 172.21.15.6:0/1387265285 conn(0x7f9a3c0b16d0 msgr2=0x7f9a3c0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.849+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 shutdown_connections 2024-03-22T13:09:15.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:15.849+0000 7f9a43e00640 1 -- 172.21.15.6:0/1387265285 wait complete. 2024-03-22T13:09:16.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:16.475 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:17.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:17.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.672+0000 7ff65f400640 1 Processor -- start 2024-03-22T13:09:17.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.672+0000 7ff65f400640 1 -- start start 2024-03-22T13:09:17.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.674+0000 7ff65f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6580ca520 0x7ff65806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:17.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.674+0000 7ff65f400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65806c7a0 con 0x7ff6580ccad0 2024-03-22T13:09:17.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.674+0000 7ff65f400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65806ca80 con 0x7ff65815b0e0 2024-03-22T13:09:17.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.674+0000 7ff65f400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6580cd000 con 0x7ff6580ca520 2024-03-22T13:09:17.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.675+0000 7ff657e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff65815b0e0 0x7ff6580cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43880/0 (socket says 172.21.15.6:43880) 2024-03-22T13:09:17.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.675+0000 7ff657e00640 1 -- 172.21.15.6:0/1989489341 learned_addr learned my addr 172.21.15.6:0/1989489341 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:17.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.675+0000 7ff65ca00640 1 --2- 172.21.15.6:0/1989489341 >> v2:172.21.15.6:3300/0 conn(0x7ff6580ca520 0x7ff65806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:17.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.678+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088874968 0 0) 0x7ff65806ca80 con 0x7ff65815b0e0 2024-03-22T13:09:17.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.678+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff640003520 con 0x7ff65815b0e0 2024-03-22T13:09:17.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346757555 0 0) 0x7ff640003520 con 0x7ff65815b0e0 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 >> v2:172.21.15.6:3300/0 conn(0x7ff6580ca520 msgr2=0x7ff65806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 --2- 172.21.15.6:0/1989489341 >> v2:172.21.15.6:3300/0 conn(0x7ff6580ca520 0x7ff65806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 >> v1:172.21.15.6:6789/0 conn(0x7ff6580ccad0 legacy=0x7ff658164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65806cd60 con 0x7ff65815b0e0 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7ff644001670 con 0x7ff65815b0e0 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 >> v1:172.21.15.92:6789/0 conn(0x7ff65815b0e0 legacy=0x7ff6580cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff640003a00 0x7ff640003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.681+0000 7ff657400640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640004800 0x7ff640004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.682+0000 7ff657400640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6400051c0 0x7ff64000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.682+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6580cd000 con 0x7ff640004800 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.682+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64000c0c0 con 0x7ff6400051c0 2024-03-22T13:09:17.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.682+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64000c2c0 con 0x7ff640003a00 2024-03-22T13:09:17.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.682+0000 7ff65ca00640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff640003a00 0x7ff640003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:17.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.685+0000 7ff65d400640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6400051c0 0x7ff64000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:17.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.686+0000 7ff65ca00640 1 -- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6400051c0 msgr2=0x7ff64000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:17.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.686+0000 7ff65ca00640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6400051c0 0x7ff64000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:17.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.686+0000 7ff65ca00640 1 -- 172.21.15.6:0/1989489341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640004800 msgr2=0x7ff640004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:17.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.686+0000 7ff65ca00640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640004800 0x7ff640004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:17.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.686+0000 7ff65ca00640 1 -- 172.21.15.6:0/1989489341 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640003520 con 0x7ff640003a00 2024-03-22T13:09:17.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.686+0000 7ff65d400640 1 -- 172.21.15.6:0/1989489341 reap_dead start 2024-03-22T13:09:17.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.689+0000 7ff65ca00640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff640003a00 0x7ff640003e20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009ac0 tx=0x7ff640005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37163e5e4e591e9f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:17.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.693+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c039070 con 0x7ff640003a00 2024-03-22T13:09:17.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.693+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c02c6b0 con 0x7ff640003a00 2024-03-22T13:09:17.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.693+0000 7ff657400640 1 -- 172.21.15.6:0/1989489341 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c009690 con 0x7ff640003a00 2024-03-22T13:09:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.694+0000 7ff65f400640 1 -- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff640003a00 msgr2=0x7ff640003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.694+0000 7ff65f400640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff640003a00 0x7ff640003e20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009ac0 tx=0x7ff640005830 comp rx=0 tx=0).stop 2024-03-22T13:09:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.694+0000 7ff65f400640 1 -- 172.21.15.6:0/1989489341 shutdown_connections 2024-03-22T13:09:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.694+0000 7ff65f400640 1 --2- 172.21.15.6:0/1989489341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff640003a00 0x7ff640003e20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.694+0000 7ff65f400640 1 -- 172.21.15.6:0/1989489341 >> 172.21.15.6:0/1989489341 conn(0x7ff6580b16b0 msgr2=0x7ff6580b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.694+0000 7ff65f400640 1 -- 172.21.15.6:0/1989489341 shutdown_connections 2024-03-22T13:09:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.694+0000 7ff65f400640 1 -- 172.21.15.6:0/1989489341 wait complete. 2024-03-22T13:09:17.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.696+0000 7ff65f400640 1 Processor -- start 2024-03-22T13:09:17.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.696+0000 7ff65f400640 1 -- start start 2024-03-22T13:09:17.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640003a00 0x7ff658155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:17.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff65815c280 0x7ff658151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:17.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6580cebc0 0x7ff6580cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:17.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65f400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff658154c20 con 0x7ff640003a00 2024-03-22T13:09:17.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65f400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65806d320 con 0x7ff65815c280 2024-03-22T13:09:17.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65f400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65806d840 con 0x7ff6580cebc0 2024-03-22T13:09:17.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640003a00 0x7ff658155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:17.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff657e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff65815c280 0x7ff658151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:17.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.698+0000 7ff65d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6580cebc0 0x7ff6580cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:17.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.699+0000 7ff65ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640003a00 0x7ff658155450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36234/0 (socket says 172.21.15.6:36234) 2024-03-22T13:09:17.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.699+0000 7ff65ca00640 1 -- 172.21.15.6:0/4147681741 learned_addr learned my addr 172.21.15.6:0/4147681741 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:17.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.703+0000 7ff65ca00640 1 -- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6580cebc0 msgr2=0x7ff6580cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:17.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.703+0000 7ff65ca00640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6580cebc0 0x7ff6580cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:17.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.703+0000 7ff65ca00640 1 -- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff65815c280 msgr2=0x7ff658151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:17.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.703+0000 7ff65ca00640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff65815c280 0x7ff658151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:17.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.703+0000 7ff65ca00640 1 -- 172.21.15.6:0/4147681741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6581565e0 con 0x7ff640003a00 2024-03-22T13:09:17.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.706+0000 7ff65ca00640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640003a00 0x7ff658155450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff64c037040 tx=0x7ff64c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:17.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.710+0000 7ff654c00640 1 -- 172.21.15.6:0/4147681741 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c042020 con 0x7ff640003a00 2024-03-22T13:09:17.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.710+0000 7ff654c00640 1 -- 172.21.15.6:0/4147681741 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c03a040 con 0x7ff640003a00 2024-03-22T13:09:17.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.710+0000 7ff654c00640 1 -- 172.21.15.6:0/4147681741 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c039070 con 0x7ff640003a00 2024-03-22T13:09:17.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.710+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6581f0ce0 con 0x7ff640003a00 2024-03-22T13:09:17.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.711+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65815ba80 con 0x7ff640003a00 2024-03-22T13:09:17.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:17.712+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6580756c0 con 0x7ff640003a00 2024-03-22T13:09:18.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.009+0000 7ff654c00640 1 -- 172.21.15.6:0/4147681741 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff64c045020 con 0x7ff640003a00 2024-03-22T13:09:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.010+0000 7ff654c00640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff63806bc00 0x7ff63806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:18.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.011+0000 7ff657e00640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff63806bc00 0x7ff63806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:18.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.012+0000 7ff657e00640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff63806bc00 0x7ff63806e0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff644000fb0 tx=0x7ff6440019f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.038+0000 7ff654c00640 1 -- 172.21.15.6:0/4147681741 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff64c079000 con 0x7ff640003a00 2024-03-22T13:09:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.038+0000 7ff654c00640 1 -- 172.21.15.6:0/4147681741 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff64c04c030 con 0x7ff640003a00 2024-03-22T13:09:18.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.345+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6580cb080 con 0x7ff640003a00 2024-03-22T13:09:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.372+0000 7ff654c00640 1 -- 172.21.15.6:0/4147681741 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7ff64c07af60 con 0x7ff640003a00 2024-03-22T13:09:18.372 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:18.372 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.374+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff63806bc00 msgr2=0x7ff63806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.374+0000 7ff65f400640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff63806bc00 0x7ff63806e0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff644000fb0 tx=0x7ff6440019f0 comp rx=0 tx=0).stop 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.374+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640003a00 msgr2=0x7ff658155450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.374+0000 7ff65f400640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640003a00 0x7ff658155450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff64c037040 tx=0x7ff64c004730 comp rx=0 tx=0).stop 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 shutdown_connections 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6580cebc0 0x7ff6580cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff63806bc00 0x7ff63806e0c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff65815c280 0x7ff658151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 --2- 172.21.15.6:0/4147681741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff640003a00 0x7ff658155450 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 >> 172.21.15.6:0/4147681741 conn(0x7ff6580b16b0 msgr2=0x7ff658153e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 shutdown_connections 2024-03-22T13:09:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:18.375+0000 7ff65f400640 1 -- 172.21.15.6:0/4147681741 wait complete. 2024-03-22T13:09:18.996 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:18.996 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:19.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:20.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.196+0000 7f94c7a00640 1 Processor -- start 2024-03-22T13:09:20.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.196+0000 7f94c7a00640 1 -- start start 2024-03-22T13:09:20.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.197+0000 7f94c7a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f94c00c5040 0x7f94c00c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:20.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.198+0000 7f94c7a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c01590b0 con 0x7f94c006f8c0 2024-03-22T13:09:20.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.198+0000 7f94c7a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c00721f0 con 0x7f94c0152400 2024-03-22T13:09:20.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.198+0000 7f94c7a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c00724d0 con 0x7f94c00c5040 2024-03-22T13:09:20.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.198+0000 7f94c5000640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f94c0152400 0x7f94c015b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43890/0 (socket says 172.21.15.6:43890) 2024-03-22T13:09:20.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.198+0000 7f94c5000640 1 -- 172.21.15.6:0/3853668599 learned_addr learned my addr 172.21.15.6:0/3853668599 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:20.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.199+0000 7f94c5a00640 1 --2- 172.21.15.6:0/3853668599 >> v2:172.21.15.6:3300/0 conn(0x7f94c00c5040 0x7f94c00c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:20.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.202+0000 7f94bfe00640 1 -- 172.21.15.6:0/3853668599 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222332489 0 0) 0x7f94c00721f0 con 0x7f94c0152400 2024-03-22T13:09:20.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.202+0000 7f94bfe00640 1 -- 172.21.15.6:0/3853668599 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a4003520 con 0x7f94c0152400 2024-03-22T13:09:20.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.203+0000 7f94c5a00640 1 -- 172.21.15.6:0/3853668599 >> v1:172.21.15.92:6789/0 conn(0x7f94c0152400 legacy=0x7f94c015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:20.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.203+0000 7f94c5a00640 1 -- 172.21.15.6:0/3853668599 >> v1:172.21.15.6:6789/0 conn(0x7f94c006f8c0 legacy=0x7f94c00716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:20.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.203+0000 7f94c5a00640 1 -- 172.21.15.6:0/3853668599 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c01582f0 con 0x7f94c00c5040 2024-03-22T13:09:20.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.207+0000 7f94c5a00640 1 --2- 172.21.15.6:0/3853668599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00c5420 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009d90 tx=0x7f94b802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880ea15cd0cf5157 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:20.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.215+0000 7f94bfe00640 1 -- 172.21.15.6:0/3853668599 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8039070 con 0x7f94c00c5040 2024-03-22T13:09:20.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.215+0000 7f94bfe00640 1 -- 172.21.15.6:0/3853668599 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b802cd80 con 0x7f94c00c5040 2024-03-22T13:09:20.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.215+0000 7f94bfe00640 1 -- 172.21.15.6:0/3853668599 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8034660 con 0x7f94c00c5040 2024-03-22T13:09:20.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.215+0000 7f94c7a00640 1 -- 172.21.15.6:0/3853668599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 msgr2=0x7f94c00c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:20.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.215+0000 7f94c7a00640 1 --2- 172.21.15.6:0/3853668599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00c5420 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009d90 tx=0x7f94b802bfd0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.216+0000 7f94c7a00640 1 -- 172.21.15.6:0/3853668599 shutdown_connections 2024-03-22T13:09:20.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.216+0000 7f94c7a00640 1 --2- 172.21.15.6:0/3853668599 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00c5420 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.216+0000 7f94c7a00640 1 -- 172.21.15.6:0/3853668599 >> 172.21.15.6:0/3853668599 conn(0x7f94c00cb930 msgr2=0x7f94c00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:20.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.216+0000 7f94c7a00640 1 -- 172.21.15.6:0/3853668599 shutdown_connections 2024-03-22T13:09:20.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.216+0000 7f94c7a00640 1 -- 172.21.15.6:0/3853668599 wait complete. 2024-03-22T13:09:20.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.219+0000 7f94c7a00640 1 Processor -- start 2024-03-22T13:09:20.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.220+0000 7f94c7a00640 1 -- start start 2024-03-22T13:09:20.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.221+0000 7f94c7a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94c006f8c0 0x7f94c01425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:20.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.221+0000 7f94c7a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:20.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.221+0000 7f94c7a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94c0152400 0x7f94c0155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:20.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.221+0000 7f94c7a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0158b90 con 0x7f94c00c5040 2024-03-22T13:09:20.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.221+0000 7f94c7a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c01588b0 con 0x7f94c006f8c0 2024-03-22T13:09:20.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.221+0000 7f94c7a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c01527e0 con 0x7f94c0152400 2024-03-22T13:09:20.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.222+0000 7f94b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:20.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.222+0000 7f94c5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94c006f8c0 0x7f94c01425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:20.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.222+0000 7f94c5a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94c0152400 0x7f94c0155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:20.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.222+0000 7f94b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36252/0 (socket says 172.21.15.6:36252) 2024-03-22T13:09:20.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.222+0000 7f94b7e00640 1 -- 172.21.15.6:0/2342990671 learned_addr learned my addr 172.21.15.6:0/2342990671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:20.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.226+0000 7f94b7e00640 1 -- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94c0152400 msgr2=0x7f94c0155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:20.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.226+0000 7f94b7e00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94c0152400 0x7f94c0155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.226+0000 7f94b7e00640 1 -- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94c006f8c0 msgr2=0x7f94c01425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:20.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.226+0000 7f94b7e00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94c006f8c0 0x7f94c01425e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.226+0000 7f94b7e00640 1 -- 172.21.15.6:0/2342990671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c01ee870 con 0x7f94c00c5040 2024-03-22T13:09:20.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.229+0000 7f94b7e00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00ca8b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f94a800c400 tx=0x7f94a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:20.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.236+0000 7f94bd600640 1 -- 172.21.15.6:0/2342990671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a80052d0 con 0x7f94c00c5040 2024-03-22T13:09:20.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.236+0000 7f94bd600640 1 -- 172.21.15.6:0/2342990671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8005490 con 0x7f94c00c5040 2024-03-22T13:09:20.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.236+0000 7f94bd600640 1 -- 172.21.15.6:0/2342990671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8007260 con 0x7f94c00c5040 2024-03-22T13:09:20.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.237+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c01eeb30 con 0x7f94c00c5040 2024-03-22T13:09:20.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.237+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c0145790 con 0x7f94c00c5040 2024-03-22T13:09:20.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.238+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c0072d10 con 0x7f94c00c5040 2024-03-22T13:09:20.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.551+0000 7f94bd600640 1 -- 172.21.15.6:0/2342990671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f94a8007400 con 0x7f94c00c5040 2024-03-22T13:09:20.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.554+0000 7f94bd600640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f949c06bc20 0x7f949c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:20.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.554+0000 7f94c5000640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f949c06bc20 0x7f949c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.555+0000 7f94c5000640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f949c06bc20 0x7f949c06e0e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f94b00011e0 tx=0x7f94b0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.581+0000 7f94bd600640 1 -- 172.21.15.6:0/2342990671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f94a8056930 con 0x7f94c00c5040 2024-03-22T13:09:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.582+0000 7f94bd600640 1 -- 172.21.15.6:0/2342990671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f94c0072d10 con 0x7f94c00c5040 2024-03-22T13:09:20.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.837+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94c00756c0 con 0x7f94c00c5040 2024-03-22T13:09:20.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.858+0000 7f94bd600640 1 -- 172.21.15.6:0/2342990671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f94a8058890 con 0x7f94c00c5040 2024-03-22T13:09:20.858 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:20.858 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:20.858 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f949c06bc20 msgr2=0x7f949c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f949c06bc20 0x7f949c06e0e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f94b00011e0 tx=0x7f94b0007d30 comp rx=0 tx=0).stop 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 msgr2=0x7f94c00ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00ca8b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f94a800c400 tx=0x7f94a800c8c0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 shutdown_connections 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94c0152400 0x7f94c0155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c00c5040 0x7f94c00ca8b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f949c06bc20 0x7f949c06e0e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 --2- 172.21.15.6:0/2342990671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94c006f8c0 0x7f94c01425e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.860+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 >> 172.21.15.6:0/2342990671 conn(0x7f94c00cb930 msgr2=0x7f94c01443e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.861+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 shutdown_connections 2024-03-22T13:09:20.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:20.861+0000 7f94c7a00640 1 -- 172.21.15.6:0/2342990671 wait complete. 2024-03-22T13:09:21.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:21.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:22.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:22.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.683+0000 7f42b8a00640 1 Processor -- start 2024-03-22T13:09:22.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.683+0000 7f42b8a00640 1 -- start start 2024-03-22T13:09:22.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.684+0000 7f42b8a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f42b40cc640 0x7f42b406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.684+0000 7f42b8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b406dd00 con 0x7f42b40cca20 2024-03-22T13:09:22.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.684+0000 7f42b8a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b406c440 con 0x7f42b415b7a0 2024-03-22T13:09:22.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.684+0000 7f42b8a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b406d330 con 0x7f42b40cc640 2024-03-22T13:09:22.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.720+0000 7f42b1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42b40cca20 0x7f42b40ca340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58038/0 (socket says 172.21.15.6:58038) 2024-03-22T13:09:22.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.720+0000 7f42b1e00640 1 -- 172.21.15.6:0/1096736673 learned_addr learned my addr 172.21.15.6:0/1096736673 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:22.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.725+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170754451 0 0) 0x7f42b406dd00 con 0x7f42b40cca20 2024-03-22T13:09:22.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.725+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4298003520 con 0x7f42b40cca20 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1559723498 0 0) 0x7f4298003520 con 0x7f42b40cca20 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 >> v2:172.21.15.92:3300/0 conn(0x7f42b40cc640 msgr2=0x7f42b406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 --2- 172.21.15.6:0/1096736673 >> v2:172.21.15.92:3300/0 conn(0x7f42b40cc640 0x7f42b406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 >> v1:172.21.15.92:6789/0 conn(0x7f42b415b7a0 legacy=0x7f42b4164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b406c720 con 0x7f42b40cca20 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f42a0001670 con 0x7f42b40cca20 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 >> v1:172.21.15.6:6789/0 conn(0x7f42b40cca20 legacy=0x7f42b40ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.731+0000 7f42b1400640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f4298003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.732+0000 7f42b1400640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4298004800 0x7f4298004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.732+0000 7f42b1400640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42980053c0 0x7f429800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.732+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b406d330 con 0x7f4298004800 2024-03-22T13:09:22.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.732+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429800c220 con 0x7f4298003a00 2024-03-22T13:09:22.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.732+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429800c420 con 0x7f42980053c0 2024-03-22T13:09:22.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.737+0000 7f42b2800640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f4298003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:22.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.740+0000 7f42b1e00640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4298004800 0x7f4298004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:22.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.743+0000 7f42b2800640 1 -- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42980053c0 msgr2=0x7f429800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:22.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.743+0000 7f42b2800640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42980053c0 0x7f429800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:22.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.743+0000 7f42b2800640 1 -- 172.21.15.6:0/1096736673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4298004800 msgr2=0x7f4298004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:22.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.743+0000 7f42b2800640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4298004800 0x7f4298004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:22.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.743+0000 7f42b2800640 1 -- 172.21.15.6:0/1096736673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298003520 con 0x7f4298003a00 2024-03-22T13:09:22.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.743+0000 7f42b3200640 1 -- 172.21.15.6:0/1096736673 reap_dead start 2024-03-22T13:09:22.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.746+0000 7f42b2800640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f4298003e20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009860 tx=0x7f4298005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40e10ed7eba7d9f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:22.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.749+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4037070 con 0x7f4298003a00 2024-03-22T13:09:22.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.752+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a4032620 con 0x7f4298003a00 2024-03-22T13:09:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.752+0000 7f42b1400640 1 -- 172.21.15.6:0/1096736673 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4009460 con 0x7f4298003a00 2024-03-22T13:09:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.753+0000 7f42b8a00640 1 -- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 msgr2=0x7f4298003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:22.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.753+0000 7f42b8a00640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f4298003e20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009860 tx=0x7f4298005830 comp rx=0 tx=0).stop 2024-03-22T13:09:22.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.753+0000 7f42b8a00640 1 -- 172.21.15.6:0/1096736673 shutdown_connections 2024-03-22T13:09:22.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.753+0000 7f42b8a00640 1 --2- 172.21.15.6:0/1096736673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f4298003e20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:22.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.753+0000 7f42b8a00640 1 -- 172.21.15.6:0/1096736673 >> 172.21.15.6:0/1096736673 conn(0x7f42b40cf9c0 msgr2=0x7f42b40cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:22.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.753+0000 7f42b8a00640 1 -- 172.21.15.6:0/1096736673 shutdown_connections 2024-03-22T13:09:22.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.753+0000 7f42b8a00640 1 -- 172.21.15.6:0/1096736673 wait complete. 2024-03-22T13:09:22.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.755+0000 7f42b8a00640 1 Processor -- start 2024-03-22T13:09:22.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.755+0000 7f42b8a00640 1 -- start start 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.756+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f42b41f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.756+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42b4157900 0x7f42b41e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.756+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42b4153d10 0x7f42b4154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.756+0000 7f42b8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b406cfc0 con 0x7f42b4153d10 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.756+0000 7f42b8a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b406cce0 con 0x7f4298003a00 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.756+0000 7f42b8a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b415b000 con 0x7f42b4157900 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.757+0000 7f42b3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42b4153d10 0x7f42b4154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.757+0000 7f42b2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f42b41f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.757+0000 7f42b3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42b4153d10 0x7f42b4154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36272/0 (socket says 172.21.15.6:36272) 2024-03-22T13:09:22.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.757+0000 7f42b3200640 1 -- 172.21.15.6:0/528753803 learned_addr learned my addr 172.21.15.6:0/528753803 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:22.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.761+0000 7f42b3200640 1 -- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42b4157900 msgr2=0x7f42b41e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:22.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.761+0000 7f42b3200640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42b4157900 0x7f42b41e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:22.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.761+0000 7f42b3200640 1 -- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 msgr2=0x7f42b41f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:22.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.761+0000 7f42b3200640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f42b41f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:22.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.761+0000 7f42b3200640 1 -- 172.21.15.6:0/528753803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b41f0520 con 0x7f42b4153d10 2024-03-22T13:09:22.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.764+0000 7f42b3200640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42b4153d10 0x7f42b4154190 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f42b408db90 tx=0x7f42ac00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:22.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.768+0000 7f42aaa00640 1 -- 172.21.15.6:0/528753803 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac007110 con 0x7f42b4153d10 2024-03-22T13:09:22.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.768+0000 7f42aaa00640 1 -- 172.21.15.6:0/528753803 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ac0072d0 con 0x7f42b4153d10 2024-03-22T13:09:22.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.768+0000 7f42aaa00640 1 -- 172.21.15.6:0/528753803 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0076c0 con 0x7f42b4153d10 2024-03-22T13:09:22.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.768+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b41f0800 con 0x7f42b4153d10 2024-03-22T13:09:22.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.768+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b41f0dc0 con 0x7f42b4153d10 2024-03-22T13:09:22.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.770+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b415ade0 con 0x7f42b4153d10 2024-03-22T13:09:22.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.792+0000 7f42aaa00640 1 -- 172.21.15.6:0/528753803 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f42ac007860 con 0x7f42b4153d10 2024-03-22T13:09:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.793+0000 7f42aaa00640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f429406bc20 0x7f429406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:22.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.793+0000 7f42b2800640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f429406bc20 0x7f429406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:22.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.794+0000 7f42b2800640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f429406bc20 0x7f429406e0e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f42a4034000 tx=0x7f42a4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:22.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:22.795+0000 7f42aaa00640 1 -- 172.21.15.6:0/528753803 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f42ac0568f0 con 0x7f42b4153d10 2024-03-22T13:09:23.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.071+0000 7f42aaa00640 1 -- 172.21.15.6:0/528753803 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f42b415ade0 con 0x7f42b4153d10 2024-03-22T13:09:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.343+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42b4075650 con 0x7f42b4153d10 2024-03-22T13:09:23.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.364+0000 7f42aaa00640 1 -- 172.21.15.6:0/528753803 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7f42ac058f50 con 0x7f42b4153d10 2024-03-22T13:09:23.364 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:23.365 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:23.365 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f429406bc20 msgr2=0x7f429406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f429406bc20 0x7f429406e0e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f42a4034000 tx=0x7f42a4008030 comp rx=0 tx=0).stop 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42b4153d10 msgr2=0x7f42b4154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42b4153d10 0x7f42b4154190 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f42b408db90 tx=0x7f42ac00c960 comp rx=0 tx=0).stop 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 shutdown_connections 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42b4153d10 0x7f42b4154190 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42b4157900 0x7f42b41e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f429406bc20 0x7f429406e0e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 --2- 172.21.15.6:0/528753803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4298003a00 0x7f42b41f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 >> 172.21.15.6:0/528753803 conn(0x7f42b40cf9c0 msgr2=0x7f42b4158500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:23.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 shutdown_connections 2024-03-22T13:09:23.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:23.367+0000 7f42b8a00640 1 -- 172.21.15.6:0/528753803 wait complete. 2024-03-22T13:09:23.993 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:23.993 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:24.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:25.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.200+0000 7fc42fc00640 1 Processor -- start 2024-03-22T13:09:25.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.200+0000 7fc42fc00640 1 -- start start 2024-03-22T13:09:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.202+0000 7fc42fc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4280c9c30 0x7fc4280c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.202+0000 7fc42fc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42815a3c0 con 0x7fc4281545a0 2024-03-22T13:09:25.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.202+0000 7fc42fc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280c4130 con 0x7fc4280c96e0 2024-03-22T13:09:25.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.202+0000 7fc42fc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4280c4410 con 0x7fc4280c9c30 2024-03-22T13:09:25.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.203+0000 7fc42dc00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc4280c96e0 0x7fc428164bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37756/0 (socket says 172.21.15.6:37756) 2024-03-22T13:09:25.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.203+0000 7fc42dc00640 1 -- 172.21.15.6:0/2758987487 learned_addr learned my addr 172.21.15.6:0/2758987487 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:25.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.204+0000 7fc427e00640 1 --2- 172.21.15.6:0/2758987487 >> v2:172.21.15.6:3300/0 conn(0x7fc4280c9c30 0x7fc4280c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:25.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.209+0000 7fc427e00640 1 -- 172.21.15.6:0/2758987487 >> v1:172.21.15.92:6789/0 conn(0x7fc4280c96e0 legacy=0x7fc428164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:25.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.209+0000 7fc427e00640 1 -- 172.21.15.6:0/2758987487 >> v1:172.21.15.6:6789/0 conn(0x7fc4281545a0 legacy=0x7fc4280ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:25.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.209+0000 7fc427e00640 1 -- 172.21.15.6:0/2758987487 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42815a660 con 0x7fc4280c9c30 2024-03-22T13:09:25.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.216+0000 7fc427e00640 1 --2- 172.21.15.6:0/2758987487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c9c30 0x7fc4280c3af0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4140097a0 tx=0x7fc41402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e89bb5dcee9084 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:25.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.221+0000 7fc427400640 1 -- 172.21.15.6:0/2758987487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41402f810 con 0x7fc4280c9c30 2024-03-22T13:09:25.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.221+0000 7fc427400640 1 -- 172.21.15.6:0/2758987487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41402f9b0 con 0x7fc4280c9c30 2024-03-22T13:09:25.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.221+0000 7fc427400640 1 -- 172.21.15.6:0/2758987487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41402fd80 con 0x7fc4280c9c30 2024-03-22T13:09:25.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.222+0000 7fc42fc00640 1 -- 172.21.15.6:0/2758987487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c9c30 msgr2=0x7fc4280c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:25.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.222+0000 7fc42fc00640 1 --2- 172.21.15.6:0/2758987487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c9c30 0x7fc4280c3af0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4140097a0 tx=0x7fc41402b9f0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.222+0000 7fc42fc00640 1 -- 172.21.15.6:0/2758987487 shutdown_connections 2024-03-22T13:09:25.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.222+0000 7fc42fc00640 1 --2- 172.21.15.6:0/2758987487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c9c30 0x7fc4280c3af0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.222+0000 7fc42fc00640 1 -- 172.21.15.6:0/2758987487 >> 172.21.15.6:0/2758987487 conn(0x7fc4280714f0 msgr2=0x7fc428071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:25.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.222+0000 7fc42fc00640 1 -- 172.21.15.6:0/2758987487 shutdown_connections 2024-03-22T13:09:25.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.222+0000 7fc42fc00640 1 -- 172.21.15.6:0/2758987487 wait complete. 2024-03-22T13:09:25.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.224+0000 7fc42fc00640 1 Processor -- start 2024-03-22T13:09:25.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.224+0000 7fc42fc00640 1 -- start start 2024-03-22T13:09:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.225+0000 7fc42fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c96e0 0x7fc4280adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.225+0000 7fc42fc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4280c9c30 0x7fc4280c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.225+0000 7fc42fc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4281545a0 0x7fc4280c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.225+0000 7fc42fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc428154980 con 0x7fc4280c96e0 2024-03-22T13:09:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.225+0000 7fc42fc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280c46f0 con 0x7fc4280c9c30 2024-03-22T13:09:25.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.225+0000 7fc42fc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280cadc0 con 0x7fc4281545a0 2024-03-22T13:09:25.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.226+0000 7fc427e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4280c9c30 0x7fc4280c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:25.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.226+0000 7fc42dc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4281545a0 0x7fc4280c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:25.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.226+0000 7fc42d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c96e0 0x7fc4280adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:25.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.227+0000 7fc42d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c96e0 0x7fc4280adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34454/0 (socket says 172.21.15.6:34454) 2024-03-22T13:09:25.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.227+0000 7fc42d200640 1 -- 172.21.15.6:0/1101805395 learned_addr learned my addr 172.21.15.6:0/1101805395 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:25.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.230+0000 7fc42dc00640 1 -- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4280c9c30 msgr2=0x7fc4280c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:25.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.230+0000 7fc42dc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4280c9c30 0x7fc4280c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.230+0000 7fc42dc00640 1 -- 172.21.15.6:0/1101805395 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c96e0 msgr2=0x7fc4280adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:25.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.230+0000 7fc42dc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c96e0 0x7fc4280adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.230+0000 7fc42dc00640 1 -- 172.21.15.6:0/1101805395 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280cd470 con 0x7fc4281545a0 2024-03-22T13:09:25.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.232+0000 7fc42dc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4281545a0 0x7fc4280c6d10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc42000cad0 tx=0x7fc42000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:25.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.236+0000 7fc424c00640 1 -- 172.21.15.6:0/1101805395 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420017070 con 0x7fc4281545a0 2024-03-22T13:09:25.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.236+0000 7fc424c00640 1 -- 172.21.15.6:0/1101805395 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42000ab70 con 0x7fc4281545a0 2024-03-22T13:09:25.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.236+0000 7fc424c00640 1 -- 172.21.15.6:0/1101805395 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420012400 con 0x7fc4281545a0 2024-03-22T13:09:25.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.236+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4280cd750 con 0x7fc4281545a0 2024-03-22T13:09:25.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.237+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4280c5830 con 0x7fc4281545a0 2024-03-22T13:09:25.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.238+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4280756c0 con 0x7fc4281545a0 2024-03-22T13:09:25.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.259+0000 7fc424c00640 1 -- 172.21.15.6:0/1101805395 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc42001a070 con 0x7fc4281545a0 2024-03-22T13:09:25.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.260+0000 7fc424c00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc40806bc20 0x7fc40806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:25.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.260+0000 7fc42d200640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc40806bc20 0x7fc40806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:25.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.261+0000 7fc42d200640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc40806bc20 0x7fc40806e0e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0017a0 tx=0x7fc41c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:25.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.263+0000 7fc424c00640 1 -- 172.21.15.6:0/1101805395 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fc4200573a0 con 0x7fc4281545a0 2024-03-22T13:09:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.518+0000 7fc424c00640 1 -- 172.21.15.6:0/1101805395 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc420059300 con 0x7fc4281545a0 2024-03-22T13:09:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.770+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4280c9410 con 0x7fc4281545a0 2024-03-22T13:09:25.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.823+0000 7fc424c00640 1 -- 172.21.15.6:0/1101805395 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fc42005d3a0 con 0x7fc4281545a0 2024-03-22T13:09:25.823 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:25.823 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc40806bc20 msgr2=0x7fc40806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc40806bc20 0x7fc40806e0e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0017a0 tx=0x7fc41c009040 comp rx=0 tx=0).stop 2024-03-22T13:09:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4281545a0 msgr2=0x7fc4280c6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4281545a0 0x7fc4280c6d10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc42000cad0 tx=0x7fc42000cf90 comp rx=0 tx=0).stop 2024-03-22T13:09:25.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 shutdown_connections 2024-03-22T13:09:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4281545a0 0x7fc4280c6d10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fc40806bc20 0x7fc40806e0e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4280c9c30 0x7fc4280c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 --2- 172.21.15.6:0/1101805395 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4280c96e0 0x7fc4280adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 >> 172.21.15.6:0/1101805395 conn(0x7fc4280714f0 msgr2=0x7fc428159010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 shutdown_connections 2024-03-22T13:09:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:25.826+0000 7fc42fc00640 1 -- 172.21.15.6:0/1101805395 wait complete. 2024-03-22T13:09:26.448 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:26.448 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:27.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:27.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.647+0000 7fb6a1400640 1 Processor -- start 2024-03-22T13:09:27.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.647+0000 7fb6a1400640 1 -- start start 2024-03-22T13:09:27.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.648+0000 7fb6a1400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:27.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.648+0000 7fb6a1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb69c0cf5c0 0x7fb69c0ca510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:27.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.648+0000 7fb6a1400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c15a6b0 con 0x7fb69c0caa50 2024-03-22T13:09:27.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.648+0000 7fb6a1400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69c154d60 con 0x7fb69c0cf5c0 2024-03-22T13:09:27.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.648+0000 7fb6a1400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c15b4d0 con 0x7fb69c0ccd80 2024-03-22T13:09:27.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.649+0000 7fb69a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:27.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.650+0000 7fb69a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36510/0 (socket says 172.21.15.6:36510) 2024-03-22T13:09:27.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.650+0000 7fb69a800640 1 -- 172.21.15.6:0/882648864 learned_addr learned my addr 172.21.15.6:0/882648864 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:27.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.650+0000 7fb699e00640 1 --2- 172.21.15.6:0/882648864 >> v2:172.21.15.6:3300/0 conn(0x7fb69c0cf5c0 0x7fb69c0ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:27.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.653+0000 7fb69a800640 1 -- 172.21.15.6:0/882648864 >> v2:172.21.15.6:3300/0 conn(0x7fb69c0cf5c0 msgr2=0x7fb69c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:27.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.653+0000 7fb69a800640 1 --2- 172.21.15.6:0/882648864 >> v2:172.21.15.6:3300/0 conn(0x7fb69c0cf5c0 0x7fb69c0ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:27.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.653+0000 7fb69a800640 1 -- 172.21.15.6:0/882648864 >> v1:172.21.15.6:6789/0 conn(0x7fb69c0caa50 legacy=0x7fb69c06c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:27.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.653+0000 7fb69a800640 1 -- 172.21.15.6:0/882648864 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c15a950 con 0x7fb69c0ccd80 2024-03-22T13:09:27.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.656+0000 7fb69a800640 1 --2- 172.21.15.6:0/882648864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0cf080 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009870 tx=0x7fb68c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b99c21a88c6ff4e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:27.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.661+0000 7fb699400640 1 -- 172.21.15.6:0/882648864 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c034070 con 0x7fb69c0ccd80 2024-03-22T13:09:27.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.661+0000 7fb699400640 1 -- 172.21.15.6:0/882648864 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c030840 con 0x7fb69c0ccd80 2024-03-22T13:09:27.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.661+0000 7fb699400640 1 -- 172.21.15.6:0/882648864 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c030c10 con 0x7fb69c0ccd80 2024-03-22T13:09:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.662+0000 7fb6a1400640 1 -- 172.21.15.6:0/882648864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 msgr2=0x7fb69c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.662+0000 7fb6a1400640 1 --2- 172.21.15.6:0/882648864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0cf080 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009870 tx=0x7fb68c02b980 comp rx=0 tx=0).stop 2024-03-22T13:09:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.662+0000 7fb6a1400640 1 -- 172.21.15.6:0/882648864 shutdown_connections 2024-03-22T13:09:27.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.662+0000 7fb6a1400640 1 --2- 172.21.15.6:0/882648864 >> v2:172.21.15.6:3300/0 conn(0x7fb69c0cf5c0 0x7fb69c0ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:27.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.662+0000 7fb6a1400640 1 --2- 172.21.15.6:0/882648864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0cf080 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:27.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.662+0000 7fb6a1400640 1 -- 172.21.15.6:0/882648864 >> 172.21.15.6:0/882648864 conn(0x7fb69c0b16c0 msgr2=0x7fb69c0b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:27.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.662+0000 7fb6a1400640 1 -- 172.21.15.6:0/882648864 shutdown_connections 2024-03-22T13:09:27.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.663+0000 7fb6a1400640 1 -- 172.21.15.6:0/882648864 wait complete. 2024-03-22T13:09:27.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.665+0000 7fb6a1400640 1 Processor -- start 2024-03-22T13:09:27.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.665+0000 7fb6a1400640 1 -- start start 2024-03-22T13:09:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.667+0000 7fb6a1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb69c0caa50 0x7fb69c0c3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.667+0000 7fb6a1400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.667+0000 7fb6a1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb69c0cf5c0 0x7fb69c152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.667+0000 7fb6a1400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c15b1f0 con 0x7fb69c0caa50 2024-03-22T13:09:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.667+0000 7fb6a1400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c15af10 con 0x7fb69c0cf5c0 2024-03-22T13:09:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.667+0000 7fb6a1400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c06db10 con 0x7fb69c0ccd80 2024-03-22T13:09:27.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.668+0000 7fb699e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:27.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.668+0000 7fb69a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb69c0caa50 0x7fb69c0c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:27.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.668+0000 7fb69b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb69c0cf5c0 0x7fb69c152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:27.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.668+0000 7fb699e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36518/0 (socket says 172.21.15.6:36518) 2024-03-22T13:09:27.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.668+0000 7fb699e00640 1 -- 172.21.15.6:0/3675948364 learned_addr learned my addr 172.21.15.6:0/3675948364 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:27.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.672+0000 7fb699e00640 1 -- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb69c0cf5c0 msgr2=0x7fb69c152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.672+0000 7fb699e00640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb69c0cf5c0 0x7fb69c152760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.672+0000 7fb699e00640 1 -- 172.21.15.6:0/3675948364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb69c0caa50 msgr2=0x7fb69c0c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.672+0000 7fb699e00640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb69c0caa50 0x7fb69c0c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.672+0000 7fb699e00640 1 -- 172.21.15.6:0/3675948364 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c1f0760 con 0x7fb69c0ccd80 2024-03-22T13:09:27.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.675+0000 7fb699e00640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0c8150 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e8a0 tx=0x7fb68800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.679+0000 7fb692a00640 1 -- 172.21.15.6:0/3675948364 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688017070 con 0x7fb69c0ccd80 2024-03-22T13:09:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.679+0000 7fb692a00640 1 -- 172.21.15.6:0/3675948364 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68800c940 con 0x7fb69c0ccd80 2024-03-22T13:09:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.679+0000 7fb692a00640 1 -- 172.21.15.6:0/3675948364 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68800cd30 con 0x7fb69c0ccd80 2024-03-22T13:09:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.679+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c1f0a40 con 0x7fb69c0ccd80 2024-03-22T13:09:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.679+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c1f1000 con 0x7fb69c0ccd80 2024-03-22T13:09:27.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.680+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c06d810 con 0x7fb69c0ccd80 2024-03-22T13:09:27.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.700+0000 7fb692a00640 1 -- 172.21.15.6:0/3675948364 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb68801a070 con 0x7fb69c0ccd80 2024-03-22T13:09:27.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.701+0000 7fb692a00640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb67c06bbd0 0x7fb67c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:27.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.702+0000 7fb69a800640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb67c06bbd0 0x7fb67c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.702+0000 7fb69a800640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb67c06bbd0 0x7fb67c06e090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb68c036000 tx=0x7fb68c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:27.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.943+0000 7fb692a00640 1 -- 172.21.15.6:0/3675948364 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb688057590 con 0x7fb69c0ccd80 2024-03-22T13:09:27.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:27.955+0000 7fb692a00640 1 -- 172.21.15.6:0/3675948364 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb69c06d810 con 0x7fb69c0ccd80 2024-03-22T13:09:28.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.224+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb69c0c9020 con 0x7fb69c0ccd80 2024-03-22T13:09:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.237+0000 7fb692a00640 1 -- 172.21.15.6:0/3675948364 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5981 (secure 0 0 0) 0x7fb6880594f0 con 0x7fb69c0ccd80 2024-03-22T13:09:28.237 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:28.237 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:08:28.701005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4268,4261,4265,4252]},"id":1}]} 2024-03-22T13:09:28.238 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-22T13:09:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb67c06bbd0 msgr2=0x7fb67c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb67c06bbd0 0x7fb67c06e090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb68c036000 tx=0x7fb68c008030 comp rx=0 tx=0).stop 2024-03-22T13:09:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 msgr2=0x7fb69c0c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0c8150 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e8a0 tx=0x7fb68800ed60 comp rx=0 tx=0).stop 2024-03-22T13:09:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 shutdown_connections 2024-03-22T13:09:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb67c06bbd0 0x7fb67c06e090 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb69c0cf5c0 0x7fb69c152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:28.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb69c0ccd80 0x7fb69c0c8150 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:28.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 --2- 172.21.15.6:0/3675948364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb69c0caa50 0x7fb69c0c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:28.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 >> 172.21.15.6:0/3675948364 conn(0x7fb69c0b16c0 msgr2=0x7fb69c1e6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:28.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 shutdown_connections 2024-03-22T13:09:28.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:28.240+0000 7fb6a1400640 1 -- 172.21.15.6:0/3675948364 wait complete. 2024-03-22T13:09:28.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:08:28.701005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4268, 4261, 4265, 4252]} 2024-03-22T13:09:28.861 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:29.445 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:13:47.780 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.505 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:47.895 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.507 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:47.897 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.540 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:13:47.929 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.544 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:13:47.879 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.547 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:13:47.937 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.587 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:47.978 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.682 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:13:48.017 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.688 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:13:48.078 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:09:29.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:30.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.048+0000 7f5567e00640 1 Processor -- start 2024-03-22T13:09:30.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.048+0000 7f5567e00640 1 -- start start 2024-03-22T13:09:30.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.049+0000 7f5567e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5560152400 0x7f556015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:30.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.049+0000 7f5567e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556006f8c0 0x7f55600c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:30.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.049+0000 7f5567e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55601590b0 con 0x7f556015bc80 2024-03-22T13:09:30.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.049+0000 7f5567e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55600724d0 con 0x7f5560152400 2024-03-22T13:09:30.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.049+0000 7f5567e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55600721f0 con 0x7f556006f8c0 2024-03-22T13:09:30.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.053+0000 7f5565400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5560152400 0x7f556015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:30.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.053+0000 7f5565e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556006f8c0 0x7f55600c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:30.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.054+0000 7f5565400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5560152400 0x7f556015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:58990/0 (socket says 172.21.15.6:58990) 2024-03-22T13:09:30.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.054+0000 7f5565400640 1 -- 172.21.15.6:0/398667730 learned_addr learned my addr 172.21.15.6:0/398667730 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:30.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.062+0000 7f5565e00640 1 -- 172.21.15.6:0/398667730 >> v2:172.21.15.92:3300/0 conn(0x7f5560152400 msgr2=0x7f556015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:30.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.062+0000 7f5565e00640 1 --2- 172.21.15.6:0/398667730 >> v2:172.21.15.92:3300/0 conn(0x7f5560152400 0x7f556015b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.062+0000 7f5565e00640 1 -- 172.21.15.6:0/398667730 >> v1:172.21.15.92:6789/0 conn(0x7f556015bc80 legacy=0x7f55600716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:30.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.062+0000 7f5565e00640 1 -- 172.21.15.6:0/398667730 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55601582f0 con 0x7f556006f8c0 2024-03-22T13:09:30.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.069+0000 7f5565e00640 1 --2- 172.21.15.6:0/398667730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556006f8c0 0x7f55600c5040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5558009d90 tx=0x7f555802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17c44a2bd0b24ec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.078+0000 7f555fe00640 1 -- 172.21.15.6:0/398667730 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5558039070 con 0x7f556006f8c0 2024-03-22T13:09:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.078+0000 7f555fe00640 1 -- 172.21.15.6:0/398667730 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555802cce0 con 0x7f556006f8c0 2024-03-22T13:09:30.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.078+0000 7f555fe00640 1 -- 172.21.15.6:0/398667730 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5558034660 con 0x7f556006f8c0 2024-03-22T13:09:30.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.079+0000 7f5567e00640 1 -- 172.21.15.6:0/398667730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556006f8c0 msgr2=0x7f55600c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:30.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.079+0000 7f5567e00640 1 --2- 172.21.15.6:0/398667730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556006f8c0 0x7f55600c5040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5558009d90 tx=0x7f555802bf30 comp rx=0 tx=0).stop 2024-03-22T13:09:30.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.080+0000 7f5567e00640 1 -- 172.21.15.6:0/398667730 shutdown_connections 2024-03-22T13:09:30.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.080+0000 7f5567e00640 1 --2- 172.21.15.6:0/398667730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556006f8c0 0x7f55600c5040 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.080+0000 7f5567e00640 1 --2- 172.21.15.6:0/398667730 >> v2:172.21.15.92:3300/0 conn(0x7f5560152400 0x7f556015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.080+0000 7f5567e00640 1 -- 172.21.15.6:0/398667730 >> 172.21.15.6:0/398667730 conn(0x7f55600cb930 msgr2=0x7f55600cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:30.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.080+0000 7f5567e00640 1 -- 172.21.15.6:0/398667730 shutdown_connections 2024-03-22T13:09:30.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.080+0000 7f5567e00640 1 -- 172.21.15.6:0/398667730 wait complete. 2024-03-22T13:09:30.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.082+0000 7f5567e00640 1 Processor -- start 2024-03-22T13:09:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.082+0000 7f5567e00640 1 -- start start 2024-03-22T13:09:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.083+0000 7f5567e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556006f8c0 0x7f556015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.083+0000 7f5567e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5560152400 0x7f55600c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.083+0000 7f5567e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556015bc80 0x7f55601483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.083+0000 7f5567e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560158b90 con 0x7f5560152400 2024-03-22T13:09:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.083+0000 7f5567e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55601588b0 con 0x7f556006f8c0 2024-03-22T13:09:30.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.083+0000 7f5567e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55601527e0 con 0x7f556015bc80 2024-03-22T13:09:30.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.086+0000 7f5565400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556006f8c0 0x7f556015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:30.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.086+0000 7f5564a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5560152400 0x7f55600c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:30.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.087+0000 7f5565400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556006f8c0 0x7f556015b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:59006/0 (socket says 172.21.15.6:59006) 2024-03-22T13:09:30.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.087+0000 7f5565400640 1 -- 172.21.15.6:0/2189057412 learned_addr learned my addr 172.21.15.6:0/2189057412 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:30.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.087+0000 7f5565e00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556015bc80 0x7f55601483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.094+0000 7f5565400640 1 -- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556015bc80 msgr2=0x7f55601483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.094+0000 7f5565400640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556015bc80 0x7f55601483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.094+0000 7f5565400640 1 -- 172.21.15.6:0/2189057412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5560152400 msgr2=0x7f55600c3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.094+0000 7f5565400640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5560152400 0x7f55600c3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.094+0000 7f5565400640 1 -- 172.21.15.6:0/2189057412 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560159310 con 0x7f556006f8c0 2024-03-22T13:09:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.102+0000 7f5565400640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556006f8c0 0x7f556015b470 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f555400e8b0 tx=0x7f555400ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:30.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.114+0000 7f555d600640 1 -- 172.21.15.6:0/2189057412 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554017070 con 0x7f556006f8c0 2024-03-22T13:09:30.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.114+0000 7f555d600640 1 -- 172.21.15.6:0/2189057412 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555400c940 con 0x7f556006f8c0 2024-03-22T13:09:30.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.114+0000 7f555d600640 1 -- 172.21.15.6:0/2189057412 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555400cd10 con 0x7f556006f8c0 2024-03-22T13:09:30.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.114+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556014cb70 con 0x7f556006f8c0 2024-03-22T13:09:30.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.115+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55601598b0 con 0x7f556006f8c0 2024-03-22T13:09:30.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.116+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55600756c0 con 0x7f556006f8c0 2024-03-22T13:09:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.448+0000 7f555d600640 1 -- 172.21.15.6:0/2189057412 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f555401a070 con 0x7f556006f8c0 2024-03-22T13:09:30.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.450+0000 7f555d600640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554006bc20 0x7f554006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:30.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.451+0000 7f555d600640 1 -- 172.21.15.6:0/2189057412 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5554057690 con 0x7f556006f8c0 2024-03-22T13:09:30.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.451+0000 7f5564a00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554006bc20 0x7f554006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:30.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.452+0000 7f5564a00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554006bc20 0x7f554006e0e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f55600c4fc0 tx=0x7f554c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:30.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.465+0000 7f555d600640 1 -- 172.21.15.6:0/2189057412 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f55540595f0 con 0x7f556006f8c0 2024-03-22T13:09:30.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.714+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5560073080 con 0x7f556006f8c0 2024-03-22T13:09:30.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.729+0000 7f555d600640 1 -- 172.21.15.6:0/2189057412 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f555405d690 con 0x7f556006f8c0 2024-03-22T13:09:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:30.730 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:30.730 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.732+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554006bc20 msgr2=0x7f554006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.732+0000 7f5567e00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554006bc20 0x7f554006e0e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f55600c4fc0 tx=0x7f554c009040 comp rx=0 tx=0).stop 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556006f8c0 msgr2=0x7f556015b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556006f8c0 0x7f556015b470 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f555400e8b0 tx=0x7f555400ed70 comp rx=0 tx=0).stop 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 shutdown_connections 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f556015bc80 0x7f55601483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5560152400 0x7f55600c3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f554006bc20 0x7f554006e0e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 --2- 172.21.15.6:0/2189057412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f556006f8c0 0x7f556015b470 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 >> 172.21.15.6:0/2189057412 conn(0x7f55600cb930 msgr2=0x7f55600c24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:30.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 shutdown_connections 2024-03-22T13:09:30.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:30.733+0000 7f5567e00640 1 -- 172.21.15.6:0/2189057412 wait complete. 2024-03-22T13:09:31.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:31.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:32.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:32.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.558+0000 7ff999000640 1 Processor -- start 2024-03-22T13:09:32.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.558+0000 7ff999000640 1 -- start start 2024-03-22T13:09:32.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.559+0000 7ff999000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff99406c230 0x7ff99415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:32.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.559+0000 7ff999000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99406c640 con 0x7ff9940ce8a0 2024-03-22T13:09:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.559+0000 7ff999000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9940cee00 con 0x7ff9940cc380 2024-03-22T13:09:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.559+0000 7ff999000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9940cf0c0 con 0x7ff99406c230 2024-03-22T13:09:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.560+0000 7ff993400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff9940cc380 0x7ff9940cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37780/0 (socket says 172.21.15.6:37780) 2024-03-22T13:09:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.560+0000 7ff993400640 1 -- 172.21.15.6:0/2350827502 learned_addr learned my addr 172.21.15.6:0/2350827502 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:32.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.561+0000 7ff993e00640 1 --2- 172.21.15.6:0/2350827502 >> v2:172.21.15.92:3300/0 conn(0x7ff99406c230 0x7ff99415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:32.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.564+0000 7ff992000640 1 -- 172.21.15.6:0/2350827502 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171547538 0 0) 0x7ff980001110 con 0x7ff9940ce8a0 2024-03-22T13:09:32.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.564+0000 7ff992000640 1 -- 172.21.15.6:0/2350827502 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff978003520 con 0x7ff9940ce8a0 2024-03-22T13:09:32.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.564+0000 7ff992000640 1 -- 172.21.15.6:0/2350827502 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417748178 0 0) 0x7ff9940cee00 con 0x7ff9940cc380 2024-03-22T13:09:32.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.564+0000 7ff992000640 1 -- 172.21.15.6:0/2350827502 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff980001110 con 0x7ff9940cc380 2024-03-22T13:09:32.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.566+0000 7ff993e00640 1 -- 172.21.15.6:0/2350827502 >> v1:172.21.15.92:6789/0 conn(0x7ff9940cc380 legacy=0x7ff9940cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:32.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.566+0000 7ff993e00640 1 -- 172.21.15.6:0/2350827502 >> v1:172.21.15.6:6789/0 conn(0x7ff9940ce8a0 legacy=0x7ff994162b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:32.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.566+0000 7ff993e00640 1 -- 172.21.15.6:0/2350827502 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940cf3a0 con 0x7ff99406c230 2024-03-22T13:09:32.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.570+0000 7ff993e00640 1 --2- 172.21.15.6:0/2350827502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99415ae30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff9940cbef0 tx=0x7ff98c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a0bf51425dfb864 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:32.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.575+0000 7ff992000640 1 -- 172.21.15.6:0/2350827502 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c039070 con 0x7ff99406c230 2024-03-22T13:09:32.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.575+0000 7ff992000640 1 -- 172.21.15.6:0/2350827502 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02cce0 con 0x7ff99406c230 2024-03-22T13:09:32.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.575+0000 7ff992000640 1 -- 172.21.15.6:0/2350827502 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c034660 con 0x7ff99406c230 2024-03-22T13:09:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.576+0000 7ff999000640 1 -- 172.21.15.6:0/2350827502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 msgr2=0x7ff99415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.576+0000 7ff999000640 1 --2- 172.21.15.6:0/2350827502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99415ae30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff9940cbef0 tx=0x7ff98c02bf30 comp rx=0 tx=0).stop 2024-03-22T13:09:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.576+0000 7ff999000640 1 -- 172.21.15.6:0/2350827502 shutdown_connections 2024-03-22T13:09:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.576+0000 7ff999000640 1 --2- 172.21.15.6:0/2350827502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99415ae30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.576+0000 7ff999000640 1 -- 172.21.15.6:0/2350827502 >> 172.21.15.6:0/2350827502 conn(0x7ff9940b1480 msgr2=0x7ff9940b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.576+0000 7ff999000640 1 -- 172.21.15.6:0/2350827502 shutdown_connections 2024-03-22T13:09:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.576+0000 7ff999000640 1 -- 172.21.15.6:0/2350827502 wait complete. 2024-03-22T13:09:32.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.578+0000 7ff999000640 1 Processor -- start 2024-03-22T13:09:32.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.578+0000 7ff999000640 1 -- start start 2024-03-22T13:09:32.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.579+0000 7ff999000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99414df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:32.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.579+0000 7ff999000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9940cc380 0x7ff9941f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:32.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.579+0000 7ff999000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9940ce8a0 0x7ff9941521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:32.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.579+0000 7ff999000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99415a910 con 0x7ff9940cc380 2024-03-22T13:09:32.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.579+0000 7ff999000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99415a630 con 0x7ff99406c230 2024-03-22T13:09:32.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.579+0000 7ff999000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9941549d0 con 0x7ff9940ce8a0 2024-03-22T13:09:32.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.580+0000 7ff993400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99414df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:32.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.580+0000 7ff992a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9940cc380 0x7ff9941f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:32.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.580+0000 7ff993e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9940ce8a0 0x7ff9941521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:32.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.580+0000 7ff993400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99414df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:59010/0 (socket says 172.21.15.6:59010) 2024-03-22T13:09:32.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.580+0000 7ff993400640 1 -- 172.21.15.6:0/3735709481 learned_addr learned my addr 172.21.15.6:0/3735709481 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:32.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.584+0000 7ff993400640 1 -- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9940ce8a0 msgr2=0x7ff9941521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:32.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.584+0000 7ff993400640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9940ce8a0 0x7ff9941521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:32.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.584+0000 7ff993400640 1 -- 172.21.15.6:0/3735709481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9940cc380 msgr2=0x7ff9941f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:32.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.584+0000 7ff993400640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9940cc380 0x7ff9941f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:32.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.584+0000 7ff993400640 1 -- 172.21.15.6:0/3735709481 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99415e3b0 con 0x7ff99406c230 2024-03-22T13:09:32.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.587+0000 7ff993400640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99414df90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff98400c530 tx=0x7ff98400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:32.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.592+0000 7ff98b400640 1 -- 172.21.15.6:0/3735709481 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984017070 con 0x7ff99406c230 2024-03-22T13:09:32.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.592+0000 7ff98b400640 1 -- 172.21.15.6:0/3735709481 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984005ce0 con 0x7ff99406c230 2024-03-22T13:09:32.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.592+0000 7ff98b400640 1 -- 172.21.15.6:0/3735709481 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840072b0 con 0x7ff99406c230 2024-03-22T13:09:32.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.592+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9940b4e80 con 0x7ff99406c230 2024-03-22T13:09:32.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.592+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9941efbe0 con 0x7ff99406c230 2024-03-22T13:09:32.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.593+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff994075650 con 0x7ff99406c230 2024-03-22T13:09:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.626+0000 7ff98b400640 1 -- 172.21.15.6:0/3735709481 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff98401a070 con 0x7ff99406c230 2024-03-22T13:09:32.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.627+0000 7ff98b400640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff97406bc20 0x7ff97406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:32.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.627+0000 7ff992a00640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff97406bc20 0x7ff97406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.628+0000 7ff992a00640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff97406bc20 0x7ff97406e0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff994158f10 tx=0x7ff980009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.628+0000 7ff98b400640 1 -- 172.21.15.6:0/3735709481 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff984056880 con 0x7ff99406c230 2024-03-22T13:09:32.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:32.905+0000 7ff98b400640 1 -- 172.21.15.6:0/3735709481 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff984021020 con 0x7ff99406c230 2024-03-22T13:09:33.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:09:33.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:09:33.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.175+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff994158a50 con 0x7ff99406c230 2024-03-22T13:09:33.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.206+0000 7ff98b400640 1 -- 172.21.15.6:0/3735709481 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7ff9840587e0 con 0x7ff99406c230 2024-03-22T13:09:33.206 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:33.206 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:33.206 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.208+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff97406bc20 msgr2=0x7ff97406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.208+0000 7ff999000640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff97406bc20 0x7ff97406e0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff994158f10 tx=0x7ff980009040 comp rx=0 tx=0).stop 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.208+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 msgr2=0x7ff99414df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.208+0000 7ff999000640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99414df90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff98400c530 tx=0x7ff98400c9f0 comp rx=0 tx=0).stop 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 shutdown_connections 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9940ce8a0 0x7ff9941521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9940cc380 0x7ff9941f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff97406bc20 0x7ff97406e0e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 --2- 172.21.15.6:0/3735709481 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff99406c230 0x7ff99414df90 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 >> 172.21.15.6:0/3735709481 conn(0x7ff9940b1480 msgr2=0x7ff994159110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 shutdown_connections 2024-03-22T13:09:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:33.209+0000 7ff999000640 1 -- 172.21.15.6:0/3735709481 wait complete. 2024-03-22T13:09:33.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:33.830 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:34.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.025+0000 7f1508e00640 1 Processor -- start 2024-03-22T13:09:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.025+0000 7f1508e00640 1 -- start start 2024-03-22T13:09:35.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.026+0000 7f1508e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f15040cc380 0x7f150406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:35.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.026+0000 7f1508e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f150415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:35.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.026+0000 7f1508e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15040cee00 con 0x7f15040cc760 2024-03-22T13:09:35.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.026+0000 7f1508e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15040cf0c0 con 0x7f15040cc380 2024-03-22T13:09:35.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.026+0000 7f1508e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15040cf3a0 con 0x7f15040ce8a0 2024-03-22T13:09:35.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.027+0000 7f1502a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f150415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:35.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.028+0000 7f1503e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f15040cc760 0x7f15040cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35290/0 (socket says 172.21.15.6:35290) 2024-03-22T13:09:35.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.028+0000 7f1503e00640 1 -- 172.21.15.6:0/4086713558 learned_addr learned my addr 172.21.15.6:0/4086713558 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:35.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.028+0000 7f1503400640 1 --2- 172.21.15.6:0/4086713558 >> v2:172.21.15.6:3300/0 conn(0x7f15040cc380 0x7f150406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:35.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.031+0000 7f1502a00640 1 -- 172.21.15.6:0/4086713558 >> v2:172.21.15.6:3300/0 conn(0x7f15040cc380 msgr2=0x7f150406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:35.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.031+0000 7f1502a00640 1 --2- 172.21.15.6:0/4086713558 >> v2:172.21.15.6:3300/0 conn(0x7f15040cc380 0x7f150406c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.031+0000 7f1502a00640 1 -- 172.21.15.6:0/4086713558 >> v1:172.21.15.6:6789/0 conn(0x7f15040cc760 legacy=0x7f15040cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:35.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.031+0000 7f1502a00640 1 -- 172.21.15.6:0/4086713558 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15040cf680 con 0x7f15040ce8a0 2024-03-22T13:09:35.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.034+0000 7f1502a00640 1 --2- 172.21.15.6:0/4086713558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f150415ae30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f14f00097a0 tx=0x7f14f002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bbb631e67e5ad9a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:35.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.038+0000 7f1502000640 1 -- 172.21.15.6:0/4086713558 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0035020 con 0x7f15040ce8a0 2024-03-22T13:09:35.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.038+0000 7f1502000640 1 -- 172.21.15.6:0/4086713558 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f0031830 con 0x7f15040ce8a0 2024-03-22T13:09:35.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.038+0000 7f1502000640 1 -- 172.21.15.6:0/4086713558 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0031c00 con 0x7f15040ce8a0 2024-03-22T13:09:35.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 -- 172.21.15.6:0/4086713558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 msgr2=0x7f150415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:35.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 --2- 172.21.15.6:0/4086713558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f150415ae30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f14f00097a0 tx=0x7f14f002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 -- 172.21.15.6:0/4086713558 shutdown_connections 2024-03-22T13:09:35.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 --2- 172.21.15.6:0/4086713558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f150415ae30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 --2- 172.21.15.6:0/4086713558 >> v2:172.21.15.6:3300/0 conn(0x7f15040cc380 0x7f150406c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 -- 172.21.15.6:0/4086713558 >> 172.21.15.6:0/4086713558 conn(0x7f15040b1480 msgr2=0x7f15040b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:35.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 -- 172.21.15.6:0/4086713558 shutdown_connections 2024-03-22T13:09:35.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.039+0000 7f1508e00640 1 -- 172.21.15.6:0/4086713558 wait complete. 2024-03-22T13:09:35.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.041+0000 7f1508e00640 1 Processor -- start 2024-03-22T13:09:35.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.041+0000 7f1508e00640 1 -- start start 2024-03-22T13:09:35.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.042+0000 7f1508e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15040cc380 0x7f1504149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:35.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.042+0000 7f1508e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15040cc760 0x7f15040c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:35.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.042+0000 7f1508e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f15040c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:35.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.042+0000 7f1508e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15041549d0 con 0x7f15040cc760 2024-03-22T13:09:35.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.042+0000 7f1508e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150415a910 con 0x7f15040cc380 2024-03-22T13:09:35.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.042+0000 7f1508e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15040ccb90 con 0x7f15040ce8a0 2024-03-22T13:09:35.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.043+0000 7f1502a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15040cc760 0x7f15040c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:35.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.043+0000 7f1503e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f15040c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:35.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.043+0000 7f1502a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15040cc760 0x7f15040c86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48276/0 (socket says 172.21.15.6:48276) 2024-03-22T13:09:35.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.043+0000 7f1502a00640 1 -- 172.21.15.6:0/2141706799 learned_addr learned my addr 172.21.15.6:0/2141706799 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.047+0000 7f1502a00640 1 -- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 msgr2=0x7f15040c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.047+0000 7f1502a00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f15040c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.047+0000 7f1502a00640 1 -- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15040cc380 msgr2=0x7f1504149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.047+0000 7f1502a00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15040cc380 0x7f1504149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.047+0000 7f1502a00640 1 -- 172.21.15.6:0/2141706799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150414a740 con 0x7f15040cc760 2024-03-22T13:09:35.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.049+0000 7f1502a00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15040cc760 0x7f15040c86f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009890 tx=0x7f14f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:35.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.054+0000 7f14fb400640 1 -- 172.21.15.6:0/2141706799 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0041070 con 0x7f15040cc760 2024-03-22T13:09:35.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.054+0000 7f14fb400640 1 -- 172.21.15.6:0/2141706799 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f003a9b0 con 0x7f15040cc760 2024-03-22T13:09:35.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.054+0000 7f14fb400640 1 -- 172.21.15.6:0/2141706799 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0035040 con 0x7f15040cc760 2024-03-22T13:09:35.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.054+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150414fc90 con 0x7f15040cc760 2024-03-22T13:09:35.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.054+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1504150250 con 0x7f15040cc760 2024-03-22T13:09:35.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.055+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1504075650 con 0x7f15040cc760 2024-03-22T13:09:35.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.079+0000 7f14fb400640 1 -- 172.21.15.6:0/2141706799 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f14f0038020 con 0x7f15040cc760 2024-03-22T13:09:35.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.080+0000 7f14fb400640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f14e406bc00 0x7f14e406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:35.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.081+0000 7f1503400640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f14e406bc00 0x7f14e406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:35.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.081+0000 7f1503400640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f14e406bc00 0x7f14e406e0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f14f4006fd0 tx=0x7f14f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:35.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.446+0000 7f14fb400640 1 -- 172.21.15.6:0/2141706799 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f14f0079160 con 0x7f15040cc760 2024-03-22T13:09:35.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.447+0000 7f14fb400640 1 -- 172.21.15.6:0/2141706799 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f150414a740 con 0x7f15040cc760 2024-03-22T13:09:35.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.742+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f150415e680 con 0x7f15040cc760 2024-03-22T13:09:35.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.774+0000 7f14fb400640 1 -- 172.21.15.6:0/2141706799 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f14f007b0c0 con 0x7f15040cc760 2024-03-22T13:09:35.774 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:35.775 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:35.775 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.777+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f14e406bc00 msgr2=0x7f14e406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.777+0000 7f1508e00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f14e406bc00 0x7f14e406e0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f14f4006fd0 tx=0x7f14f4008040 comp rx=0 tx=0).stop 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.777+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15040cc760 msgr2=0x7f15040c86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.777+0000 7f1508e00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15040cc760 0x7f15040c86f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009890 tx=0x7f14f00046c0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 shutdown_connections 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15040ce8a0 0x7f15040c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15040cc760 0x7f15040c86f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f14e406bc00 0x7f14e406e0c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 --2- 172.21.15.6:0/2141706799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15040cc380 0x7f1504149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 >> 172.21.15.6:0/2141706799 conn(0x7f15040b1480 msgr2=0x7f15040af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 shutdown_connections 2024-03-22T13:09:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:35.778+0000 7f1508e00640 1 -- 172.21.15.6:0/2141706799 wait complete. 2024-03-22T13:09:36.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:36.399 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:37.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.594+0000 7fd447600640 1 Processor -- start 2024-03-22T13:09:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.594+0000 7fd447600640 1 -- start start 2024-03-22T13:09:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.596+0000 7fd447600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.596+0000 7fd447600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4400cf5c0 0x7fd4400ca510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.596+0000 7fd447600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44015a6b0 con 0x7fd4400caa50 2024-03-22T13:09:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.596+0000 7fd447600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd440154d60 con 0x7fd4400cf5c0 2024-03-22T13:09:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.596+0000 7fd447600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44015b4d0 con 0x7fd4400ccd80 2024-03-22T13:09:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.597+0000 7fd43fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4400cf5c0 0x7fd4400ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:37.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.597+0000 7fd444c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:37.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.597+0000 7fd43fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4400cf5c0 0x7fd4400ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:38964/0 (socket says 172.21.15.6:38964) 2024-03-22T13:09:37.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.597+0000 7fd444c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57036/0 (socket says 172.21.15.6:57036) 2024-03-22T13:09:37.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.597+0000 7fd43fe00640 1 -- 172.21.15.6:0/1421512329 learned_addr learned my addr 172.21.15.6:0/1421512329 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.600+0000 7fd444c00640 1 -- 172.21.15.6:0/1421512329 >> v2:172.21.15.92:3300/0 conn(0x7fd4400cf5c0 msgr2=0x7fd4400ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.600+0000 7fd444c00640 1 --2- 172.21.15.6:0/1421512329 >> v2:172.21.15.92:3300/0 conn(0x7fd4400cf5c0 0x7fd4400ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.600+0000 7fd444c00640 1 -- 172.21.15.6:0/1421512329 >> v1:172.21.15.92:6789/0 conn(0x7fd4400caa50 legacy=0x7fd44006c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.600+0000 7fd444c00640 1 -- 172.21.15.6:0/1421512329 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44015a950 con 0x7fd4400ccd80 2024-03-22T13:09:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.603+0000 7fd444c00640 1 --2- 172.21.15.6:0/1421512329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400cf080 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd4340097a0 tx=0x7fd43402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e48ce95e2c3d05 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.608+0000 7fd43f400640 1 -- 172.21.15.6:0/1421512329 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434035020 con 0x7fd4400ccd80 2024-03-22T13:09:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.608+0000 7fd43f400640 1 -- 172.21.15.6:0/1421512329 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd434031830 con 0x7fd4400ccd80 2024-03-22T13:09:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.608+0000 7fd43f400640 1 -- 172.21.15.6:0/1421512329 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434031c00 con 0x7fd4400ccd80 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 -- 172.21.15.6:0/1421512329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 msgr2=0x7fd4400cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 --2- 172.21.15.6:0/1421512329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400cf080 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd4340097a0 tx=0x7fd43402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 -- 172.21.15.6:0/1421512329 shutdown_connections 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 --2- 172.21.15.6:0/1421512329 >> v2:172.21.15.92:3300/0 conn(0x7fd4400cf5c0 0x7fd4400ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 --2- 172.21.15.6:0/1421512329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400cf080 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 -- 172.21.15.6:0/1421512329 >> 172.21.15.6:0/1421512329 conn(0x7fd4400b16c0 msgr2=0x7fd4400b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 -- 172.21.15.6:0/1421512329 shutdown_connections 2024-03-22T13:09:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.609+0000 7fd447600640 1 -- 172.21.15.6:0/1421512329 wait complete. 2024-03-22T13:09:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.611+0000 7fd447600640 1 Processor -- start 2024-03-22T13:09:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.611+0000 7fd447600640 1 -- start start 2024-03-22T13:09:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.612+0000 7fd447600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4400caa50 0x7fd4400c3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.612+0000 7fd447600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.612+0000 7fd447600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4400cf5c0 0x7fd440152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.612+0000 7fd447600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44015b1f0 con 0x7fd4400caa50 2024-03-22T13:09:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.612+0000 7fd447600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44015af10 con 0x7fd4400cf5c0 2024-03-22T13:09:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.612+0000 7fd447600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44006db10 con 0x7fd4400ccd80 2024-03-22T13:09:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.613+0000 7fd43fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.613+0000 7fd444c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4400caa50 0x7fd4400c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.613+0000 7fd445600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4400cf5c0 0x7fd440152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.613+0000 7fd43fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:57038/0 (socket says 172.21.15.6:57038) 2024-03-22T13:09:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.613+0000 7fd43fe00640 1 -- 172.21.15.6:0/651419219 learned_addr learned my addr 172.21.15.6:0/651419219 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.617+0000 7fd43fe00640 1 -- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4400cf5c0 msgr2=0x7fd440152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.617+0000 7fd43fe00640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4400cf5c0 0x7fd440152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.617+0000 7fd43fe00640 1 -- 172.21.15.6:0/651419219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4400caa50 msgr2=0x7fd4400c3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.617+0000 7fd43fe00640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4400caa50 0x7fd4400c3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.617+0000 7fd43fe00640 1 -- 172.21.15.6:0/651419219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd440153200 con 0x7fd4400ccd80 2024-03-22T13:09:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.619+0000 7fd43fe00640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400c8150 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e870 tx=0x7fd42c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.623+0000 7fd43cc00640 1 -- 172.21.15.6:0/651419219 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c017070 con 0x7fd4400ccd80 2024-03-22T13:09:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.623+0000 7fd43cc00640 1 -- 172.21.15.6:0/651419219 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c00c8e0 con 0x7fd4400ccd80 2024-03-22T13:09:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.623+0000 7fd43cc00640 1 -- 172.21.15.6:0/651419219 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c00ccd0 con 0x7fd4400ccd80 2024-03-22T13:09:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.623+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4401f0bb0 con 0x7fd4400ccd80 2024-03-22T13:09:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.624+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd440153c80 con 0x7fd4400ccd80 2024-03-22T13:09:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.625+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4400756c0 con 0x7fd4400ccd80 2024-03-22T13:09:37.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.649+0000 7fd43cc00640 1 -- 172.21.15.6:0/651419219 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd42c01a070 con 0x7fd4400ccd80 2024-03-22T13:09:37.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.650+0000 7fd43cc00640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd42006bc50 0x7fd42006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:37.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.651+0000 7fd43cc00640 1 -- 172.21.15.6:0/651419219 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd42c057630 con 0x7fd4400ccd80 2024-03-22T13:09:37.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.651+0000 7fd444c00640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd42006bc50 0x7fd42006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:37.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.652+0000 7fd444c00640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd42006bc50 0x7fd42006e110 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd434036000 tx=0x7fd434008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:37.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:37.900+0000 7fd43cc00640 1 -- 172.21.15.6:0/651419219 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd42c059590 con 0x7fd4400ccd80 2024-03-22T13:09:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.165+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4400ca410 con 0x7fd4400ccd80 2024-03-22T13:09:38.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.180+0000 7fd43cc00640 1 -- 172.21.15.6:0/651419219 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fd42c05d630 con 0x7fd4400ccd80 2024-03-22T13:09:38.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:38.181 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:38.181 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:38.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd42006bc50 msgr2=0x7fd42006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:38.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd42006bc50 0x7fd42006e110 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd434036000 tx=0x7fd434008030 comp rx=0 tx=0).stop 2024-03-22T13:09:38.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 msgr2=0x7fd4400c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:38.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400c8150 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e870 tx=0x7fd42c00ed30 comp rx=0 tx=0).stop 2024-03-22T13:09:38.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 shutdown_connections 2024-03-22T13:09:38.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd42006bc50 0x7fd42006e110 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:38.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4400cf5c0 0x7fd440152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:38.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4400ccd80 0x7fd4400c8150 secure :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e870 tx=0x7fd42c00ed30 comp rx=0 tx=0).stop 2024-03-22T13:09:38.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 --2- 172.21.15.6:0/651419219 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4400caa50 0x7fd4400c3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:38.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 >> 172.21.15.6:0/651419219 conn(0x7fd4400b16c0 msgr2=0x7fd4400b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:38.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.183+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 shutdown_connections 2024-03-22T13:09:38.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:38.184+0000 7fd447600640 1 -- 172.21.15.6:0/651419219 wait complete. 2024-03-22T13:09:38.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:38.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:39.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:39.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:39.997+0000 7f61e5c00640 1 Processor -- start 2024-03-22T13:09:39.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:39.998+0000 7f61e5c00640 1 -- start start 2024-03-22T13:09:40.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:39.999+0000 7f61e5c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f61e00c68c0 0x7f61e015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:40.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:39.999+0000 7f61e5c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e00c4740 con 0x7f61e0154240 2024-03-22T13:09:40.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:39.999+0000 7f61e5c00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e0154620 con 0x7f61e00c7ee0 2024-03-22T13:09:40.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:39.999+0000 7f61e5c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e015e130 con 0x7f61e00c68c0 2024-03-22T13:09:40.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.000+0000 7f61dea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61e0154240 0x7f61e00c4030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35302/0 (socket says 172.21.15.6:35302) 2024-03-22T13:09:40.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.000+0000 7f61dea00640 1 -- 172.21.15.6:0/66851506 learned_addr learned my addr 172.21.15.6:0/66851506 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:40.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.002+0000 7f61df400640 1 --2- 172.21.15.6:0/66851506 >> v2:172.21.15.92:3300/0 conn(0x7f61e00c68c0 0x7f61e015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:40.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.003+0000 7f61dd600640 1 -- 172.21.15.6:0/66851506 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968809385 0 0) 0x7f61e00c4740 con 0x7f61e0154240 2024-03-22T13:09:40.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.003+0000 7f61dd600640 1 -- 172.21.15.6:0/66851506 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c4003520 con 0x7f61e0154240 2024-03-22T13:09:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.006+0000 7f61df400640 1 -- 172.21.15.6:0/66851506 >> v1:172.21.15.92:6789/0 conn(0x7f61e00c7ee0 legacy=0x7f61e006b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.006+0000 7f61df400640 1 -- 172.21.15.6:0/66851506 >> v1:172.21.15.6:6789/0 conn(0x7f61e0154240 legacy=0x7f61e00c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:40.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.006+0000 7f61df400640 1 -- 172.21.15.6:0/66851506 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e00c6e00 con 0x7f61e00c68c0 2024-03-22T13:09:40.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.009+0000 7f61df400640 1 --2- 172.21.15.6:0/66851506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c68c0 0x7f61e015ac90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009d90 tx=0x7f61d802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69aa704fc2beda1c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.016+0000 7f61dd600640 1 -- 172.21.15.6:0/66851506 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8039070 con 0x7f61e00c68c0 2024-03-22T13:09:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.016+0000 7f61dd600640 1 -- 172.21.15.6:0/66851506 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d802cd80 con 0x7f61e00c68c0 2024-03-22T13:09:40.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.016+0000 7f61dd600640 1 -- 172.21.15.6:0/66851506 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8034660 con 0x7f61e00c68c0 2024-03-22T13:09:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.017+0000 7f61e5c00640 1 -- 172.21.15.6:0/66851506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c68c0 msgr2=0x7f61e015ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.017+0000 7f61e5c00640 1 --2- 172.21.15.6:0/66851506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c68c0 0x7f61e015ac90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009d90 tx=0x7f61d802bfd0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.017+0000 7f61e5c00640 1 -- 172.21.15.6:0/66851506 shutdown_connections 2024-03-22T13:09:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.017+0000 7f61e5c00640 1 --2- 172.21.15.6:0/66851506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c68c0 0x7f61e015ac90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.017+0000 7f61e5c00640 1 -- 172.21.15.6:0/66851506 >> 172.21.15.6:0/66851506 conn(0x7f61e00cb930 msgr2=0x7f61e00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.017+0000 7f61e5c00640 1 -- 172.21.15.6:0/66851506 shutdown_connections 2024-03-22T13:09:40.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.017+0000 7f61e5c00640 1 -- 172.21.15.6:0/66851506 wait complete. 2024-03-22T13:09:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.019+0000 7f61e5c00640 1 Processor -- start 2024-03-22T13:09:40.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.019+0000 7f61e5c00640 1 -- start start 2024-03-22T13:09:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.020+0000 7f61e5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61e00c68c0 0x7f61e015d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.020+0000 7f61e5c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c7ee0 0x7f61e0151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.020+0000 7f61e5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e0154240 0x7f61e01f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.020+0000 7f61e5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e015dae0 con 0x7f61e0154240 2024-03-22T13:09:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.020+0000 7f61e5c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e015d800 con 0x7f61e00c7ee0 2024-03-22T13:09:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.020+0000 7f61e5c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e015ddc0 con 0x7f61e00c68c0 2024-03-22T13:09:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.021+0000 7f61de000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c7ee0 0x7f61e0151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.021+0000 7f61dea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61e00c68c0 0x7f61e015d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.021+0000 7f61de000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c7ee0 0x7f61e0151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:38996/0 (socket says 172.21.15.6:38996) 2024-03-22T13:09:40.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.021+0000 7f61de000640 1 -- 172.21.15.6:0/1996956433 learned_addr learned my addr 172.21.15.6:0/1996956433 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.026+0000 7f61de000640 1 -- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61e00c68c0 msgr2=0x7f61e015d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.026+0000 7f61de000640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61e00c68c0 0x7f61e015d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.026+0000 7f61de000640 1 -- 172.21.15.6:0/1996956433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e0154240 msgr2=0x7f61e01f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.026+0000 7f61de000640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e0154240 0x7f61e01f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.026+0000 7f61de000640 1 -- 172.21.15.6:0/1996956433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e01f3870 con 0x7f61e00c7ee0 2024-03-22T13:09:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.030+0000 7f61de000640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c7ee0 0x7f61e0151cb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00c470 tx=0x7f61cc001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.035+0000 7f61d6a00640 1 -- 172.21.15.6:0/1996956433 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc0052d0 con 0x7f61e00c7ee0 2024-03-22T13:09:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.035+0000 7f61d6a00640 1 -- 172.21.15.6:0/1996956433 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc005470 con 0x7f61e00c7ee0 2024-03-22T13:09:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.035+0000 7f61d6a00640 1 -- 172.21.15.6:0/1996956433 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc007260 con 0x7f61e00c7ee0 2024-03-22T13:09:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.035+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e015d5d0 con 0x7f61e00c7ee0 2024-03-22T13:09:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.035+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e01f0860 con 0x7f61e00c7ee0 2024-03-22T13:09:40.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.036+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e00756c0 con 0x7f61e00c7ee0 2024-03-22T13:09:40.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.298+0000 7f61d6a00640 1 -- 172.21.15.6:0/1996956433 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f61cc007400 con 0x7f61e00c7ee0 2024-03-22T13:09:40.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.300+0000 7f61d6a00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f61c006bc20 0x7f61c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:40.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.301+0000 7f61d6a00640 1 -- 172.21.15.6:0/1996956433 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f61cc08ab40 con 0x7f61e00c7ee0 2024-03-22T13:09:40.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.301+0000 7f61dea00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f61c006bc20 0x7f61c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:40.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.302+0000 7f61dea00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f61c006bc20 0x7f61c006e0e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f61d0005880 tx=0x7f61d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:40.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.312+0000 7f61d6a00640 1 -- 172.21.15.6:0/1996956433 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f61cc058ba0 con 0x7f61e00c7ee0 2024-03-22T13:09:40.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.585+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e00c7ab0 con 0x7f61e00c7ee0 2024-03-22T13:09:40.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.605+0000 7f61d6a00640 1 -- 172.21.15.6:0/1996956433 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f61cc05cc40 con 0x7f61e00c7ee0 2024-03-22T13:09:40.606 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:40.607 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:40.607 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:40.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f61c006bc20 msgr2=0x7f61c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:40.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f61c006bc20 0x7f61c006e0e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f61d0005880 tx=0x7f61d0009040 comp rx=0 tx=0).stop 2024-03-22T13:09:40.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c7ee0 msgr2=0x7f61e0151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:40.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c7ee0 0x7f61e0151cb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00c470 tx=0x7f61cc001e70 comp rx=0 tx=0).stop 2024-03-22T13:09:40.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 shutdown_connections 2024-03-22T13:09:40.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e0154240 0x7f61e01f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f61c006bc20 0x7f61c006e0e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61e00c7ee0 0x7f61e0151cb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 --2- 172.21.15.6:0/1996956433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61e00c68c0 0x7f61e015d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 >> 172.21.15.6:0/1996956433 conn(0x7f61e00cb930 msgr2=0x7f61e01587b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 shutdown_connections 2024-03-22T13:09:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:40.608+0000 7f61e5c00640 1 -- 172.21.15.6:0/1996956433 wait complete. 2024-03-22T13:09:41.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:41.230 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:42.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:42.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.469+0000 7f87b6e00640 1 Processor -- start 2024-03-22T13:09:42.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.469+0000 7f87b6e00640 1 -- start start 2024-03-22T13:09:42.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.470+0000 7f87b6e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc100 0x7f87b00cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:42.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.470+0000 7f87b6e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f87b00c9ab0 0x7f87b006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:42.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.470+0000 7f87b6e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87b00cd080 0x7f87b006d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:42.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.470+0000 7f87b6e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b00ca360 con 0x7f87b00cd080 2024-03-22T13:09:42.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.470+0000 7f87b6e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b0154980 con 0x7f87b00c9ab0 2024-03-22T13:09:42.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.470+0000 7f87b6e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b015afc0 con 0x7f87b00cc100 2024-03-22T13:09:42.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.471+0000 7f87b6400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87b00cd080 0x7f87b006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:42.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.471+0000 7f87b5a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc100 0x7f87b00cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:42.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.471+0000 7f87b6400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87b00cd080 0x7f87b006d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48298/0 (socket says 172.21.15.6:48298) 2024-03-22T13:09:42.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.471+0000 7f87b6400640 1 -- 172.21.15.6:0/2534766422 learned_addr learned my addr 172.21.15.6:0/2534766422 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:42.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.472+0000 7f87b5000640 1 --2- 172.21.15.6:0/2534766422 >> v2:172.21.15.92:3300/0 conn(0x7f87b00c9ab0 0x7f87b006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:42.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.474+0000 7f87b6400640 1 -- 172.21.15.6:0/2534766422 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc100 msgr2=0x7f87b00cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:42.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.474+0000 7f87b6400640 1 --2- 172.21.15.6:0/2534766422 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc100 0x7f87b00cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:42.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.474+0000 7f87b6400640 1 -- 172.21.15.6:0/2534766422 >> v2:172.21.15.92:3300/0 conn(0x7f87b00c9ab0 msgr2=0x7f87b006cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:42.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.474+0000 7f87b6400640 1 --2- 172.21.15.6:0/2534766422 >> v2:172.21.15.92:3300/0 conn(0x7f87b00c9ab0 0x7f87b006cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:42.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.474+0000 7f87b6400640 1 -- 172.21.15.6:0/2534766422 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b00ca5b0 con 0x7f87b00cd080 2024-03-22T13:09:42.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.477+0000 7f87b6400640 1 --2- 172.21.15.6:0/2534766422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cd080 0x7f87b006d240 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f87b008cdd0 tx=0x7f87a802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6605516a36677fa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.481+0000 7f87afe00640 1 -- 172.21.15.6:0/2534766422 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a802cb00 con 0x7f87b00cd080 2024-03-22T13:09:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.481+0000 7f87afe00640 1 -- 172.21.15.6:0/2534766422 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a802cca0 con 0x7f87b00cd080 2024-03-22T13:09:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.481+0000 7f87afe00640 1 -- 172.21.15.6:0/2534766422 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8034610 con 0x7f87b00cd080 2024-03-22T13:09:42.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 -- 172.21.15.6:0/2534766422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cd080 msgr2=0x7f87b006d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:42.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 --2- 172.21.15.6:0/2534766422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cd080 0x7f87b006d240 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f87b008cdd0 tx=0x7f87a802bf60 comp rx=0 tx=0).stop 2024-03-22T13:09:42.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 -- 172.21.15.6:0/2534766422 shutdown_connections 2024-03-22T13:09:42.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 --2- 172.21.15.6:0/2534766422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cd080 0x7f87b006d240 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:42.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 --2- 172.21.15.6:0/2534766422 >> v2:172.21.15.92:3300/0 conn(0x7f87b00c9ab0 0x7f87b006cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:42.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 --2- 172.21.15.6:0/2534766422 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc100 0x7f87b00cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 -- 172.21.15.6:0/2534766422 >> 172.21.15.6:0/2534766422 conn(0x7f87b00b14f0 msgr2=0x7f87b00b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.482+0000 7f87b6e00640 1 -- 172.21.15.6:0/2534766422 shutdown_connections 2024-03-22T13:09:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.483+0000 7f87b6e00640 1 -- 172.21.15.6:0/2534766422 wait complete. 2024-03-22T13:09:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.484+0000 7f87b6e00640 1 Processor -- start 2024-03-22T13:09:42.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.485+0000 7f87b6e00640 1 -- start start 2024-03-22T13:09:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.485+0000 7f87b6e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00c9ab0 0x7f87b00c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.485+0000 7f87b6e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cc100 0x7f87b0159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.486+0000 7f87b6e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cd080 0x7f87b01f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.486+0000 7f87b6e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b015ac60 con 0x7f87b00cc100 2024-03-22T13:09:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.486+0000 7f87b6e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b006c7e0 con 0x7f87b00cd080 2024-03-22T13:09:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.486+0000 7f87b6e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00ccd50 con 0x7f87b00c9ab0 2024-03-22T13:09:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.486+0000 7f87b5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cc100 0x7f87b0159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:42.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.486+0000 7f87b6400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cd080 0x7f87b01f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:42.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.486+0000 7f87b5a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00c9ab0 0x7f87b00c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:42.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.487+0000 7f87b5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cc100 0x7f87b0159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48300/0 (socket says 172.21.15.6:48300) 2024-03-22T13:09:42.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.487+0000 7f87b5000640 1 -- 172.21.15.6:0/3544167775 learned_addr learned my addr 172.21.15.6:0/3544167775 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:42.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.490+0000 7f87b5000640 1 -- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00c9ab0 msgr2=0x7f87b00c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:42.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.490+0000 7f87b5000640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00c9ab0 0x7f87b00c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:42.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.490+0000 7f87b5000640 1 -- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cd080 msgr2=0x7f87b01f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:42.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.490+0000 7f87b5000640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cd080 0x7f87b01f06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:42.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.490+0000 7f87b5000640 1 -- 172.21.15.6:0/3544167775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b01f45f0 con 0x7f87b00cc100 2024-03-22T13:09:42.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.493+0000 7f87b5000640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cc100 0x7f87b0159c50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f879c00e870 tx=0x7f879c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:42.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.498+0000 7f87ad600640 1 -- 172.21.15.6:0/3544167775 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c017070 con 0x7f87b00cc100 2024-03-22T13:09:42.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.498+0000 7f87ad600640 1 -- 172.21.15.6:0/3544167775 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879c00c940 con 0x7f87b00cc100 2024-03-22T13:09:42.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.498+0000 7f87ad600640 1 -- 172.21.15.6:0/3544167775 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c00cd30 con 0x7f87b00cc100 2024-03-22T13:09:42.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.498+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b01f4890 con 0x7f87b00cc100 2024-03-22T13:09:42.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.499+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b01574e0 con 0x7f87b00cc100 2024-03-22T13:09:42.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.500+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b00c7c70 con 0x7f87b00cc100 2024-03-22T13:09:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.805+0000 7f87ad600640 1 -- 172.21.15.6:0/3544167775 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f879c01a070 con 0x7f87b00cc100 2024-03-22T13:09:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.807+0000 7f87ad600640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc20 0x7f879006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.808+0000 7f87b5a00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc20 0x7f879006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:42.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.808+0000 7f87b5a00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc20 0x7f879006e0e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009a20 tx=0x7f87a4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.825+0000 7f87ad600640 1 -- 172.21.15.6:0/3544167775 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f879c0575f0 con 0x7f87b00cc100 2024-03-22T13:09:42.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:42.826+0000 7f87ad600640 1 -- 172.21.15.6:0/3544167775 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f87b00c7c70 con 0x7f87b00cc100 2024-03-22T13:09:43.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.071+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b0075650 con 0x7f87b00cc100 2024-03-22T13:09:43.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.128+0000 7f87ad600640 1 -- 172.21.15.6:0/3544167775 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f879c059550 con 0x7f87b00cc100 2024-03-22T13:09:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:43.130 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:43.130 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:43.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc20 msgr2=0x7f879006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:43.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc20 0x7f879006e0e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009a20 tx=0x7f87a4009190 comp rx=0 tx=0).stop 2024-03-22T13:09:43.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cc100 msgr2=0x7f87b0159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:43.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cc100 0x7f87b0159c50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f879c00e870 tx=0x7f879c00ed30 comp rx=0 tx=0).stop 2024-03-22T13:09:43.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 shutdown_connections 2024-03-22T13:09:43.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc20 0x7f879006e0e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:43.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cd080 0x7f87b01f06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:43.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b00cc100 0x7f87b0159c50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:43.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 --2- 172.21.15.6:0/3544167775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00c9ab0 0x7f87b00c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:43.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 >> 172.21.15.6:0/3544167775 conn(0x7f87b00b14f0 msgr2=0x7f87b0153fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:43.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 shutdown_connections 2024-03-22T13:09:43.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:43.132+0000 7f87b6e00640 1 -- 172.21.15.6:0/3544167775 wait complete. 2024-03-22T13:09:43.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:43.756 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:44.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:44.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.948+0000 7f1d05e00640 1 Processor -- start 2024-03-22T13:09:44.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.949+0000 7f1d05e00640 1 -- start start 2024-03-22T13:09:44.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.950+0000 7f1d05e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1d000c7ee0 0x7f1d0015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:44.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.950+0000 7f1d05e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d000c4740 con 0x7f1d000c68c0 2024-03-22T13:09:44.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.950+0000 7f1d05e00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d00154620 con 0x7f1d00154240 2024-03-22T13:09:44.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.950+0000 7f1d05e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0015e130 con 0x7f1d000c7ee0 2024-03-22T13:09:44.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.951+0000 7f1cf7e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1d000c7ee0 0x7f1d0015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:44.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.951+0000 7f1cff400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1d000c68c0 0x7f1d0006b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53908/0 (socket says 172.21.15.6:53908) 2024-03-22T13:09:44.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.951+0000 7f1cff400640 1 -- 172.21.15.6:0/294857047 learned_addr learned my addr 172.21.15.6:0/294857047 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:44.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.954+0000 7f1cf7e00640 1 -- 172.21.15.6:0/294857047 >> v1:172.21.15.92:6789/0 conn(0x7f1d00154240 legacy=0x7f1d000c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:44.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.954+0000 7f1cf7e00640 1 -- 172.21.15.6:0/294857047 >> v1:172.21.15.6:6789/0 conn(0x7f1d000c68c0 legacy=0x7f1d0006b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:44.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.954+0000 7f1cf7e00640 1 -- 172.21.15.6:0/294857047 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d000c6e00 con 0x7f1d000c7ee0 2024-03-22T13:09:44.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.958+0000 7f1cf7e00640 1 --2- 172.21.15.6:0/294857047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c7ee0 0x7f1d0015ac90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80097a0 tx=0x7f1ce802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718e79ee3b96072b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:44.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.962+0000 7f1cfe000640 1 -- 172.21.15.6:0/294857047 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce802f810 con 0x7f1d000c7ee0 2024-03-22T13:09:44.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.963+0000 7f1cfe000640 1 -- 172.21.15.6:0/294857047 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce802f9b0 con 0x7f1d000c7ee0 2024-03-22T13:09:44.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.963+0000 7f1cfe000640 1 -- 172.21.15.6:0/294857047 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce802fd80 con 0x7f1d000c7ee0 2024-03-22T13:09:44.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.963+0000 7f1d05e00640 1 -- 172.21.15.6:0/294857047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c7ee0 msgr2=0x7f1d0015ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:44.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.963+0000 7f1d05e00640 1 --2- 172.21.15.6:0/294857047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c7ee0 0x7f1d0015ac90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80097a0 tx=0x7f1ce802b9f0 comp rx=0 tx=0).stop 2024-03-22T13:09:44.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.964+0000 7f1d05e00640 1 -- 172.21.15.6:0/294857047 shutdown_connections 2024-03-22T13:09:44.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.964+0000 7f1d05e00640 1 --2- 172.21.15.6:0/294857047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c7ee0 0x7f1d0015ac90 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:44.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.964+0000 7f1d05e00640 1 -- 172.21.15.6:0/294857047 >> 172.21.15.6:0/294857047 conn(0x7f1d000cb930 msgr2=0x7f1d000cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:44.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.964+0000 7f1d05e00640 1 -- 172.21.15.6:0/294857047 shutdown_connections 2024-03-22T13:09:44.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.964+0000 7f1d05e00640 1 -- 172.21.15.6:0/294857047 wait complete. 2024-03-22T13:09:44.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.965+0000 7f1d05e00640 1 Processor -- start 2024-03-22T13:09:44.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.966+0000 7f1d05e00640 1 -- start start 2024-03-22T13:09:44.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.966+0000 7f1d05e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c68c0 0x7f1d001e4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:44.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.966+0000 7f1d05e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d000c7ee0 0x7f1d000cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:44.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.966+0000 7f1d05e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d00154240 0x7f1d001f0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:44.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.966+0000 7f1d05e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0015dae0 con 0x7f1d00154240 2024-03-22T13:09:44.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.966+0000 7f1d05e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0015d800 con 0x7f1d000c68c0 2024-03-22T13:09:44.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.966+0000 7f1d05e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0015ddc0 con 0x7f1d000c7ee0 2024-03-22T13:09:44.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.967+0000 7f1cf7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d000c7ee0 0x7f1d000cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:44.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.967+0000 7f1cff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d00154240 0x7f1d001f0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:44.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.968+0000 7f1cfea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c68c0 0x7f1d001e4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:44.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.968+0000 7f1cf7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d000c7ee0 0x7f1d000cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35652/0 (socket says 172.21.15.6:35652) 2024-03-22T13:09:44.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.968+0000 7f1cff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d00154240 0x7f1d001f0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44726/0 (socket says 172.21.15.6:44726) 2024-03-22T13:09:44.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.968+0000 7f1cf7e00640 1 -- 172.21.15.6:0/3100661089 learned_addr learned my addr 172.21.15.6:0/3100661089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:44.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.971+0000 7f1cfea00640 1 -- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d000c7ee0 msgr2=0x7f1d000cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:44.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.971+0000 7f1cfea00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d000c7ee0 0x7f1d000cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:44.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.971+0000 7f1cfea00640 1 -- 172.21.15.6:0/3100661089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d00154240 msgr2=0x7f1d001f0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:44.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.971+0000 7f1cfea00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d00154240 0x7f1d001f0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:44.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.971+0000 7f1cfea00640 1 -- 172.21.15.6:0/3100661089 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d00158ce0 con 0x7f1d000c68c0 2024-03-22T13:09:44.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.974+0000 7f1cfea00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c68c0 0x7f1d001e4c30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000c3f0 tx=0x7f1cf000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:44.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.978+0000 7f1cf7400640 1 -- 172.21.15.6:0/3100661089 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0017070 con 0x7f1d000c68c0 2024-03-22T13:09:44.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.978+0000 7f1cf7400640 1 -- 172.21.15.6:0/3100661089 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf00052d0 con 0x7f1d000c68c0 2024-03-22T13:09:44.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.978+0000 7f1cf7400640 1 -- 172.21.15.6:0/3100661089 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0003830 con 0x7f1d000c68c0 2024-03-22T13:09:44.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.978+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d00158f80 con 0x7f1d000c68c0 2024-03-22T13:09:44.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.978+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d001e5b10 con 0x7f1d000c68c0 2024-03-22T13:09:44.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:44.979+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d000756c0 con 0x7f1d000c68c0 2024-03-22T13:09:45.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.000+0000 7f1cf7400640 1 -- 172.21.15.6:0/3100661089 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1cf001a070 con 0x7f1d000c68c0 2024-03-22T13:09:45.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.001+0000 7f1cf7400640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cdc06bc20 0x7f1cdc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:45.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.001+0000 7f1cf7e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cdc06bc20 0x7f1cdc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:45.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.002+0000 7f1cf7e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cdc06bc20 0x7f1cdc06e0e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1d0014b190 tx=0x7f1ce8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:45.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.241+0000 7f1cf7400640 1 -- 172.21.15.6:0/3100661089 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1cf0056660 con 0x7f1d000c68c0 2024-03-22T13:09:45.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.252+0000 7f1cf7400640 1 -- 172.21.15.6:0/3100661089 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1cf0021020 con 0x7f1d000c68c0 2024-03-22T13:09:45.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.502+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d0006fac0 con 0x7f1d000c68c0 2024-03-22T13:09:45.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.545+0000 7f1cf7400640 1 -- 172.21.15.6:0/3100661089 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f1cf00585c0 con 0x7f1d000c68c0 2024-03-22T13:09:45.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:45.546 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:45.547 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cdc06bc20 msgr2=0x7f1cdc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cdc06bc20 0x7f1cdc06e0e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1d0014b190 tx=0x7f1ce8008030 comp rx=0 tx=0).stop 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c68c0 msgr2=0x7f1d001e4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c68c0 0x7f1d001e4c30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000c3f0 tx=0x7f1cf000c8b0 comp rx=0 tx=0).stop 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 shutdown_connections 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1d00154240 0x7f1d001f0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1d000c7ee0 0x7f1d000cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cdc06bc20 0x7f1cdc06e0e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 --2- 172.21.15.6:0/3100661089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d000c68c0 0x7f1d001e4c30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 >> 172.21.15.6:0/3100661089 conn(0x7f1d000cb930 msgr2=0x7f1d000744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:45.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 shutdown_connections 2024-03-22T13:09:45.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:45.549+0000 7f1d05e00640 1 -- 172.21.15.6:0/3100661089 wait complete. 2024-03-22T13:09:46.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:46.179 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:47.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:47.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.376+0000 7fe075400640 1 Processor -- start 2024-03-22T13:09:47.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.376+0000 7fe075400640 1 -- start start 2024-03-22T13:09:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.378+0000 7fe075400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe070152400 0x7fe07015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.378+0000 7fe075400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0701590b0 con 0x7fe07006f8c0 2024-03-22T13:09:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.378+0000 7fe075400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0700721f0 con 0x7fe07015bc80 2024-03-22T13:09:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.378+0000 7fe075400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700724d0 con 0x7fe070152400 2024-03-22T13:09:47.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.379+0000 7fe06e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe070152400 0x7fe07015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:47.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.379+0000 7fe06f200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe07006f8c0 0x7fe0700c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53922/0 (socket says 172.21.15.6:53922) 2024-03-22T13:09:47.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.379+0000 7fe06f200640 1 -- 172.21.15.6:0/3407228303 learned_addr learned my addr 172.21.15.6:0/3407228303 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:47.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.382+0000 7fe06e800640 1 -- 172.21.15.6:0/3407228303 >> v1:172.21.15.92:6789/0 conn(0x7fe07015bc80 legacy=0x7fe0700716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:47.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.382+0000 7fe06e800640 1 -- 172.21.15.6:0/3407228303 >> v1:172.21.15.6:6789/0 conn(0x7fe07006f8c0 legacy=0x7fe0700c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:47.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.382+0000 7fe06e800640 1 -- 172.21.15.6:0/3407228303 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0701582f0 con 0x7fe070152400 2024-03-22T13:09:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.385+0000 7fe06e800640 1 --2- 172.21.15.6:0/3407228303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe070152400 0x7fe07015b740 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe070072650 tx=0x7fe06002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8050ac7c5fc3fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:47.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.390+0000 7fe06de00640 1 -- 172.21.15.6:0/3407228303 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060034070 con 0x7fe070152400 2024-03-22T13:09:47.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.390+0000 7fe06de00640 1 -- 172.21.15.6:0/3407228303 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe060030880 con 0x7fe070152400 2024-03-22T13:09:47.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.390+0000 7fe06de00640 1 -- 172.21.15.6:0/3407228303 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060030c50 con 0x7fe070152400 2024-03-22T13:09:47.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.390+0000 7fe075400640 1 -- 172.21.15.6:0/3407228303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe070152400 msgr2=0x7fe07015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:47.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.390+0000 7fe075400640 1 --2- 172.21.15.6:0/3407228303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe070152400 0x7fe07015b740 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe070072650 tx=0x7fe06002b9c0 comp rx=0 tx=0).stop 2024-03-22T13:09:47.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.390+0000 7fe075400640 1 -- 172.21.15.6:0/3407228303 shutdown_connections 2024-03-22T13:09:47.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.390+0000 7fe075400640 1 --2- 172.21.15.6:0/3407228303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe070152400 0x7fe07015b740 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.391+0000 7fe075400640 1 -- 172.21.15.6:0/3407228303 >> 172.21.15.6:0/3407228303 conn(0x7fe0700cb930 msgr2=0x7fe0700cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.391+0000 7fe075400640 1 -- 172.21.15.6:0/3407228303 shutdown_connections 2024-03-22T13:09:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.391+0000 7fe075400640 1 -- 172.21.15.6:0/3407228303 wait complete. 2024-03-22T13:09:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.392+0000 7fe075400640 1 Processor -- start 2024-03-22T13:09:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.392+0000 7fe075400640 1 -- start start 2024-03-22T13:09:47.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.393+0000 7fe075400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07006f8c0 0x7fe0701468e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:47.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.393+0000 7fe075400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe070152400 0x7fe070144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:47.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.393+0000 7fe075400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07015bc80 0x7fe07015a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:47.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.393+0000 7fe075400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070158b90 con 0x7fe070152400 2024-03-22T13:09:47.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.393+0000 7fe075400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0701588b0 con 0x7fe07006f8c0 2024-03-22T13:09:47.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.393+0000 7fe075400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0701527e0 con 0x7fe07015bc80 2024-03-22T13:09:47.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.394+0000 7fe06f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07015bc80 0x7fe07015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:47.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.394+0000 7fe06e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07006f8c0 0x7fe0701468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:47.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.394+0000 7fe067e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe070152400 0x7fe070144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:47.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.394+0000 7fe06f200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07015bc80 0x7fe07015a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35668/0 (socket says 172.21.15.6:35668) 2024-03-22T13:09:47.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.394+0000 7fe06f200640 1 -- 172.21.15.6:0/1268445888 learned_addr learned my addr 172.21.15.6:0/1268445888 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.398+0000 7fe067e00640 1 -- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07015bc80 msgr2=0x7fe07015a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.398+0000 7fe067e00640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07015bc80 0x7fe07015a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.398+0000 7fe067e00640 1 -- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07006f8c0 msgr2=0x7fe0701468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.398+0000 7fe067e00640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07006f8c0 0x7fe0701468e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.398+0000 7fe067e00640 1 -- 172.21.15.6:0/1268445888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070146e20 con 0x7fe070152400 2024-03-22T13:09:47.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.401+0000 7fe067e00640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe070152400 0x7fe070144140 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe07015c080 tx=0x7fe05800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:47.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.405+0000 7fe066a00640 1 -- 172.21.15.6:0/1268445888 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058017070 con 0x7fe070152400 2024-03-22T13:09:47.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.406+0000 7fe066a00640 1 -- 172.21.15.6:0/1268445888 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0580052d0 con 0x7fe070152400 2024-03-22T13:09:47.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.406+0000 7fe066a00640 1 -- 172.21.15.6:0/1268445888 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058003810 con 0x7fe070152400 2024-03-22T13:09:47.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.406+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07015b790 con 0x7fe070152400 2024-03-22T13:09:47.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.406+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe070159840 con 0x7fe070152400 2024-03-22T13:09:47.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.407+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0700756c0 con 0x7fe070152400 2024-03-22T13:09:47.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.431+0000 7fe066a00640 1 -- 172.21.15.6:0/1268445888 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe05801a070 con 0x7fe070152400 2024-03-22T13:09:47.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.432+0000 7fe066a00640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe04c06bc50 0x7fe04c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:47.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.433+0000 7fe06e800640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe04c06bc50 0x7fe04c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.434+0000 7fe06e800640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe04c06bc50 0x7fe04c06e110 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe060036000 tx=0x7fe06003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:47.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.435+0000 7fe066a00640 1 -- 172.21.15.6:0/1268445888 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe0580566b0 con 0x7fe070152400 2024-03-22T13:09:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.712+0000 7fe066a00640 1 -- 172.21.15.6:0/1268445888 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe058021020 con 0x7fe070152400 2024-03-22T13:09:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.975+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe07014fb90 con 0x7fe070152400 2024-03-22T13:09:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.996+0000 7fe066a00640 1 -- 172.21.15.6:0/1268445888 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fe058058610 con 0x7fe070152400 2024-03-22T13:09:47.998 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:47.998 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.999+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe04c06bc50 msgr2=0x7fe04c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.999+0000 7fe075400640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe04c06bc50 0x7fe04c06e110 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe060036000 tx=0x7fe06003b040 comp rx=0 tx=0).stop 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.999+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe070152400 msgr2=0x7fe070144140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:47.999+0000 7fe075400640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe070152400 0x7fe070144140 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe07015c080 tx=0x7fe05800c820 comp rx=0 tx=0).stop 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 shutdown_connections 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07015bc80 0x7fe07015a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe070152400 0x7fe070144140 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe04c06bc50 0x7fe04c06e110 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 --2- 172.21.15.6:0/1268445888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07006f8c0 0x7fe0701468e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 >> 172.21.15.6:0/1268445888 conn(0x7fe0700cb930 msgr2=0x7fe0700c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 shutdown_connections 2024-03-22T13:09:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:48.000+0000 7fe075400640 1 -- 172.21.15.6:0/1268445888 wait complete. 2024-03-22T13:09:48.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:48.622 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:49.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:49.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.820+0000 7fae32000640 1 Processor -- start 2024-03-22T13:09:49.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.820+0000 7fae32000640 1 -- start start 2024-03-22T13:09:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.821+0000 7fae32000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c0cc640 0x7fae2c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.821+0000 7fae32000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fae2c15b7a0 0x7fae2c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.821+0000 7fae32000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c06dd00 con 0x7fae2c0cca20 2024-03-22T13:09:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.821+0000 7fae32000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2c06d330 con 0x7fae2c15b7a0 2024-03-22T13:09:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.821+0000 7fae32000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c06c440 con 0x7fae2c0cc640 2024-03-22T13:09:49.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.843+0000 7fae31600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fae2c15b7a0 0x7fae2c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:49.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.843+0000 7fae31600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fae2c15b7a0 0x7fae2c15e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54868/0 (socket says 172.21.15.6:54868) 2024-03-22T13:09:49.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.843+0000 7fae31600640 1 -- 172.21.15.6:0/2755823256 learned_addr learned my addr 172.21.15.6:0/2755823256 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.846+0000 7fae31600640 1 -- 172.21.15.6:0/2755823256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c0cc640 msgr2=0x7fae2c06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.846+0000 7fae31600640 1 --2- 172.21.15.6:0/2755823256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c0cc640 0x7fae2c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.846+0000 7fae31600640 1 -- 172.21.15.6:0/2755823256 >> v1:172.21.15.6:6789/0 conn(0x7fae2c0cca20 legacy=0x7fae2c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.846+0000 7fae31600640 1 -- 172.21.15.6:0/2755823256 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c06c720 con 0x7fae2c15b7a0 2024-03-22T13:09:49.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.849+0000 7fae31600640 1 --2- 172.21.15.6:0/2755823256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c15b7a0 0x7fae2c15e260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fae24009e50 tx=0x7fae2402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22664b394a10900f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.854+0000 7fae2b400640 1 -- 172.21.15.6:0/2755823256 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2402cb00 con 0x7fae2c15b7a0 2024-03-22T13:09:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.854+0000 7fae2b400640 1 -- 172.21.15.6:0/2755823256 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2402cca0 con 0x7fae2c15b7a0 2024-03-22T13:09:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.854+0000 7fae2b400640 1 -- 172.21.15.6:0/2755823256 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24034610 con 0x7fae2c15b7a0 2024-03-22T13:09:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 -- 172.21.15.6:0/2755823256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c15b7a0 msgr2=0x7fae2c15e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 --2- 172.21.15.6:0/2755823256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c15b7a0 0x7fae2c15e260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fae24009e50 tx=0x7fae2402bf60 comp rx=0 tx=0).stop 2024-03-22T13:09:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 -- 172.21.15.6:0/2755823256 shutdown_connections 2024-03-22T13:09:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 --2- 172.21.15.6:0/2755823256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c15b7a0 0x7fae2c15e260 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 --2- 172.21.15.6:0/2755823256 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c0cc640 0x7fae2c06d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 -- 172.21.15.6:0/2755823256 >> 172.21.15.6:0/2755823256 conn(0x7fae2c0cf9c0 msgr2=0x7fae2c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 -- 172.21.15.6:0/2755823256 shutdown_connections 2024-03-22T13:09:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.856+0000 7fae32000640 1 -- 172.21.15.6:0/2755823256 wait complete. 2024-03-22T13:09:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.859+0000 7fae32000640 1 Processor -- start 2024-03-22T13:09:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.859+0000 7fae32000640 1 -- start start 2024-03-22T13:09:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.861+0000 7fae32000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c0cc640 0x7fae2c15ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.861+0000 7fae32000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae2c0cca20 0x7fae2c1e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.861+0000 7fae32000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c15b7a0 0x7fae2c1e54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.861+0000 7fae32000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c06cfc0 con 0x7fae2c0cca20 2024-03-22T13:09:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.861+0000 7fae32000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c06cce0 con 0x7fae2c0cc640 2024-03-22T13:09:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.861+0000 7fae32000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c0caa50 con 0x7fae2c15b7a0 2024-03-22T13:09:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.863+0000 7fae30c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c0cc640 0x7fae2c15ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.864+0000 7fae30c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c0cc640 0x7fae2c15ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54874/0 (socket says 172.21.15.6:54874) 2024-03-22T13:09:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.864+0000 7fae30c00640 1 -- 172.21.15.6:0/4237432528 learned_addr learned my addr 172.21.15.6:0/4237432528 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:49.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.869+0000 7fae30c00640 1 -- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c15b7a0 msgr2=0x7fae2c1e54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:49.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.869+0000 7fae30c00640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c15b7a0 0x7fae2c1e54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:49.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.869+0000 7fae30c00640 1 -- 172.21.15.6:0/4237432528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae2c0cca20 msgr2=0x7fae2c1e4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:49.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.869+0000 7fae30c00640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae2c0cca20 0x7fae2c1e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:49.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.869+0000 7fae30c00640 1 -- 172.21.15.6:0/4237432528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c1e6140 con 0x7fae2c0cc640 2024-03-22T13:09:49.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.872+0000 7fae30c00640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c0cc640 0x7fae2c15ce70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fae2000b4a0 tx=0x7fae2000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.876+0000 7fae28c00640 1 -- 172.21.15.6:0/4237432528 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20011830 con 0x7fae2c0cc640 2024-03-22T13:09:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.876+0000 7fae28c00640 1 -- 172.21.15.6:0/4237432528 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae20016070 con 0x7fae2c0cc640 2024-03-22T13:09:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.876+0000 7fae28c00640 1 -- 172.21.15.6:0/4237432528 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20011c10 con 0x7fae2c0cc640 2024-03-22T13:09:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.876+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c0b00d0 con 0x7fae2c0cc640 2024-03-22T13:09:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.876+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c1e4a70 con 0x7fae2c0cc640 2024-03-22T13:09:49.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.878+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c075650 con 0x7fae2c0cc640 2024-03-22T13:09:49.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.910+0000 7fae28c00640 1 -- 172.21.15.6:0/4237432528 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fae2001b460 con 0x7fae2c0cc640 2024-03-22T13:09:49.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.912+0000 7fae28c00640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fae0c06bbd0 0x7fae0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:49.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.912+0000 7fae2be00640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fae0c06bbd0 0x7fae0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:49.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.913+0000 7fae2be00640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fae0c06bbd0 0x7fae0c06e090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0c2f80 tx=0x7fae18007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:49.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:49.915+0000 7fae28c00640 1 -- 172.21.15.6:0/4237432528 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fae200577c0 con 0x7fae2c0cc640 2024-03-22T13:09:50.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.177+0000 7fae28c00640 1 -- 172.21.15.6:0/4237432528 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fae20059e20 con 0x7fae2c0cc640 2024-03-22T13:09:50.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.425+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae2c0af100 con 0x7fae2c0cc640 2024-03-22T13:09:50.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.439+0000 7fae28c00640 1 -- 172.21.15.6:0/4237432528 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fae20059720 con 0x7fae2c0cc640 2024-03-22T13:09:50.439 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:50.440 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:50.440 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fae0c06bbd0 msgr2=0x7fae0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fae0c06bbd0 0x7fae0c06e090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0c2f80 tx=0x7fae18007990 comp rx=0 tx=0).stop 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c0cc640 msgr2=0x7fae2c15ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c0cc640 0x7fae2c15ce70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fae2000b4a0 tx=0x7fae2000b960 comp rx=0 tx=0).stop 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 shutdown_connections 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fae2c15b7a0 0x7fae2c1e54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae2c0cca20 0x7fae2c1e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:50.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fae0c06bbd0 0x7fae0c06e090 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:50.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 --2- 172.21.15.6:0/4237432528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae2c0cc640 0x7fae2c15ce70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:50.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 >> 172.21.15.6:0/4237432528 conn(0x7fae2c0cf9c0 msgr2=0x7fae2c158cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:50.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 shutdown_connections 2024-03-22T13:09:50.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:50.442+0000 7fae32000640 1 -- 172.21.15.6:0/4237432528 wait complete. 2024-03-22T13:09:51.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:51.064 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:52.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:52.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.262+0000 7f5e21a00640 1 Processor -- start 2024-03-22T13:09:52.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.263+0000 7f5e21a00640 1 -- start start 2024-03-22T13:09:52.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.264+0000 7f5e21a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e1c15bf60 0x7f5e1c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.264+0000 7f5e21a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c158010 con 0x7f5e1c0c2e50 2024-03-22T13:09:52.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.264+0000 7f5e21a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c1582d0 con 0x7f5e1c0c9630 2024-03-22T13:09:52.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.264+0000 7f5e21a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c158ee0 con 0x7f5e1c15bf60 2024-03-22T13:09:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.265+0000 7f5e13e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5e1c0c2e50 0x7f5e1c15b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53928/0 (socket says 172.21.15.6:53928) 2024-03-22T13:09:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.265+0000 7f5e13e00640 1 -- 172.21.15.6:0/4052907848 learned_addr learned my addr 172.21.15.6:0/4052907848 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:09:52.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.266+0000 7f5e1b400640 1 --2- 172.21.15.6:0/4052907848 >> v2:172.21.15.92:3300/0 conn(0x7f5e1c15bf60 0x7f5e1c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.268+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330658494 0 0) 0x7f5e1c158010 con 0x7f5e1c0c2e50 2024-03-22T13:09:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.268+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e00003520 con 0x7f5e1c0c2e50 2024-03-22T13:09:52.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.269+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937079806 0 0) 0x7f5e1c1582d0 con 0x7f5e1c0c9630 2024-03-22T13:09:52.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.269+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c158010 con 0x7f5e1c0c9630 2024-03-22T13:09:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4195812569 0 0) 0x7f5e00003520 con 0x7f5e1c0c2e50 2024-03-22T13:09:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 >> v2:172.21.15.92:3300/0 conn(0x7f5e1c15bf60 msgr2=0x7f5e1c0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 --2- 172.21.15.6:0/4052907848 >> v2:172.21.15.92:3300/0 conn(0x7f5e1c15bf60 0x7f5e1c0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:52.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 >> v1:172.21.15.92:6789/0 conn(0x7f5e1c0c9630 legacy=0x7f5e1c0c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c1585b0 con 0x7f5e1c0c2e50 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f5e04001670 con 0x7f5e1c0c2e50 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 >> v1:172.21.15.6:6789/0 conn(0x7f5e1c0c2e50 legacy=0x7f5e1c15b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00003b90 0x7f5e00003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e00004800 0x7f5e00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e000051c0 0x7f5e0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c158ee0 con 0x7f5e000051c0 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0000c010 con 0x7f5e00003b90 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.272+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0000c210 con 0x7f5e00004800 2024-03-22T13:09:52.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.273+0000 7f5e13e00640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e00004800 0x7f5e00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:52.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.276+0000 7f5e1b400640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e000051c0 0x7f5e0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:52.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.277+0000 7f5e13e00640 1 -- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00003b90 msgr2=0x7f5e00003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:52.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.277+0000 7f5e13e00640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00003b90 0x7f5e00003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:52.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.277+0000 7f5e13e00640 1 -- 172.21.15.6:0/4052907848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e000051c0 msgr2=0x7f5e0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:52.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.277+0000 7f5e13e00640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e000051c0 0x7f5e0000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:52.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.277+0000 7f5e13e00640 1 -- 172.21.15.6:0/4052907848 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00003520 con 0x7f5e00004800 2024-03-22T13:09:52.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.277+0000 7f5e1b400640 1 -- 172.21.15.6:0/4052907848 reap_dead start 2024-03-22T13:09:52.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.280+0000 7f5e13e00640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e00004800 0x7f5e00004c80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5e04007fc0 tx=0x7f5e00005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8626c02f1b8defbc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:52.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.285+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04007be0 con 0x7f5e00004800 2024-03-22T13:09:52.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.285+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e04030070 con 0x7f5e00004800 2024-03-22T13:09:52.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.285+0000 7f5e1a000640 1 -- 172.21.15.6:0/4052907848 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04005d30 con 0x7f5e00004800 2024-03-22T13:09:52.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.285+0000 7f5e21a00640 1 -- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e00004800 msgr2=0x7f5e00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:52.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.285+0000 7f5e21a00640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e00004800 0x7f5e00004c80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5e04007fc0 tx=0x7f5e00005830 comp rx=0 tx=0).stop 2024-03-22T13:09:52.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.286+0000 7f5e21a00640 1 -- 172.21.15.6:0/4052907848 shutdown_connections 2024-03-22T13:09:52.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.286+0000 7f5e21a00640 1 --2- 172.21.15.6:0/4052907848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e00004800 0x7f5e00004c80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:52.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.286+0000 7f5e21a00640 1 -- 172.21.15.6:0/4052907848 >> 172.21.15.6:0/4052907848 conn(0x7f5e1c0714f0 msgr2=0x7f5e1c071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:52.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.286+0000 7f5e21a00640 1 -- 172.21.15.6:0/4052907848 shutdown_connections 2024-03-22T13:09:52.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.286+0000 7f5e21a00640 1 -- 172.21.15.6:0/4052907848 wait complete. 2024-03-22T13:09:52.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.288+0000 7f5e21a00640 1 Processor -- start 2024-03-22T13:09:52.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.288+0000 7f5e21a00640 1 -- start start 2024-03-22T13:09:52.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e21a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00004800 0x7f5e1c1f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e21a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e1c14ee30 0x7f5e1c14bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e21a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1c0c58c0 0x7f5e1c0cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e21a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c0ca330 con 0x7f5e1c14ee30 2024-03-22T13:09:52.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e21a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c158b70 con 0x7f5e00004800 2024-03-22T13:09:52.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e21a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c1525f0 con 0x7f5e1c0c58c0 2024-03-22T13:09:52.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e1b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1c0c58c0 0x7f5e1c0cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:52.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e1aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00004800 0x7f5e1c1f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:52.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.289+0000 7f5e13e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e1c14ee30 0x7f5e1c14bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:52.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.290+0000 7f5e1b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1c0c58c0 0x7f5e1c0cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35716/0 (socket says 172.21.15.6:35716) 2024-03-22T13:09:52.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.290+0000 7f5e1b400640 1 -- 172.21.15.6:0/579769427 learned_addr learned my addr 172.21.15.6:0/579769427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:52.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.290+0000 7f5e1aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00004800 0x7f5e1c1f0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:54910/0 (socket says 172.21.15.6:54910) 2024-03-22T13:09:52.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.293+0000 7f5e1aa00640 1 -- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1c0c58c0 msgr2=0x7f5e1c0cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:52.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.293+0000 7f5e1aa00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1c0c58c0 0x7f5e1c0cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:52.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.293+0000 7f5e1aa00640 1 -- 172.21.15.6:0/579769427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e1c14ee30 msgr2=0x7f5e1c14bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:52.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.293+0000 7f5e1aa00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e1c14ee30 0x7f5e1c14bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:52.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.293+0000 7f5e1aa00640 1 -- 172.21.15.6:0/579769427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c14f320 con 0x7f5e00004800 2024-03-22T13:09:52.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.296+0000 7f5e1aa00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00004800 0x7f5e1c1f0100 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0697f0 tx=0x7f5e0c00c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:52.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.300+0000 7f5e12a00640 1 -- 172.21.15.6:0/579769427 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c017070 con 0x7f5e00004800 2024-03-22T13:09:52.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.300+0000 7f5e12a00640 1 -- 172.21.15.6:0/579769427 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0c0052f0 con 0x7f5e00004800 2024-03-22T13:09:52.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.300+0000 7f5e12a00640 1 -- 172.21.15.6:0/579769427 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c003830 con 0x7f5e00004800 2024-03-22T13:09:52.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.300+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c14f600 con 0x7f5e00004800 2024-03-22T13:09:52.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.300+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c14fbc0 con 0x7f5e00004800 2024-03-22T13:09:52.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.301+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c0712c0 con 0x7f5e00004800 2024-03-22T13:09:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.322+0000 7f5e12a00640 1 -- 172.21.15.6:0/579769427 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e0c01a070 con 0x7f5e00004800 2024-03-22T13:09:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.323+0000 7f5e12a00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5df806bc50 0x7f5df806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:52.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.324+0000 7f5e13e00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5df806bc50 0x7f5df806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.324+0000 7f5e13e00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5df806bc50 0x7f5df806e110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5e04034000 tx=0x7f5e04007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.326+0000 7f5e12a00640 1 -- 172.21.15.6:0/579769427 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5e0c056750 con 0x7f5e00004800 2024-03-22T13:09:52.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.601+0000 7f5e12a00640 1 -- 172.21.15.6:0/579769427 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5e1c0712c0 con 0x7f5e00004800 2024-03-22T13:09:52.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:52.861+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e1c0756c0 con 0x7f5e00004800 2024-03-22T13:09:53.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.042+0000 7f5e12a00640 1 -- 172.21.15.6:0/579769427 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f5e0c021020 con 0x7f5e00004800 2024-03-22T13:09:53.042 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:53.042 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:53.043 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:53.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5df806bc50 msgr2=0x7f5df806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:53.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5df806bc50 0x7f5df806e110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5e04034000 tx=0x7f5e04007000 comp rx=0 tx=0).stop 2024-03-22T13:09:53.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00004800 msgr2=0x7f5e1c1f0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:53.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00004800 0x7f5e1c1f0100 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0697f0 tx=0x7f5e0c00c960 comp rx=0 tx=0).stop 2024-03-22T13:09:53.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 shutdown_connections 2024-03-22T13:09:53.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1c0c58c0 0x7f5e1c0cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:53.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e1c14ee30 0x7f5e1c14bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:53.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5df806bc50 0x7f5df806e110 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:53.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 --2- 172.21.15.6:0/579769427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e00004800 0x7f5e1c1f0100 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:53.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 >> 172.21.15.6:0/579769427 conn(0x7f5e1c0714f0 msgr2=0x7f5e1c14c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:53.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 shutdown_connections 2024-03-22T13:09:53.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:53.045+0000 7f5e21a00640 1 -- 172.21.15.6:0/579769427 wait complete. 2024-03-22T13:09:53.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:53.668 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:54.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:54.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.869+0000 7f0e4aa00640 1 Processor -- start 2024-03-22T13:09:54.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.869+0000 7f0e4aa00640 1 -- start start 2024-03-22T13:09:54.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.871+0000 7f0e4aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e440716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:54.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.871+0000 7f0e4aa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e44071c10 0x7f0e440c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:54.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.871+0000 7f0e4aa00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e441590b0 con 0x7f0e44152400 2024-03-22T13:09:54.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.871+0000 7f0e4aa00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e440724d0 con 0x7f0e44071c10 2024-03-22T13:09:54.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.871+0000 7f0e4aa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e440721f0 con 0x7f0e4406f8c0 2024-03-22T13:09:54.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.872+0000 7f0e48a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e44071c10 0x7f0e440c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:54.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.872+0000 7f0e43400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e440716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:54.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.872+0000 7f0e43400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e440716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33228/0 (socket says 172.21.15.6:33228) 2024-03-22T13:09:54.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.872+0000 7f0e43400640 1 -- 172.21.15.6:0/1988964953 learned_addr learned my addr 172.21.15.6:0/1988964953 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:54.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.875+0000 7f0e43400640 1 -- 172.21.15.6:0/1988964953 >> v2:172.21.15.92:3300/0 conn(0x7f0e44071c10 msgr2=0x7f0e440c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:54.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.875+0000 7f0e43400640 1 --2- 172.21.15.6:0/1988964953 >> v2:172.21.15.92:3300/0 conn(0x7f0e44071c10 0x7f0e440c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:54.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.875+0000 7f0e43400640 1 -- 172.21.15.6:0/1988964953 >> v1:172.21.15.92:6789/0 conn(0x7f0e44152400 legacy=0x7f0e4415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:54.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.875+0000 7f0e43400640 1 -- 172.21.15.6:0/1988964953 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e441582f0 con 0x7f0e4406f8c0 2024-03-22T13:09:54.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.878+0000 7f0e43400640 1 --2- 172.21.15.6:0/1988964953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e440716d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009870 tx=0x7f0e3002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0a08c70b648ec62 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:54.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.883+0000 7f0e42a00640 1 -- 172.21.15.6:0/1988964953 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3002f840 con 0x7f0e4406f8c0 2024-03-22T13:09:54.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.883+0000 7f0e42a00640 1 -- 172.21.15.6:0/1988964953 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3002f9e0 con 0x7f0e4406f8c0 2024-03-22T13:09:54.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.883+0000 7f0e42a00640 1 -- 172.21.15.6:0/1988964953 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3002fdb0 con 0x7f0e4406f8c0 2024-03-22T13:09:54.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 -- 172.21.15.6:0/1988964953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 msgr2=0x7f0e440716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/1988964953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e440716d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009870 tx=0x7f0e3002ba20 comp rx=0 tx=0).stop 2024-03-22T13:09:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 -- 172.21.15.6:0/1988964953 shutdown_connections 2024-03-22T13:09:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/1988964953 >> v2:172.21.15.92:3300/0 conn(0x7f0e44071c10 0x7f0e440c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/1988964953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e440716d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 -- 172.21.15.6:0/1988964953 >> 172.21.15.6:0/1988964953 conn(0x7f0e440cb930 msgr2=0x7f0e440cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 -- 172.21.15.6:0/1988964953 shutdown_connections 2024-03-22T13:09:54.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.884+0000 7f0e4aa00640 1 -- 172.21.15.6:0/1988964953 wait complete. 2024-03-22T13:09:54.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.885+0000 7f0e4aa00640 1 Processor -- start 2024-03-22T13:09:54.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.886+0000 7f0e4aa00640 1 -- start start 2024-03-22T13:09:54.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.886+0000 7f0e4aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e4414e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:54.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.887+0000 7f0e4aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e44071c10 0x7f0e441405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:54.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.887+0000 7f0e4aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e44152400 0x7f0e44148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:54.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.887+0000 7f0e4aa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e44158b90 con 0x7f0e44071c10 2024-03-22T13:09:54.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.887+0000 7f0e4aa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441588b0 con 0x7f0e44152400 2024-03-22T13:09:54.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.887+0000 7f0e4aa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e441527e0 con 0x7f0e4406f8c0 2024-03-22T13:09:54.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.887+0000 7f0e43e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e4414e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:54.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.887+0000 7f0e48a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e44152400 0x7f0e44148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:54.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.888+0000 7f0e43400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e44071c10 0x7f0e441405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:54.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.888+0000 7f0e43e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e4414e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33232/0 (socket says 172.21.15.6:33232) 2024-03-22T13:09:54.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.888+0000 7f0e43e00640 1 -- 172.21.15.6:0/545247230 learned_addr learned my addr 172.21.15.6:0/545247230 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:54.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.891+0000 7f0e48a00640 1 -- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 msgr2=0x7f0e4414e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:54.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.891+0000 7f0e48a00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e4414e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:54.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.891+0000 7f0e48a00640 1 -- 172.21.15.6:0/545247230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e44071c10 msgr2=0x7f0e441405c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:54.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.891+0000 7f0e48a00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e44071c10 0x7f0e441405c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:54.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.891+0000 7f0e48a00640 1 -- 172.21.15.6:0/545247230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4414de70 con 0x7f0e44152400 2024-03-22T13:09:54.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.894+0000 7f0e48a00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e44152400 0x7f0e44148140 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00efc0 tx=0x7f0e3c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:54.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.897+0000 7f0e3be00640 1 -- 172.21.15.6:0/545247230 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c018020 con 0x7f0e44152400 2024-03-22T13:09:54.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.898+0000 7f0e3be00640 1 -- 172.21.15.6:0/545247230 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c014480 con 0x7f0e44152400 2024-03-22T13:09:54.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.898+0000 7f0e3be00640 1 -- 172.21.15.6:0/545247230 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c014870 con 0x7f0e44152400 2024-03-22T13:09:54.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.898+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4414da70 con 0x7f0e44152400 2024-03-22T13:09:54.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.898+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e44146190 con 0x7f0e44152400 2024-03-22T13:09:54.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.899+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e440c4b30 con 0x7f0e44152400 2024-03-22T13:09:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.920+0000 7f0e3be00640 1 -- 172.21.15.6:0/545247230 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0e3c005d10 con 0x7f0e44152400 2024-03-22T13:09:54.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.922+0000 7f0e3be00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2406bc20 0x7f0e2406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.922+0000 7f0e43e00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2406bc20 0x7f0e2406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:54.923+0000 7f0e43e00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2406bc20 0x7f0e2406e0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0e340017a0 tx=0x7f0e34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.163+0000 7f0e3be00640 1 -- 172.21.15.6:0/545247230 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0e3c058220 con 0x7f0e44152400 2024-03-22T13:09:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.181+0000 7f0e3be00640 1 -- 172.21.15.6:0/545247230 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0e440c4b30 con 0x7f0e44152400 2024-03-22T13:09:55.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.443+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e44072750 con 0x7f0e44152400 2024-03-22T13:09:55.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.458+0000 7f0e3be00640 1 -- 172.21.15.6:0/545247230 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f0e44072750 con 0x7f0e44152400 2024-03-22T13:09:55.458 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:55.458 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:55.459 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:55.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.461+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2406bc20 msgr2=0x7f0e2406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:55.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.461+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2406bc20 0x7f0e2406e0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0e340017a0 tx=0x7f0e34009040 comp rx=0 tx=0).stop 2024-03-22T13:09:55.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.461+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e44152400 msgr2=0x7f0e44148140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:55.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.461+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e44152400 0x7f0e44148140 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00efc0 tx=0x7f0e3c00c490 comp rx=0 tx=0).stop 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 shutdown_connections 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0e2406bc20 0x7f0e2406e0e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e44152400 0x7f0e44148140 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e44071c10 0x7f0e441405c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 --2- 172.21.15.6:0/545247230 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e4406f8c0 0x7f0e4414e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 >> 172.21.15.6:0/545247230 conn(0x7f0e440cb930 msgr2=0x7f0e4414e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 shutdown_connections 2024-03-22T13:09:55.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:55.462+0000 7f0e4aa00640 1 -- 172.21.15.6:0/545247230 wait complete. 2024-03-22T13:09:56.083 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:56.083 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:57.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:57.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.285+0000 7f7440800640 1 Processor -- start 2024-03-22T13:09:57.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.285+0000 7f7440800640 1 -- start start 2024-03-22T13:09:57.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.286+0000 7f7440800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74380cc790 0x7f74380ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:57.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.286+0000 7f7440800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f743815ab40 0x7f743815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:57.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.286+0000 7f7440800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74380ce230 con 0x7f74380cc3b0 2024-03-22T13:09:57.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.287+0000 7f7440800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74380cd0f0 con 0x7f74380cc790 2024-03-22T13:09:57.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.287+0000 7f7440800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74380ce510 con 0x7f743815ab40 2024-03-22T13:09:57.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.287+0000 7f743e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f743815ab40 0x7f743815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:57.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.287+0000 7f743d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74380cc790 0x7f74380ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:57.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.288+0000 7f743d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74380cc790 0x7f74380ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53386/0 (socket says 172.21.15.6:53386) 2024-03-22T13:09:57.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.288+0000 7f743d400640 1 -- 172.21.15.6:0/247694808 learned_addr learned my addr 172.21.15.6:0/247694808 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:57.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.292+0000 7f743d400640 1 -- 172.21.15.6:0/247694808 >> v2:172.21.15.92:3300/0 conn(0x7f743815ab40 msgr2=0x7f743815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:57.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.292+0000 7f743d400640 1 --2- 172.21.15.6:0/247694808 >> v2:172.21.15.92:3300/0 conn(0x7f743815ab40 0x7f743815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.292+0000 7f743d400640 1 -- 172.21.15.6:0/247694808 >> v1:172.21.15.92:6789/0 conn(0x7f74380cc3b0 legacy=0x7f74380cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:57.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.292+0000 7f743d400640 1 -- 172.21.15.6:0/247694808 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74380ce7f0 con 0x7f74380cc790 2024-03-22T13:09:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.294+0000 7f743d400640 1 --2- 172.21.15.6:0/247694808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74380cc790 0x7f74380ccbb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74380b1660 tx=0x7f742802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66624b0d64d30a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.299+0000 7f743ca00640 1 -- 172.21.15.6:0/247694808 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428035020 con 0x7f74380cc790 2024-03-22T13:09:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.299+0000 7f743ca00640 1 -- 172.21.15.6:0/247694808 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7428031870 con 0x7f74380cc790 2024-03-22T13:09:57.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.299+0000 7f743ca00640 1 -- 172.21.15.6:0/247694808 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428031c40 con 0x7f74380cc790 2024-03-22T13:09:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.300+0000 7f7440800640 1 -- 172.21.15.6:0/247694808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74380cc790 msgr2=0x7f74380ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.300+0000 7f7440800640 1 --2- 172.21.15.6:0/247694808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74380cc790 0x7f74380ccbb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74380b1660 tx=0x7f742802b910 comp rx=0 tx=0).stop 2024-03-22T13:09:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.300+0000 7f7440800640 1 -- 172.21.15.6:0/247694808 shutdown_connections 2024-03-22T13:09:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.300+0000 7f7440800640 1 --2- 172.21.15.6:0/247694808 >> v2:172.21.15.92:3300/0 conn(0x7f743815ab40 0x7f743815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.300+0000 7f7440800640 1 --2- 172.21.15.6:0/247694808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74380cc790 0x7f74380ccbb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.300+0000 7f7440800640 1 -- 172.21.15.6:0/247694808 >> 172.21.15.6:0/247694808 conn(0x7f74380b16d0 msgr2=0x7f74380b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.301+0000 7f7440800640 1 -- 172.21.15.6:0/247694808 shutdown_connections 2024-03-22T13:09:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.301+0000 7f7440800640 1 -- 172.21.15.6:0/247694808 wait complete. 2024-03-22T13:09:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.303+0000 7f7440800640 1 Processor -- start 2024-03-22T13:09:57.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.303+0000 7f7440800640 1 -- start start 2024-03-22T13:09:57.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f7440800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74380cc3b0 0x7f7438156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:57.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f7440800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74380cc790 0x7f7438156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f7440800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743815ab40 0x7f7438155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f7440800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743815b4f0 con 0x7f743815ab40 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f7440800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74380cedb0 con 0x7f74380cc3b0 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f7440800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743806dc10 con 0x7f74380cc790 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f743de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74380cc3b0 0x7f7438156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.304+0000 7f743d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74380cc790 0x7f7438156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.305+0000 7f743e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743815ab40 0x7f7438155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:57.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.305+0000 7f743de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74380cc3b0 0x7f7438156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:57754/0 (socket says 172.21.15.6:57754) 2024-03-22T13:09:57.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.305+0000 7f743de00640 1 -- 172.21.15.6:0/3588971231 learned_addr learned my addr 172.21.15.6:0/3588971231 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.309+0000 7f743e800640 1 -- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74380cc790 msgr2=0x7f7438156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.309+0000 7f743e800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74380cc790 0x7f7438156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.309+0000 7f743e800640 1 -- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74380cc3b0 msgr2=0x7f7438156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.309+0000 7f743e800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74380cc3b0 0x7f7438156880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.309+0000 7f743e800640 1 -- 172.21.15.6:0/3588971231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74381541f0 con 0x7f743815ab40 2024-03-22T13:09:57.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.314+0000 7f743e800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743815ab40 0x7f7438155610 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f743400ebd0 tx=0x7f743400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:57.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.320+0000 7f7432000640 1 -- 172.21.15.6:0/3588971231 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434018020 con 0x7f743815ab40 2024-03-22T13:09:57.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.320+0000 7f7432000640 1 -- 172.21.15.6:0/3588971231 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743400cd20 con 0x7f743815ab40 2024-03-22T13:09:57.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.320+0000 7f7432000640 1 -- 172.21.15.6:0/3588971231 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74340146c0 con 0x7f743815ab40 2024-03-22T13:09:57.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.320+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74381f0d50 con 0x7f743815ab40 2024-03-22T13:09:57.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.320+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7438153c30 con 0x7f743815ab40 2024-03-22T13:09:57.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.322+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74380756c0 con 0x7f743815ab40 2024-03-22T13:09:57.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.351+0000 7f7432000640 1 -- 172.21.15.6:0/3588971231 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7434005d10 con 0x7f743815ab40 2024-03-22T13:09:57.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.352+0000 7f7432000640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f741c06bc50 0x7f741c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:57.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.352+0000 7f743de00640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f741c06bc50 0x7f741c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:57.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.353+0000 7f743de00640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f741c06bc50 0x7f741c06e110 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f742c007fd0 tx=0x7f742c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:57.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.649+0000 7f7432000640 1 -- 172.21.15.6:0/3588971231 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f74340580d0 con 0x7f743815ab40 2024-03-22T13:09:57.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.649+0000 7f7432000640 1 -- 172.21.15.6:0/3588971231 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7434022030 con 0x7f743815ab40 2024-03-22T13:09:57.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.915+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f743806de80 con 0x7f743815ab40 2024-03-22T13:09:57.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.936+0000 7f7432000640 1 -- 172.21.15.6:0/3588971231 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f7434010040 con 0x7f743815ab40 2024-03-22T13:09:57.936 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:09:57.936 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:09:57.936 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f741c06bc50 msgr2=0x7f741c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f741c06bc50 0x7f741c06e110 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f742c007fd0 tx=0x7f742c009040 comp rx=0 tx=0).stop 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743815ab40 msgr2=0x7f7438155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743815ab40 0x7f7438155610 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f743400ebd0 tx=0x7f743400c490 comp rx=0 tx=0).stop 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 shutdown_connections 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f743815ab40 0x7f7438155610 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f74380cc790 0x7f7438156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f741c06bc50 0x7f741c06e110 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 --2- 172.21.15.6:0/3588971231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74380cc3b0 0x7f7438156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 >> 172.21.15.6:0/3588971231 conn(0x7f74380b16d0 msgr2=0x7f74380cab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 shutdown_connections 2024-03-22T13:09:57.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:57.939+0000 7f7440800640 1 -- 172.21.15.6:0/3588971231 wait complete. 2024-03-22T13:09:58.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:09:58.560 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:09:59.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:09:59.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.759+0000 7fd476000640 1 Processor -- start 2024-03-22T13:09:59.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.759+0000 7fd476000640 1 -- start start 2024-03-22T13:09:59.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.760+0000 7fd476000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd4700cc380 0x7fd47006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:59.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.761+0000 7fd476000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4700ce8a0 0x7fd47015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:59.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.761+0000 7fd476000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4700cee00 con 0x7fd4700cc760 2024-03-22T13:09:59.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.761+0000 7fd476000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4700cf0c0 con 0x7fd4700cc380 2024-03-22T13:09:59.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.761+0000 7fd476000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4700cf3a0 con 0x7fd4700ce8a0 2024-03-22T13:09:59.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.761+0000 7fd474c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd4700cc380 0x7fd47006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:59.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.762+0000 7fd46fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd4700ce8a0 0x7fd47015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:59.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.762+0000 7fd474c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd4700cc380 0x7fd47006c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53406/0 (socket says 172.21.15.6:53406) 2024-03-22T13:09:59.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.762+0000 7fd474c00640 1 -- 172.21.15.6:0/3423206013 learned_addr learned my addr 172.21.15.6:0/3423206013 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:59.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.764+0000 7fd474c00640 1 -- 172.21.15.6:0/3423206013 >> v2:172.21.15.92:3300/0 conn(0x7fd4700ce8a0 msgr2=0x7fd47015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:09:59.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.764+0000 7fd474c00640 1 --2- 172.21.15.6:0/3423206013 >> v2:172.21.15.92:3300/0 conn(0x7fd4700ce8a0 0x7fd47015ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:59.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.764+0000 7fd474c00640 1 -- 172.21.15.6:0/3423206013 >> v1:172.21.15.92:6789/0 conn(0x7fd4700cc760 legacy=0x7fd4700cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:59.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.764+0000 7fd474c00640 1 -- 172.21.15.6:0/3423206013 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4700cf680 con 0x7fd4700cc380 2024-03-22T13:09:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.767+0000 7fd474c00640 1 --2- 172.21.15.6:0/3423206013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 0x7fd47006c230 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd4700cb7f0 tx=0x7fd46402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233d88ddacd38ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.769+0000 7fd46f400640 1 -- 172.21.15.6:0/3423206013 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464034070 con 0x7fd4700cc380 2024-03-22T13:09:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.772+0000 7fd46f400640 1 -- 172.21.15.6:0/3423206013 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464030860 con 0x7fd4700cc380 2024-03-22T13:09:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.772+0000 7fd46f400640 1 -- 172.21.15.6:0/3423206013 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd464030c30 con 0x7fd4700cc380 2024-03-22T13:09:59.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.772+0000 7fd476000640 1 -- 172.21.15.6:0/3423206013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 msgr2=0x7fd47006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:59.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.772+0000 7fd476000640 1 --2- 172.21.15.6:0/3423206013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 0x7fd47006c230 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd4700cb7f0 tx=0x7fd46402b9a0 comp rx=0 tx=0).stop 2024-03-22T13:09:59.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.773+0000 7fd476000640 1 -- 172.21.15.6:0/3423206013 shutdown_connections 2024-03-22T13:09:59.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.773+0000 7fd476000640 1 --2- 172.21.15.6:0/3423206013 >> v2:172.21.15.92:3300/0 conn(0x7fd4700ce8a0 0x7fd47015ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:59.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.773+0000 7fd476000640 1 --2- 172.21.15.6:0/3423206013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 0x7fd47006c230 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:59.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.773+0000 7fd476000640 1 -- 172.21.15.6:0/3423206013 >> 172.21.15.6:0/3423206013 conn(0x7fd4700b1480 msgr2=0x7fd4700b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:09:59.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.773+0000 7fd476000640 1 -- 172.21.15.6:0/3423206013 shutdown_connections 2024-03-22T13:09:59.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.773+0000 7fd476000640 1 -- 172.21.15.6:0/3423206013 wait complete. 2024-03-22T13:09:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.774+0000 7fd476000640 1 Processor -- start 2024-03-22T13:09:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.775+0000 7fd476000640 1 -- start start 2024-03-22T13:09:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.775+0000 7fd476000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 0x7fd47015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.775+0000 7fd476000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4700cc760 0x7fd4700afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.775+0000 7fd476000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4700ce8a0 0x7fd47015db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.776+0000 7fd476000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4701549d0 con 0x7fd4700cc380 2024-03-22T13:09:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.776+0000 7fd476000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47015a910 con 0x7fd4700ce8a0 2024-03-22T13:09:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.776+0000 7fd476000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700ccb70 con 0x7fd4700cc760 2024-03-22T13:09:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.776+0000 7fd46fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4700cc760 0x7fd4700afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.776+0000 7fd474c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 0x7fd47015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.776+0000 7fd475600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4700ce8a0 0x7fd47015db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.777+0000 7fd46fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4700cc760 0x7fd4700afe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33244/0 (socket says 172.21.15.6:33244) 2024-03-22T13:09:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.777+0000 7fd46fe00640 1 -- 172.21.15.6:0/2798503953 learned_addr learned my addr 172.21.15.6:0/2798503953 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:09:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.781+0000 7fd46fe00640 1 -- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4700ce8a0 msgr2=0x7fd47015db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.781+0000 7fd46fe00640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4700ce8a0 0x7fd47015db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.781+0000 7fd46fe00640 1 -- 172.21.15.6:0/2798503953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 msgr2=0x7fd47015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:09:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.781+0000 7fd46fe00640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 0x7fd47015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:09:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.781+0000 7fd46fe00640 1 -- 172.21.15.6:0/2798503953 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd470150300 con 0x7fd4700cc760 2024-03-22T13:09:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.783+0000 7fd46fe00640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4700cc760 0x7fd4700afe20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00e8a0 tx=0x7fd45c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:09:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.788+0000 7fd46cc00640 1 -- 172.21.15.6:0/2798503953 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c017070 con 0x7fd4700cc760 2024-03-22T13:09:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.788+0000 7fd46cc00640 1 -- 172.21.15.6:0/2798503953 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c00c940 con 0x7fd4700cc760 2024-03-22T13:09:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.788+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47014efb0 con 0x7fd4700cc760 2024-03-22T13:09:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.788+0000 7fd46cc00640 1 -- 172.21.15.6:0/2798503953 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00cd30 con 0x7fd4700cc760 2024-03-22T13:09:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.788+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47015cdd0 con 0x7fd4700cc760 2024-03-22T13:09:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.789+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4700b1860 con 0x7fd4700cc760 2024-03-22T13:09:59.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.814+0000 7fd46cc00640 1 -- 172.21.15.6:0/2798503953 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd45c01a070 con 0x7fd4700cc760 2024-03-22T13:09:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.816+0000 7fd46cc00640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd45006bc20 0x7fd45006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:09:59.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.816+0000 7fd474c00640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd45006bc20 0x7fd45006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:09:59.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:09:59.817+0000 7fd474c00640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd45006bc20 0x7fd45006e0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd464036000 tx=0x7fd464008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:00.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.059+0000 7fd46cc00640 1 -- 172.21.15.6:0/2798503953 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd45c057590 con 0x7fd4700cc760 2024-03-22T13:10:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.079+0000 7fd46cc00640 1 -- 172.21.15.6:0/2798503953 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd4700b1860 con 0x7fd4700cc760 2024-03-22T13:10:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.352+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd470075650 con 0x7fd4700cc760 2024-03-22T13:10:00.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.438+0000 7fd46cc00640 1 -- 172.21.15.6:0/2798503953 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fd45c0594f0 con 0x7fd4700cc760 2024-03-22T13:10:00.439 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:00.439 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:00.439 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:00.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd45006bc20 msgr2=0x7fd45006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:00.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd45006bc20 0x7fd45006e0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd464036000 tx=0x7fd464008030 comp rx=0 tx=0).stop 2024-03-22T13:10:00.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4700cc760 msgr2=0x7fd4700afe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:00.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4700cc760 0x7fd4700afe20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00e8a0 tx=0x7fd45c00ed60 comp rx=0 tx=0).stop 2024-03-22T13:10:00.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 shutdown_connections 2024-03-22T13:10:00.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd45006bc20 0x7fd45006e0e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:00.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4700ce8a0 0x7fd47015db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:00.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4700cc760 0x7fd4700afe20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:00.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 --2- 172.21.15.6:0/2798503953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4700cc380 0x7fd47015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:00.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 >> 172.21.15.6:0/2798503953 conn(0x7fd4700b1480 msgr2=0x7fd4701594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:00.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 shutdown_connections 2024-03-22T13:10:00.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:00.442+0000 7fd476000640 1 -- 172.21.15.6:0/2798503953 wait complete. 2024-03-22T13:10:01.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:01.073 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:02.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:02.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.272+0000 7f6d80600640 1 Processor -- start 2024-03-22T13:10:02.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.272+0000 7f6d80600640 1 -- start start 2024-03-22T13:10:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.274+0000 7f6d80600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d780c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.274+0000 7f6d80600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6d780c5c20 0x7f6d78156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.274+0000 7f6d80600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d780c3bf0 con 0x7f6d78157410 2024-03-22T13:10:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.274+0000 7f6d80600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d780c3760 con 0x7f6d780c5c20 2024-03-22T13:10:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.274+0000 7f6d80600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78071e70 con 0x7f6d78150f50 2024-03-22T13:10:02.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.340+0000 7f6d7dc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d780c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:02.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.341+0000 7f6d7dc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d780c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33260/0 (socket says 172.21.15.6:33260) 2024-03-22T13:10:02.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.341+0000 7f6d7dc00640 1 -- 172.21.15.6:0/505362249 learned_addr learned my addr 172.21.15.6:0/505362249 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:02.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.344+0000 7f6d7dc00640 1 -- 172.21.15.6:0/505362249 >> v2:172.21.15.92:3300/0 conn(0x7f6d780c5c20 msgr2=0x7f6d78156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:02.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.344+0000 7f6d7dc00640 1 --2- 172.21.15.6:0/505362249 >> v2:172.21.15.92:3300/0 conn(0x7f6d780c5c20 0x7f6d78156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:02.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.344+0000 7f6d7dc00640 1 -- 172.21.15.6:0/505362249 >> v1:172.21.15.92:6789/0 conn(0x7f6d78157410 legacy=0x7f6d7806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:02.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.344+0000 7f6d7dc00640 1 -- 172.21.15.6:0/505362249 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78072150 con 0x7f6d78150f50 2024-03-22T13:10:02.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.348+0000 7f6d7dc00640 1 --2- 172.21.15.6:0/505362249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d780c79e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009870 tx=0x7f6d6c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeca9eacb5c7ea70 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:02.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.354+0000 7f6d77e00640 1 -- 172.21.15.6:0/505362249 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c034070 con 0x7f6d78150f50 2024-03-22T13:10:02.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.354+0000 7f6d77e00640 1 -- 172.21.15.6:0/505362249 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c030840 con 0x7f6d78150f50 2024-03-22T13:10:02.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.354+0000 7f6d77e00640 1 -- 172.21.15.6:0/505362249 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c030c10 con 0x7f6d78150f50 2024-03-22T13:10:02.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.355+0000 7f6d80600640 1 -- 172.21.15.6:0/505362249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 msgr2=0x7f6d780c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:02.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.355+0000 7f6d80600640 1 --2- 172.21.15.6:0/505362249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d780c79e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009870 tx=0x7f6d6c02b980 comp rx=0 tx=0).stop 2024-03-22T13:10:02.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.355+0000 7f6d80600640 1 -- 172.21.15.6:0/505362249 shutdown_connections 2024-03-22T13:10:02.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.355+0000 7f6d80600640 1 --2- 172.21.15.6:0/505362249 >> v2:172.21.15.92:3300/0 conn(0x7f6d780c5c20 0x7f6d78156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:02.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.355+0000 7f6d80600640 1 --2- 172.21.15.6:0/505362249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d780c79e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:02.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.355+0000 7f6d80600640 1 -- 172.21.15.6:0/505362249 >> 172.21.15.6:0/505362249 conn(0x7f6d780cb930 msgr2=0x7f6d780cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:02.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.356+0000 7f6d80600640 1 -- 172.21.15.6:0/505362249 shutdown_connections 2024-03-22T13:10:02.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.356+0000 7f6d80600640 1 -- 172.21.15.6:0/505362249 wait complete. 2024-03-22T13:10:02.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.358+0000 7f6d80600640 1 Processor -- start 2024-03-22T13:10:02.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.359+0000 7f6d80600640 1 -- start start 2024-03-22T13:10:02.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.360+0000 7f6d80600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d780c5c20 0x7f6d781dd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:02.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.360+0000 7f6d80600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d781f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:02.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.360+0000 7f6d80600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d78157410 0x7f6d78074730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:02.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.360+0000 7f6d80600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d780c33f0 con 0x7f6d780c5c20 2024-03-22T13:10:02.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.360+0000 7f6d80600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d780c3110 con 0x7f6d78157410 2024-03-22T13:10:02.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.360+0000 7f6d80600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78151330 con 0x7f6d78150f50 2024-03-22T13:10:02.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.361+0000 7f6d7e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d78157410 0x7f6d78074730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:02.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.361+0000 7f6d7d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d781f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:02.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.361+0000 7f6d7dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d780c5c20 0x7f6d781dd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:02.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.361+0000 7f6d7e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d78157410 0x7f6d78074730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:57784/0 (socket says 172.21.15.6:57784) 2024-03-22T13:10:02.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.361+0000 7f6d7e600640 1 -- 172.21.15.6:0/3107880593 learned_addr learned my addr 172.21.15.6:0/3107880593 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.365+0000 7f6d7dc00640 1 -- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 msgr2=0x7f6d781f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.365+0000 7f6d7dc00640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d781f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.366+0000 7f6d7dc00640 1 -- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d78157410 msgr2=0x7f6d78074730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:02.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.366+0000 7f6d7dc00640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d78157410 0x7f6d78074730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:02.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.366+0000 7f6d7dc00640 1 -- 172.21.15.6:0/3107880593 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78146e40 con 0x7f6d780c5c20 2024-03-22T13:10:02.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.369+0000 7f6d7dc00640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d780c5c20 0x7f6d781dd250 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c033000 tx=0x7f6d6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:02.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.374+0000 7f6d75600640 1 -- 172.21.15.6:0/3107880593 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c048070 con 0x7f6d780c5c20 2024-03-22T13:10:02.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.374+0000 7f6d75600640 1 -- 172.21.15.6:0/3107880593 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c030640 con 0x7f6d780c5c20 2024-03-22T13:10:02.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.374+0000 7f6d75600640 1 -- 172.21.15.6:0/3107880593 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c034040 con 0x7f6d780c5c20 2024-03-22T13:10:02.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.374+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d781f40f0 con 0x7f6d780c5c20 2024-03-22T13:10:02.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.374+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d781470c0 con 0x7f6d780c5c20 2024-03-22T13:10:02.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.376+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d780756c0 con 0x7f6d780c5c20 2024-03-22T13:10:02.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.406+0000 7f6d75600640 1 -- 172.21.15.6:0/3107880593 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6d6c04b070 con 0x7f6d780c5c20 2024-03-22T13:10:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.407+0000 7f6d75600640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d5806bc20 0x7f6d5806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.408+0000 7f6d7d200640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d5806bc20 0x7f6d5806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:02.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.408+0000 7f6d7d200640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d5806bc20 0x7f6d5806e0e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6d781467a0 tx=0x7f6d64009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:02.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.732+0000 7f6d75600640 1 -- 172.21.15.6:0/3107880593 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f6d6c0307f0 con 0x7f6d780c5c20 2024-03-22T13:10:02.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.732+0000 7f6d75600640 1 -- 172.21.15.6:0/3107880593 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6d6c052020 con 0x7f6d780c5c20 2024-03-22T13:10:02.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:02.994+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d781efc80 con 0x7f6d780c5c20 2024-03-22T13:10:03.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.017+0000 7f6d75600640 1 -- 172.21.15.6:0/3107880593 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f6d78146e40 con 0x7f6d780c5c20 2024-03-22T13:10:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:03.017 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:03.018 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:03.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d5806bc20 msgr2=0x7f6d5806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:03.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d5806bc20 0x7f6d5806e0e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6d781467a0 tx=0x7f6d64009110 comp rx=0 tx=0).stop 2024-03-22T13:10:03.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d780c5c20 msgr2=0x7f6d781dd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:03.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d780c5c20 0x7f6d781dd250 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c033000 tx=0x7f6d6c004730 comp rx=0 tx=0).stop 2024-03-22T13:10:03.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 shutdown_connections 2024-03-22T13:10:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f6d5806bc20 0x7f6d5806e0e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d78157410 0x7f6d78074730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d78150f50 0x7f6d781f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 --2- 172.21.15.6:0/3107880593 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d780c5c20 0x7f6d781dd250 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 >> 172.21.15.6:0/3107880593 conn(0x7f6d780cb930 msgr2=0x7f6d78155bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 shutdown_connections 2024-03-22T13:10:03.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:03.020+0000 7f6d80600640 1 -- 172.21.15.6:0/3107880593 wait complete. 2024-03-22T13:10:03.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:10:03.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:10:03.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:03.642 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:04.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:04.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.851+0000 7f4916a00640 1 Processor -- start 2024-03-22T13:10:04.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.851+0000 7f4916a00640 1 -- start start 2024-03-22T13:10:04.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.853+0000 7f4916a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4910154980 0x7f49100ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:04.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.853+0000 7f4916a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49100ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:04.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.853+0000 7f4916a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49100cc380 con 0x7f49100cd300 2024-03-22T13:10:04.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.853+0000 7f4916a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491015a520 con 0x7f4910154980 2024-03-22T13:10:04.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.853+0000 7f4916a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100cc660 con 0x7f491015b0c0 2024-03-22T13:10:04.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.854+0000 7f4914a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49100ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:04.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.854+0000 7f490f400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f49100cd300 0x7f491015a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:46230/0 (socket says 172.21.15.6:46230) 2024-03-22T13:10:04.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.854+0000 7f490f400640 1 -- 172.21.15.6:0/3071483298 learned_addr learned my addr 172.21.15.6:0/3071483298 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:04.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.854+0000 7f490fe00640 1 --2- 172.21.15.6:0/3071483298 >> v2:172.21.15.6:3300/0 conn(0x7f4910154980 0x7f49100ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.857+0000 7f490ea00640 1 -- 172.21.15.6:0/3071483298 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040729881 0 0) 0x7f49100cc380 con 0x7f49100cd300 2024-03-22T13:10:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.857+0000 7f490ea00640 1 -- 172.21.15.6:0/3071483298 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f4003520 con 0x7f49100cd300 2024-03-22T13:10:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.858+0000 7f4914a00640 1 -- 172.21.15.6:0/3071483298 >> v2:172.21.15.6:3300/0 conn(0x7f4910154980 msgr2=0x7f49100ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.858+0000 7f4914a00640 1 --2- 172.21.15.6:0/3071483298 >> v2:172.21.15.6:3300/0 conn(0x7f4910154980 0x7f49100ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.858+0000 7f4914a00640 1 -- 172.21.15.6:0/3071483298 >> v1:172.21.15.92:6789/0 conn(0x7f49100cd300 legacy=0x7f491015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.858+0000 7f4914a00640 1 -- 172.21.15.6:0/3071483298 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100cc940 con 0x7f491015b0c0 2024-03-22T13:10:04.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.861+0000 7f4914a00640 1 --2- 172.21.15.6:0/3071483298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49100ce840 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4908009d90 tx=0x7f490802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca46812a8ca9d2f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.866+0000 7f490ea00640 1 -- 172.21.15.6:0/3071483298 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908039070 con 0x7f491015b0c0 2024-03-22T13:10:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.866+0000 7f490ea00640 1 -- 172.21.15.6:0/3071483298 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490802cc50 con 0x7f491015b0c0 2024-03-22T13:10:04.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.866+0000 7f490ea00640 1 -- 172.21.15.6:0/3071483298 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908034440 con 0x7f491015b0c0 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 -- 172.21.15.6:0/3071483298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 msgr2=0x7f49100ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 --2- 172.21.15.6:0/3071483298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49100ce840 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4908009d90 tx=0x7f490802bea0 comp rx=0 tx=0).stop 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 -- 172.21.15.6:0/3071483298 shutdown_connections 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 --2- 172.21.15.6:0/3071483298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49100ce840 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 --2- 172.21.15.6:0/3071483298 >> v2:172.21.15.6:3300/0 conn(0x7f4910154980 0x7f49100ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 -- 172.21.15.6:0/3071483298 >> 172.21.15.6:0/3071483298 conn(0x7f49100b16c0 msgr2=0x7f49100b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 -- 172.21.15.6:0/3071483298 shutdown_connections 2024-03-22T13:10:04.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.867+0000 7f4916a00640 1 -- 172.21.15.6:0/3071483298 wait complete. 2024-03-22T13:10:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.869+0000 7f4916a00640 1 Processor -- start 2024-03-22T13:10:04.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.870+0000 7f4916a00640 1 -- start start 2024-03-22T13:10:04.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.871+0000 7f4916a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49100cd300 0x7f491015b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:04.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.871+0000 7f4916a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4910154980 0x7f491014ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:04.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.871+0000 7f4916a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49101f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:04.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.871+0000 7f4916a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100ceea0 con 0x7f4910154980 2024-03-22T13:10:04.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.871+0000 7f4916a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100ccf00 con 0x7f49100cd300 2024-03-22T13:10:04.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.871+0000 7f4916a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100cf020 con 0x7f491015b0c0 2024-03-22T13:10:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.872+0000 7f4914a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49101f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.872+0000 7f490f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4910154980 0x7f491014ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.872+0000 7f490fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49100cd300 0x7f491015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.872+0000 7f490f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4910154980 0x7f491014ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37796/0 (socket says 172.21.15.6:37796) 2024-03-22T13:10:04.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.872+0000 7f490f400640 1 -- 172.21.15.6:0/648560583 learned_addr learned my addr 172.21.15.6:0/648560583 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:04.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.876+0000 7f490f400640 1 -- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 msgr2=0x7f49101f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:04.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.876+0000 7f490f400640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49101f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:04.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.876+0000 7f490f400640 1 -- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49100cd300 msgr2=0x7f491015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:04.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.876+0000 7f490f400640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49100cd300 0x7f491015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:04.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.876+0000 7f490f400640 1 -- 172.21.15.6:0/648560583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49101efb60 con 0x7f4910154980 2024-03-22T13:10:04.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.879+0000 7f490f400640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4910154980 0x7f491014ee90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c400 tx=0x7f48fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:04.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.883+0000 7f4907e00640 1 -- 172.21.15.6:0/648560583 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc017070 con 0x7f4910154980 2024-03-22T13:10:04.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.883+0000 7f4907e00640 1 -- 172.21.15.6:0/648560583 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc005d00 con 0x7f4910154980 2024-03-22T13:10:04.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.883+0000 7f4907e00640 1 -- 172.21.15.6:0/648560583 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc007310 con 0x7f4910154980 2024-03-22T13:10:04.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.883+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49101554f0 con 0x7f4910154980 2024-03-22T13:10:04.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.883+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4910155a70 con 0x7f4910154980 2024-03-22T13:10:04.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.885+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49100756c0 con 0x7f4910154980 2024-03-22T13:10:04.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.907+0000 7f4907e00640 1 -- 172.21.15.6:0/648560583 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f48fc01a070 con 0x7f4910154980 2024-03-22T13:10:04.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.908+0000 7f4907e00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f48f006bbd0 0x7f48f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:04.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.909+0000 7f490fe00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f48f006bbd0 0x7f48f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:04.910+0000 7f490fe00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f48f006bbd0 0x7f48f006e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4900009b40 tx=0x7f4900009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:05.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.228+0000 7f4907e00640 1 -- 172.21.15.6:0/648560583 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f48fc056840 con 0x7f4910154980 2024-03-22T13:10:05.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.228+0000 7f4907e00640 1 -- 172.21.15.6:0/648560583 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f48fc021020 con 0x7f4910154980 2024-03-22T13:10:05.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.526+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49100b5b30 con 0x7f4910154980 2024-03-22T13:10:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.550+0000 7f4907e00640 1 -- 172.21.15.6:0/648560583 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f49101efb60 con 0x7f4910154980 2024-03-22T13:10:05.550 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:05.551 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f48f006bbd0 msgr2=0x7f48f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f48f006bbd0 0x7f48f006e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4900009b40 tx=0x7f4900009190 comp rx=0 tx=0).stop 2024-03-22T13:10:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4910154980 msgr2=0x7f491014ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4910154980 0x7f491014ee90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c400 tx=0x7f48fc00c8c0 comp rx=0 tx=0).stop 2024-03-22T13:10:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 shutdown_connections 2024-03-22T13:10:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f491015b0c0 0x7f49101f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4910154980 0x7f491014ee90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f48f006bbd0 0x7f48f006e090 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 --2- 172.21.15.6:0/648560583 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49100cd300 0x7f491015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 >> 172.21.15.6:0/648560583 conn(0x7f49100b16c0 msgr2=0x7f49101517c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 shutdown_connections 2024-03-22T13:10:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:05.553+0000 7f4916a00640 1 -- 172.21.15.6:0/648560583 wait complete. 2024-03-22T13:10:06.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:06.177 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:07.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.382+0000 7fee9d200640 1 Processor -- start 2024-03-22T13:10:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.382+0000 7fee9d200640 1 -- start start 2024-03-22T13:10:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.383+0000 7fee9d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee9806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.384+0000 7fee9d200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fee9815ae30 0x7fee980cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.384+0000 7fee9d200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9815b260 con 0x7fee980cc380 2024-03-22T13:10:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.384+0000 7fee9d200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee980cee00 con 0x7fee9815ae30 2024-03-22T13:10:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.384+0000 7fee9d200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee980cf0c0 con 0x7fee980ce8a0 2024-03-22T13:10:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.384+0000 7fee96a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee9806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.385+0000 7fee97e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fee9815ae30 0x7fee980cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.385+0000 7fee96a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee9806c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36840/0 (socket says 172.21.15.6:36840) 2024-03-22T13:10:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.385+0000 7fee96a00640 1 -- 172.21.15.6:0/989117839 learned_addr learned my addr 172.21.15.6:0/989117839 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:07.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.387+0000 7fee96a00640 1 -- 172.21.15.6:0/989117839 >> v2:172.21.15.6:3300/0 conn(0x7fee9815ae30 msgr2=0x7fee980cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:07.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.387+0000 7fee96a00640 1 --2- 172.21.15.6:0/989117839 >> v2:172.21.15.6:3300/0 conn(0x7fee9815ae30 0x7fee980cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:07.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.387+0000 7fee96a00640 1 -- 172.21.15.6:0/989117839 >> v1:172.21.15.6:6789/0 conn(0x7fee980cc380 legacy=0x7fee980cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:07.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.387+0000 7fee96a00640 1 -- 172.21.15.6:0/989117839 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee980cf3a0 con 0x7fee980ce8a0 2024-03-22T13:10:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.389+0000 7fee96a00640 1 --2- 172.21.15.6:0/989117839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee9806c230 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fee84009870 tx=0x7fee8402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecf5b3aeb9fc0123 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.392+0000 7fee96000640 1 -- 172.21.15.6:0/989117839 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84034070 con 0x7fee980ce8a0 2024-03-22T13:10:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.393+0000 7fee96000640 1 -- 172.21.15.6:0/989117839 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee84030840 con 0x7fee980ce8a0 2024-03-22T13:10:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.394+0000 7fee96000640 1 -- 172.21.15.6:0/989117839 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84030c10 con 0x7fee980ce8a0 2024-03-22T13:10:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.394+0000 7fee9d200640 1 -- 172.21.15.6:0/989117839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 msgr2=0x7fee9806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.394+0000 7fee9d200640 1 --2- 172.21.15.6:0/989117839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee9806c230 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fee84009870 tx=0x7fee8402b980 comp rx=0 tx=0).stop 2024-03-22T13:10:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.394+0000 7fee9d200640 1 -- 172.21.15.6:0/989117839 shutdown_connections 2024-03-22T13:10:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.394+0000 7fee9d200640 1 --2- 172.21.15.6:0/989117839 >> v2:172.21.15.6:3300/0 conn(0x7fee9815ae30 0x7fee980cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.394+0000 7fee9d200640 1 --2- 172.21.15.6:0/989117839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee9806c230 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.394+0000 7fee9d200640 1 -- 172.21.15.6:0/989117839 >> 172.21.15.6:0/989117839 conn(0x7fee980b1480 msgr2=0x7fee980b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.395+0000 7fee9d200640 1 -- 172.21.15.6:0/989117839 shutdown_connections 2024-03-22T13:10:07.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.395+0000 7fee9d200640 1 -- 172.21.15.6:0/989117839 wait complete. 2024-03-22T13:10:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.397+0000 7fee9d200640 1 Processor -- start 2024-03-22T13:10:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.397+0000 7fee9d200640 1 -- start start 2024-03-22T13:10:07.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.398+0000 7fee9d200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee980cc380 0x7fee9814f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:07.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.398+0000 7fee9d200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee981e5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.398+0000 7fee9d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee9815ae30 0x7fee981537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.398+0000 7fee9d200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9815a910 con 0x7fee9815ae30 2024-03-22T13:10:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.398+0000 7fee9d200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9815a630 con 0x7fee980cc380 2024-03-22T13:10:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.398+0000 7fee9d200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee981549d0 con 0x7fee980ce8a0 2024-03-22T13:10:07.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.398+0000 7fee96a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee981e5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.399+0000 7fee97e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee9815ae30 0x7fee981537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.399+0000 7fee97400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee980cc380 0x7fee9814f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.399+0000 7fee96a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee981e5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36846/0 (socket says 172.21.15.6:36846) 2024-03-22T13:10:07.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.399+0000 7fee96a00640 1 -- 172.21.15.6:0/2345847086 learned_addr learned my addr 172.21.15.6:0/2345847086 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.404+0000 7fee96a00640 1 -- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee980cc380 msgr2=0x7fee9814f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.404+0000 7fee96a00640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee980cc380 0x7fee9814f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.404+0000 7fee96a00640 1 -- 172.21.15.6:0/2345847086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee9815ae30 msgr2=0x7fee981537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.404+0000 7fee96a00640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee9815ae30 0x7fee981537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:07.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.404+0000 7fee96a00640 1 -- 172.21.15.6:0/2345847086 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee980cb4d0 con 0x7fee980ce8a0 2024-03-22T13:10:07.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.407+0000 7fee96a00640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee981e5b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fee84000c00 tx=0x7fee84004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:07.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.412+0000 7fee8f400640 1 -- 172.21.15.6:0/2345847086 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84048070 con 0x7fee980ce8a0 2024-03-22T13:10:07.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.412+0000 7fee8f400640 1 -- 172.21.15.6:0/2345847086 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee84030640 con 0x7fee980ce8a0 2024-03-22T13:10:07.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.412+0000 7fee8f400640 1 -- 172.21.15.6:0/2345847086 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84034040 con 0x7fee980ce8a0 2024-03-22T13:10:07.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.412+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee981e5010 con 0x7fee980ce8a0 2024-03-22T13:10:07.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.413+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee981e55d0 con 0x7fee980ce8a0 2024-03-22T13:10:07.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.414+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee98075650 con 0x7fee980ce8a0 2024-03-22T13:10:07.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.696+0000 7fee8f400640 1 -- 172.21.15.6:0/2345847086 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee8404b070 con 0x7fee980ce8a0 2024-03-22T13:10:07.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.698+0000 7fee8f400640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fee7806bc20 0x7fee7806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:07.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.698+0000 7fee97400640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fee7806bc20 0x7fee7806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:07.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.699+0000 7fee97400640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fee7806bc20 0x7fee7806e0e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fee88001e70 tx=0x7fee88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:07.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.699+0000 7fee8f400640 1 -- 172.21.15.6:0/2345847086 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fee840307f0 con 0x7fee980ce8a0 2024-03-22T13:10:07.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.715+0000 7fee8f400640 1 -- 172.21.15.6:0/2345847086 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee84052020 con 0x7fee980ce8a0 2024-03-22T13:10:07.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:07.993+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee980ca380 con 0x7fee980ce8a0 2024-03-22T13:10:08.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.014+0000 7fee8f400640 1 -- 172.21.15.6:0/2345847086 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fee8407b470 con 0x7fee980ce8a0 2024-03-22T13:10:08.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:08.014 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:08.015 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:08.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fee7806bc20 msgr2=0x7fee7806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:08.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fee7806bc20 0x7fee7806e0e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fee88001e70 tx=0x7fee88009040 comp rx=0 tx=0).stop 2024-03-22T13:10:08.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 msgr2=0x7fee981e5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee981e5b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fee84000c00 tx=0x7fee84004730 comp rx=0 tx=0).stop 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 shutdown_connections 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee9815ae30 0x7fee981537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee980ce8a0 0x7fee981e5b60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fee7806bc20 0x7fee7806e0e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 --2- 172.21.15.6:0/2345847086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee980cc380 0x7fee9814f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 >> 172.21.15.6:0/2345847086 conn(0x7fee980b1480 msgr2=0x7fee980cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 shutdown_connections 2024-03-22T13:10:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:08.018+0000 7fee9d200640 1 -- 172.21.15.6:0/2345847086 wait complete. 2024-03-22T13:10:08.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:08.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:09.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:09.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.845+0000 7f87af200640 1 Processor -- start 2024-03-22T13:10:09.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.846+0000 7f87af200640 1 -- start start 2024-03-22T13:10:09.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.847+0000 7f87af200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b015b7a0 0x7f87b015e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:09.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.847+0000 7f87af200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b006c440 con 0x7f87b00cca20 2024-03-22T13:10:09.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.847+0000 7f87af200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b006c720 con 0x7f87b00cc640 2024-03-22T13:10:09.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.847+0000 7f87af200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b006d330 con 0x7f87b015b7a0 2024-03-22T13:10:09.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.848+0000 7f87ae800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b015b7a0 0x7f87b015e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:09.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.848+0000 7f87ae800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b015b7a0 0x7f87b015e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36850/0 (socket says 172.21.15.6:36850) 2024-03-22T13:10:09.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.848+0000 7f87ae800640 1 -- 172.21.15.6:0/2059714692 learned_addr learned my addr 172.21.15.6:0/2059714692 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:09.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.852+0000 7f87ae800640 1 -- 172.21.15.6:0/2059714692 >> v1:172.21.15.92:6789/0 conn(0x7f87b00cc640 legacy=0x7f87b006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:09.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.852+0000 7f87ae800640 1 -- 172.21.15.6:0/2059714692 >> v1:172.21.15.6:6789/0 conn(0x7f87b00cca20 legacy=0x7f87b00ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:09.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.852+0000 7f87ae800640 1 -- 172.21.15.6:0/2059714692 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b006ca00 con 0x7f87b015b7a0 2024-03-22T13:10:09.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.855+0000 7f87ae800640 1 --2- 172.21.15.6:0/2059714692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b015b7a0 0x7f87b015e260 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009e80 tx=0x7f87a802bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9849ab9bb81ed6d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:09.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.859+0000 7f87aca00640 1 -- 172.21.15.6:0/2059714692 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a802caf0 con 0x7f87b015b7a0 2024-03-22T13:10:09.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.859+0000 7f87aca00640 1 -- 172.21.15.6:0/2059714692 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a802cc90 con 0x7f87b015b7a0 2024-03-22T13:10:09.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.859+0000 7f87aca00640 1 -- 172.21.15.6:0/2059714692 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8034610 con 0x7f87b015b7a0 2024-03-22T13:10:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.860+0000 7f87af200640 1 -- 172.21.15.6:0/2059714692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b015b7a0 msgr2=0x7f87b015e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.860+0000 7f87af200640 1 --2- 172.21.15.6:0/2059714692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b015b7a0 0x7f87b015e260 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009e80 tx=0x7f87a802bf50 comp rx=0 tx=0).stop 2024-03-22T13:10:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.860+0000 7f87af200640 1 -- 172.21.15.6:0/2059714692 shutdown_connections 2024-03-22T13:10:09.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.861+0000 7f87af200640 1 --2- 172.21.15.6:0/2059714692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b015b7a0 0x7f87b015e260 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:09.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.861+0000 7f87af200640 1 -- 172.21.15.6:0/2059714692 >> 172.21.15.6:0/2059714692 conn(0x7f87b00cf9c0 msgr2=0x7f87b00cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:09.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.861+0000 7f87af200640 1 -- 172.21.15.6:0/2059714692 shutdown_connections 2024-03-22T13:10:09.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.861+0000 7f87af200640 1 -- 172.21.15.6:0/2059714692 wait complete. 2024-03-22T13:10:09.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.863+0000 7f87af200640 1 Processor -- start 2024-03-22T13:10:09.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.863+0000 7f87af200640 1 -- start start 2024-03-22T13:10:09.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.864+0000 7f87af200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc640 0x7f87b0156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.864+0000 7f87af200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cca20 0x7f87b01e5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.864+0000 7f87af200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b015b7a0 0x7f87b0154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.864+0000 7f87af200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00caa50 con 0x7f87b015b7a0 2024-03-22T13:10:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.864+0000 7f87af200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b006cfc0 con 0x7f87b00cca20 2024-03-22T13:10:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.864+0000 7f87af200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b015b000 con 0x7f87b00cc640 2024-03-22T13:10:09.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.865+0000 7f87ade00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc640 0x7f87b0156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:09.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.866+0000 7f87ad400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cca20 0x7f87b01e5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:09.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.866+0000 7f87ade00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc640 0x7f87b0156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:36860/0 (socket says 172.21.15.6:36860) 2024-03-22T13:10:09.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.866+0000 7f87ade00640 1 -- 172.21.15.6:0/358297097 learned_addr learned my addr 172.21.15.6:0/358297097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.872+0000 7f87ade00640 1 -- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cca20 msgr2=0x7f87b01e5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.872+0000 7f87ade00640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cca20 0x7f87b01e5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.872+0000 7f87ade00640 1 -- 172.21.15.6:0/358297097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b015b7a0 msgr2=0x7f87b0154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.872+0000 7f87ade00640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b015b7a0 0x7f87b0154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:09.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.872+0000 7f87ade00640 1 -- 172.21.15.6:0/358297097 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b01f02a0 con 0x7f87b00cc640 2024-03-22T13:10:09.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.879+0000 7f87ade00640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc640 0x7f87b0156560 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f87a000b410 tx=0x7f87a000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:09.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.885+0000 7f87a6000640 1 -- 172.21.15.6:0/358297097 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0015020 con 0x7f87b00cc640 2024-03-22T13:10:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.885+0000 7f87a6000640 1 -- 172.21.15.6:0/358297097 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a0011850 con 0x7f87b00cc640 2024-03-22T13:10:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.885+0000 7f87a6000640 1 -- 172.21.15.6:0/358297097 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0011c40 con 0x7f87b00cc640 2024-03-22T13:10:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.885+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b01f0520 con 0x7f87b00cc640 2024-03-22T13:10:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.885+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b01f0ae0 con 0x7f87b00cc640 2024-03-22T13:10:09.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.886+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b01594a0 con 0x7f87b00cc640 2024-03-22T13:10:09.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.922+0000 7f87a6000640 1 -- 172.21.15.6:0/358297097 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f87a0018020 con 0x7f87b00cc640 2024-03-22T13:10:09.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.923+0000 7f87a6000640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc50 0x7f879006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:09.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.924+0000 7f87ad400640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc50 0x7f879006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:09.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:09.924+0000 7f87ad400640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc50 0x7f879006e110 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f87b0159ba0 tx=0x7f879c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.169+0000 7f87a6000640 1 -- 172.21.15.6:0/358297097 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f87a0011de0 con 0x7f87b00cc640 2024-03-22T13:10:10.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.187+0000 7f87a6000640 1 -- 172.21.15.6:0/358297097 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f87b01594a0 con 0x7f87b00cc640 2024-03-22T13:10:10.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.450+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b006d7a0 con 0x7f87b00cc640 2024-03-22T13:10:10.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.464+0000 7f87a6000640 1 -- 172.21.15.6:0/358297097 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f87a001f020 con 0x7f87b00cc640 2024-03-22T13:10:10.464 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:10.464 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:10.464 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:10.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc50 msgr2=0x7f879006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc50 0x7f879006e110 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f87b0159ba0 tx=0x7f879c009040 comp rx=0 tx=0).stop 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc640 msgr2=0x7f87b0156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc640 0x7f87b0156560 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f87a000b410 tx=0x7f87a000b8d0 comp rx=0 tx=0).stop 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 shutdown_connections 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87b015b7a0 0x7f87b0154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f879006bc50 0x7f879006e110 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87b00cca20 0x7f87b01e5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 --2- 172.21.15.6:0/358297097 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87b00cc640 0x7f87b0156560 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.466+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 >> 172.21.15.6:0/358297097 conn(0x7f87b00cf9c0 msgr2=0x7f87b0153c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.467+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 shutdown_connections 2024-03-22T13:10:10.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:10.467+0000 7f87af200640 1 -- 172.21.15.6:0/358297097 wait complete. 2024-03-22T13:10:11.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:11.090 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:12.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:12.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.288+0000 7f8250800640 1 Processor -- start 2024-03-22T13:10:12.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.289+0000 7f8250800640 1 -- start start 2024-03-22T13:10:12.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.290+0000 7f8250800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f824806f8c0 0x7f82480716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.290+0000 7f8250800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8248071c10 0x7f82480c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.290+0000 7f8250800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82481590b0 con 0x7f8248152400 2024-03-22T13:10:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.290+0000 7f8250800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82480724d0 con 0x7f8248071c10 2024-03-22T13:10:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.290+0000 7f8250800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82480721f0 con 0x7f824806f8c0 2024-03-22T13:10:12.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.291+0000 7f824e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8248071c10 0x7f82480c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:12.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.291+0000 7f824e800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8248071c10 0x7f82480c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36048/0 (socket says 172.21.15.6:36048) 2024-03-22T13:10:12.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.291+0000 7f824e800640 1 -- 172.21.15.6:0/1404463056 learned_addr learned my addr 172.21.15.6:0/1404463056 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:12.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.294+0000 7f824e800640 1 -- 172.21.15.6:0/1404463056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f824806f8c0 msgr2=0x7f82480716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.294+0000 7f824e800640 1 --2- 172.21.15.6:0/1404463056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f824806f8c0 0x7f82480716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.294+0000 7f824e800640 1 -- 172.21.15.6:0/1404463056 >> v1:172.21.15.6:6789/0 conn(0x7f8248152400 legacy=0x7f824815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:12.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.294+0000 7f824e800640 1 -- 172.21.15.6:0/1404463056 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82481582f0 con 0x7f8248071c10 2024-03-22T13:10:12.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.298+0000 7f824e800640 1 --2- 172.21.15.6:0/1404463056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248071c10 0x7f82480c5040 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8244009e50 tx=0x7f824402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d44216652ca8165 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.302+0000 7f824ca00640 1 -- 172.21.15.6:0/1404463056 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824402cb00 con 0x7f8248071c10 2024-03-22T13:10:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.302+0000 7f824ca00640 1 -- 172.21.15.6:0/1404463056 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824402cca0 con 0x7f8248071c10 2024-03-22T13:10:12.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.302+0000 7f824ca00640 1 -- 172.21.15.6:0/1404463056 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244034610 con 0x7f8248071c10 2024-03-22T13:10:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 -- 172.21.15.6:0/1404463056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248071c10 msgr2=0x7f82480c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 --2- 172.21.15.6:0/1404463056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248071c10 0x7f82480c5040 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8244009e50 tx=0x7f824402bf60 comp rx=0 tx=0).stop 2024-03-22T13:10:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 -- 172.21.15.6:0/1404463056 shutdown_connections 2024-03-22T13:10:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 --2- 172.21.15.6:0/1404463056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248071c10 0x7f82480c5040 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 --2- 172.21.15.6:0/1404463056 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f824806f8c0 0x7f82480716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 -- 172.21.15.6:0/1404463056 >> 172.21.15.6:0/1404463056 conn(0x7f82480cb930 msgr2=0x7f82480cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 -- 172.21.15.6:0/1404463056 shutdown_connections 2024-03-22T13:10:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.303+0000 7f8250800640 1 -- 172.21.15.6:0/1404463056 wait complete. 2024-03-22T13:10:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.305+0000 7f8250800640 1 Processor -- start 2024-03-22T13:10:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.305+0000 7f8250800640 1 -- start start 2024-03-22T13:10:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.306+0000 7f8250800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f824806f8c0 0x7f82481e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.306+0000 7f8250800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8248071c10 0x7f82481e5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.306+0000 7f8250800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248152400 0x7f824814e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.306+0000 7f8250800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8248158b90 con 0x7f824806f8c0 2024-03-22T13:10:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.306+0000 7f8250800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82481588b0 con 0x7f8248152400 2024-03-22T13:10:12.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.306+0000 7f8250800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82481527e0 con 0x7f8248071c10 2024-03-22T13:10:12.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.307+0000 7f824e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248152400 0x7f824814e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:12.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.308+0000 7f824e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248152400 0x7f824814e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36058/0 (socket says 172.21.15.6:36058) 2024-03-22T13:10:12.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.308+0000 7f824e800640 1 -- 172.21.15.6:0/824306816 learned_addr learned my addr 172.21.15.6:0/824306816 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:12.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.310+0000 7f824d400640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8248071c10 0x7f82481e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:12.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.311+0000 7f824e800640 1 -- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8248071c10 msgr2=0x7f82481e5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:12.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.311+0000 7f824e800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8248071c10 0x7f82481e5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.311+0000 7f824e800640 1 -- 172.21.15.6:0/824306816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f824806f8c0 msgr2=0x7f82481e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:12.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.311+0000 7f824e800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f824806f8c0 0x7f82481e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.311+0000 7f824e800640 1 -- 172.21.15.6:0/824306816 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82481f04c0 con 0x7f8248152400 2024-03-22T13:10:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.316+0000 7f824e800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248152400 0x7f824814e9a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8244036040 tx=0x7f8244004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:12.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.321+0000 7f8242000640 1 -- 172.21.15.6:0/824306816 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244043020 con 0x7f8248152400 2024-03-22T13:10:12.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.321+0000 7f8242000640 1 -- 172.21.15.6:0/824306816 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824402ce50 con 0x7f8248152400 2024-03-22T13:10:12.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.321+0000 7f8242000640 1 -- 172.21.15.6:0/824306816 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244009510 con 0x7f8248152400 2024-03-22T13:10:12.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.321+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82481f07a0 con 0x7f8248152400 2024-03-22T13:10:12.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.321+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82481f0d60 con 0x7f8248152400 2024-03-22T13:10:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.323+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8248072750 con 0x7f8248152400 2024-03-22T13:10:12.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.360+0000 7f8242000640 1 -- 172.21.15.6:0/824306816 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8244046020 con 0x7f8248152400 2024-03-22T13:10:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.361+0000 7f8242000640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f822c06bc50 0x7f822c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.361+0000 7f824de00640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f822c06bc50 0x7f822c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:12.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.362+0000 7f824de00640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f822c06bc50 0x7f822c06e110 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f823c009650 tx=0x7f823c006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:12.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.616+0000 7f8242000640 1 -- 172.21.15.6:0/824306816 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8244079ac0 con 0x7f8248152400 2024-03-22T13:10:12.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.637+0000 7f8242000640 1 -- 172.21.15.6:0/824306816 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8248072750 con 0x7f8248152400 2024-03-22T13:10:12.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.890+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8248072d10 con 0x7f8248152400 2024-03-22T13:10:12.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.904+0000 7f8242000640 1 -- 172.21.15.6:0/824306816 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f8248072d10 con 0x7f8248152400 2024-03-22T13:10:12.904 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:12.904 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:12.905 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f822c06bc50 msgr2=0x7f822c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f822c06bc50 0x7f822c06e110 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f823c009650 tx=0x7f823c006ba0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248152400 msgr2=0x7f824814e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248152400 0x7f824814e9a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8244036040 tx=0x7f8244004a40 comp rx=0 tx=0).stop 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 shutdown_connections 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f822c06bc50 0x7f822c06e110 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8248152400 0x7f824814e9a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8248071c10 0x7f82481e5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 --2- 172.21.15.6:0/824306816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f824806f8c0 0x7f82481e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 >> 172.21.15.6:0/824306816 conn(0x7f82480cb930 msgr2=0x7f82480c1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:12.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 shutdown_connections 2024-03-22T13:10:12.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:12.908+0000 7f8250800640 1 -- 172.21.15.6:0/824306816 wait complete. 2024-03-22T13:10:13.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:13.538 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:14.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:14.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.742+0000 7f5cbe600640 1 Processor -- start 2024-03-22T13:10:14.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.742+0000 7f5cbe600640 1 -- start start 2024-03-22T13:10:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.744+0000 7f5cbe600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.744+0000 7f5cbe600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5cb80ccad0 0x7f5cb815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.744+0000 7f5cbe600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb806c7a0 con 0x7f5cb815b0e0 2024-03-22T13:10:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.744+0000 7f5cbe600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb80cd000 con 0x7f5cb80ccad0 2024-03-22T13:10:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.744+0000 7f5cbe600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb806ca80 con 0x7f5cb80ca520 2024-03-22T13:10:14.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.745+0000 7f5cb7e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5cb80ccad0 0x7f5cb815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:14.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.745+0000 7f5cb7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:14.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.745+0000 7f5cb7e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5cb80ccad0 0x7f5cb815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:41550/0 (socket says 172.21.15.6:41550) 2024-03-22T13:10:14.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.745+0000 7f5cb7e00640 1 -- 172.21.15.6:0/2843313608 learned_addr learned my addr 172.21.15.6:0/2843313608 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.749+0000 7f5cb7400640 1 -- 172.21.15.6:0/2843313608 >> v2:172.21.15.92:3300/0 conn(0x7f5cb80ccad0 msgr2=0x7f5cb815b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.749+0000 7f5cb7400640 1 --2- 172.21.15.6:0/2843313608 >> v2:172.21.15.92:3300/0 conn(0x7f5cb80ccad0 0x7f5cb815b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.749+0000 7f5cb7400640 1 -- 172.21.15.6:0/2843313608 >> v1:172.21.15.92:6789/0 conn(0x7f5cb815b0e0 legacy=0x7f5cb80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.749+0000 7f5cb7400640 1 -- 172.21.15.6:0/2843313608 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb806cd60 con 0x7f5cb80ca520 2024-03-22T13:10:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.751+0000 7f5cb7400640 1 --2- 172.21.15.6:0/2843313608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb806d840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80097a0 tx=0x7f5ca802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf47a49434269e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:14.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.756+0000 7f5cb6000640 1 -- 172.21.15.6:0/2843313608 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8035020 con 0x7f5cb80ca520 2024-03-22T13:10:14.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.756+0000 7f5cb6000640 1 -- 172.21.15.6:0/2843313608 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca8031810 con 0x7f5cb80ca520 2024-03-22T13:10:14.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.756+0000 7f5cb6000640 1 -- 172.21.15.6:0/2843313608 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8031be0 con 0x7f5cb80ca520 2024-03-22T13:10:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 -- 172.21.15.6:0/2843313608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 msgr2=0x7f5cb806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 --2- 172.21.15.6:0/2843313608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb806d840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80097a0 tx=0x7f5ca802b8b0 comp rx=0 tx=0).stop 2024-03-22T13:10:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 -- 172.21.15.6:0/2843313608 shutdown_connections 2024-03-22T13:10:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 --2- 172.21.15.6:0/2843313608 >> v2:172.21.15.92:3300/0 conn(0x7f5cb80ccad0 0x7f5cb815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 --2- 172.21.15.6:0/2843313608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb806d840 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 -- 172.21.15.6:0/2843313608 >> 172.21.15.6:0/2843313608 conn(0x7f5cb80b16b0 msgr2=0x7f5cb80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 -- 172.21.15.6:0/2843313608 shutdown_connections 2024-03-22T13:10:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.757+0000 7f5cbe600640 1 -- 172.21.15.6:0/2843313608 wait complete. 2024-03-22T13:10:14.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.759+0000 7f5cbe600640 1 Processor -- start 2024-03-22T13:10:14.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.759+0000 7f5cbe600640 1 -- start start 2024-03-22T13:10:14.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.760+0000 7f5cbe600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb815db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:14.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.760+0000 7f5cbe600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cb80ccad0 0x7f5cb81f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:14.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.760+0000 7f5cbe600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cb815b0e0 0x7f5cb81f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:14.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.760+0000 7f5cbe600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb8154c20 con 0x7f5cb815b0e0 2024-03-22T13:10:14.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.760+0000 7f5cbe600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb806d320 con 0x7f5cb80ccad0 2024-03-22T13:10:14.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.760+0000 7f5cbe600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80ca900 con 0x7f5cb80ca520 2024-03-22T13:10:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.761+0000 7f5cb6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cb80ccad0 0x7f5cb81f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.761+0000 7f5cb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cb815b0e0 0x7f5cb81f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.761+0000 7f5cb7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb815db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.761+0000 7f5cb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cb815b0e0 0x7f5cb81f3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49264/0 (socket says 172.21.15.6:49264) 2024-03-22T13:10:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.761+0000 7f5cb7e00640 1 -- 172.21.15.6:0/1781380266 learned_addr learned my addr 172.21.15.6:0/1781380266 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.765+0000 7f5cb7e00640 1 -- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 msgr2=0x7f5cb815db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.765+0000 7f5cb7e00640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb815db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.765+0000 7f5cb7e00640 1 -- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cb80ccad0 msgr2=0x7f5cb81f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.765+0000 7f5cb7e00640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cb80ccad0 0x7f5cb81f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.765+0000 7f5cb7e00640 1 -- 172.21.15.6:0/1781380266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb815a210 con 0x7f5cb815b0e0 2024-03-22T13:10:14.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.768+0000 7f5cb7e00640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cb815b0e0 0x7f5cb81f3ec0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000efc0 tx=0x7f5cb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.771+0000 7f5caf400640 1 -- 172.21.15.6:0/1781380266 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb000cd20 con 0x7f5cb815b0e0 2024-03-22T13:10:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.771+0000 7f5caf400640 1 -- 172.21.15.6:0/1781380266 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb0003990 con 0x7f5cb815b0e0 2024-03-22T13:10:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.771+0000 7f5caf400640 1 -- 172.21.15.6:0/1781380266 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0003d80 con 0x7f5cb815b0e0 2024-03-22T13:10:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.771+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb8156fa0 con 0x7f5cb815b0e0 2024-03-22T13:10:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.771+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb8157560 con 0x7f5cb815b0e0 2024-03-22T13:10:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.773+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb80756c0 con 0x7f5cb815b0e0 2024-03-22T13:10:14.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.797+0000 7f5caf400640 1 -- 172.21.15.6:0/1781380266 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5cb0005d10 con 0x7f5cb815b0e0 2024-03-22T13:10:14.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.798+0000 7f5caf400640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5c9806bbd0 0x7f5c9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:14.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.799+0000 7f5caf400640 1 -- 172.21.15.6:0/1781380266 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5cb0058240 con 0x7f5cb815b0e0 2024-03-22T13:10:14.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.799+0000 7f5cb7400640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5c9806bbd0 0x7f5c9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:14.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:14.800+0000 7f5cb7400640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5c9806bbd0 0x7f5c9806e090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8036000 tx=0x7f5ca8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:15.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.117+0000 7f5caf400640 1 -- 172.21.15.6:0/1781380266 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5cb0022020 con 0x7f5cb815b0e0 2024-03-22T13:10:15.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.373+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cb80cb080 con 0x7f5cb815b0e0 2024-03-22T13:10:15.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.392+0000 7f5caf400640 1 -- 172.21.15.6:0/1781380266 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f5cb0010040 con 0x7f5cb815b0e0 2024-03-22T13:10:15.393 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:15.393 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:15.393 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5c9806bbd0 msgr2=0x7f5c9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5c9806bbd0 0x7f5c9806e090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8036000 tx=0x7f5ca8008030 comp rx=0 tx=0).stop 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cb815b0e0 msgr2=0x7f5cb81f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cb815b0e0 0x7f5cb81f3ec0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000efc0 tx=0x7f5cb000c490 comp rx=0 tx=0).stop 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 shutdown_connections 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cb815b0e0 0x7f5cb81f3ec0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5c9806bbd0 0x7f5c9806e090 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cb80ccad0 0x7f5cb81f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 --2- 172.21.15.6:0/1781380266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cb80ca520 0x7f5cb815db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:15.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 >> 172.21.15.6:0/1781380266 conn(0x7f5cb80b16b0 msgr2=0x7f5cb80af620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:15.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 shutdown_connections 2024-03-22T13:10:15.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:15.395+0000 7f5cbe600640 1 -- 172.21.15.6:0/1781380266 wait complete. 2024-03-22T13:10:16.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:16.024 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:17.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:17.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.221+0000 7f9152e00640 1 Processor -- start 2024-03-22T13:10:17.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.221+0000 7f9152e00640 1 -- start start 2024-03-22T13:10:17.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.222+0000 7f9152e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f914c152400 0x7f914c15b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:17.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.222+0000 7f9152e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f914c15bc80 0x7f914c0716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:17.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.222+0000 7f9152e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c1590b0 con 0x7f914c06f8c0 2024-03-22T13:10:17.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.222+0000 7f9152e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914c0724d0 con 0x7f914c15bc80 2024-03-22T13:10:17.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.222+0000 7f9152e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914c0721f0 con 0x7f914c152400 2024-03-22T13:10:17.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.224+0000 7f914be00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f914c152400 0x7f914c15b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:17.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.224+0000 7f9150e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f914c06f8c0 0x7f914c0c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35098/0 (socket says 172.21.15.6:35098) 2024-03-22T13:10:17.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.224+0000 7f9150e00640 1 -- 172.21.15.6:0/1665564635 learned_addr learned my addr 172.21.15.6:0/1665564635 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:17.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.224+0000 7f914b400640 1 --2- 172.21.15.6:0/1665564635 >> v2:172.21.15.6:3300/0 conn(0x7f914c15bc80 0x7f914c0716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.227+0000 7f914be00640 1 -- 172.21.15.6:0/1665564635 >> v2:172.21.15.6:3300/0 conn(0x7f914c15bc80 msgr2=0x7f914c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.227+0000 7f914be00640 1 --2- 172.21.15.6:0/1665564635 >> v2:172.21.15.6:3300/0 conn(0x7f914c15bc80 0x7f914c0716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.227+0000 7f914be00640 1 -- 172.21.15.6:0/1665564635 >> v1:172.21.15.6:6789/0 conn(0x7f914c06f8c0 legacy=0x7f914c0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.227+0000 7f914be00640 1 -- 172.21.15.6:0/1665564635 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c1582f0 con 0x7f914c152400 2024-03-22T13:10:17.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.230+0000 7f914be00640 1 --2- 172.21.15.6:0/1665564635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c152400 0x7f914c15b740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f914c071390 tx=0x7f913c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a467fe33bd268b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:17.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.235+0000 7f914aa00640 1 -- 172.21.15.6:0/1665564635 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c034070 con 0x7f914c152400 2024-03-22T13:10:17.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.235+0000 7f914aa00640 1 -- 172.21.15.6:0/1665564635 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913c030610 con 0x7f914c152400 2024-03-22T13:10:17.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.235+0000 7f914aa00640 1 -- 172.21.15.6:0/1665564635 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c0309e0 con 0x7f914c152400 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 -- 172.21.15.6:0/1665564635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c152400 msgr2=0x7f914c15b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 --2- 172.21.15.6:0/1665564635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c152400 0x7f914c15b740 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f914c071390 tx=0x7f913c02b960 comp rx=0 tx=0).stop 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 -- 172.21.15.6:0/1665564635 shutdown_connections 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 --2- 172.21.15.6:0/1665564635 >> v2:172.21.15.6:3300/0 conn(0x7f914c15bc80 0x7f914c0716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 --2- 172.21.15.6:0/1665564635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c152400 0x7f914c15b740 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 -- 172.21.15.6:0/1665564635 >> 172.21.15.6:0/1665564635 conn(0x7f914c0cb930 msgr2=0x7f914c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 -- 172.21.15.6:0/1665564635 shutdown_connections 2024-03-22T13:10:17.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.236+0000 7f9152e00640 1 -- 172.21.15.6:0/1665564635 wait complete. 2024-03-22T13:10:17.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.238+0000 7f9152e00640 1 Processor -- start 2024-03-22T13:10:17.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.238+0000 7f9152e00640 1 -- start start 2024-03-22T13:10:17.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.239+0000 7f9152e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c06f8c0 0x7f914c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.239+0000 7f9152e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c152400 0x7f914c147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.239+0000 7f9152e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914c15bc80 0x7f914c0ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.239+0000 7f9152e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c158b90 con 0x7f914c152400 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.239+0000 7f9152e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c1588b0 con 0x7f914c06f8c0 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.239+0000 7f9152e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c1527e0 con 0x7f914c15bc80 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.240+0000 7f9150e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914c15bc80 0x7f914c0ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.240+0000 7f914be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c06f8c0 0x7f914c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.240+0000 7f914b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c152400 0x7f914c147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:17.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.240+0000 7f9150e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914c15bc80 0x7f914c0ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46688/0 (socket says 172.21.15.6:46688) 2024-03-22T13:10:17.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.240+0000 7f914b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c152400 0x7f914c147d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49292/0 (socket says 172.21.15.6:49292) 2024-03-22T13:10:17.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.240+0000 7f9150e00640 1 -- 172.21.15.6:0/3416619240 learned_addr learned my addr 172.21.15.6:0/3416619240 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:17.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.244+0000 7f914b400640 1 -- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914c15bc80 msgr2=0x7f914c0ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:17.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.244+0000 7f914b400640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914c15bc80 0x7f914c0ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.244+0000 7f914b400640 1 -- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c06f8c0 msgr2=0x7f914c156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:17.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.244+0000 7f914b400640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c06f8c0 0x7f914c156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.244+0000 7f914b400640 1 -- 172.21.15.6:0/3416619240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c1ef4e0 con 0x7f914c152400 2024-03-22T13:10:17.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.247+0000 7f914b400640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c152400 0x7f914c147d10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f914c071680 tx=0x7f913800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:17.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.253+0000 7f9143e00640 1 -- 172.21.15.6:0/3416619240 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138017070 con 0x7f914c152400 2024-03-22T13:10:17.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.253+0000 7f9143e00640 1 -- 172.21.15.6:0/3416619240 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913800c730 con 0x7f914c152400 2024-03-22T13:10:17.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.253+0000 7f9143e00640 1 -- 172.21.15.6:0/3416619240 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913800cb20 con 0x7f914c152400 2024-03-22T13:10:17.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.253+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c1ef780 con 0x7f914c152400 2024-03-22T13:10:17.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.253+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c1efd40 con 0x7f914c152400 2024-03-22T13:10:17.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.255+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c14f520 con 0x7f914c152400 2024-03-22T13:10:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.284+0000 7f9143e00640 1 -- 172.21.15.6:0/3416619240 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f913801a070 con 0x7f914c152400 2024-03-22T13:10:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.285+0000 7f9143e00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f912c06bc20 0x7f912c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.286+0000 7f914be00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f912c06bc20 0x7f912c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.287+0000 7f9143e00640 1 -- 172.21.15.6:0/3416619240 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9138057070 con 0x7f914c152400 2024-03-22T13:10:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.287+0000 7f914be00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f912c06bc20 0x7f912c06e0e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f913c036000 tx=0x7f913c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:17.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.604+0000 7f9143e00640 1 -- 172.21.15.6:0/3416619240 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9138058fd0 con 0x7f914c152400 2024-03-22T13:10:17.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.854+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f914c14f260 con 0x7f914c152400 2024-03-22T13:10:17.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.878+0000 7f9143e00640 1 -- 172.21.15.6:0/3416619240 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f913805d070 con 0x7f914c152400 2024-03-22T13:10:17.879 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:17.879 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:17.879 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:17.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f912c06bc20 msgr2=0x7f912c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:17.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f912c06bc20 0x7f912c06e0e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f913c036000 tx=0x7f913c03b040 comp rx=0 tx=0).stop 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c152400 msgr2=0x7f914c147d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c152400 0x7f914c147d10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f914c071680 tx=0x7f913800ecd0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 shutdown_connections 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914c15bc80 0x7f914c0ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c152400 0x7f914c147d10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f912c06bc20 0x7f912c06e0e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.881+0000 7f9152e00640 1 --2- 172.21.15.6:0/3416619240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914c06f8c0 0x7f914c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.882+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 >> 172.21.15.6:0/3416619240 conn(0x7f914c0cb930 msgr2=0x7f914c0c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.882+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 shutdown_connections 2024-03-22T13:10:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:17.882+0000 7f9152e00640 1 -- 172.21.15.6:0/3416619240 wait complete. 2024-03-22T13:10:18.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:18.504 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:19.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:19.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.706+0000 7f22bd800640 1 Processor -- start 2024-03-22T13:10:19.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.706+0000 7f22bd800640 1 -- start start 2024-03-22T13:10:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.707+0000 7f22bd800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f22b80ccad0 0x7f22b806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.707+0000 7f22bd800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22b815b0e0 0x7f22b815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.708+0000 7f22bd800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b806c7a0 con 0x7f22b80ca520 2024-03-22T13:10:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.708+0000 7f22bd800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b80cd000 con 0x7f22b815b0e0 2024-03-22T13:10:19.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.708+0000 7f22bd800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b806ca80 con 0x7f22b80ccad0 2024-03-22T13:10:19.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.708+0000 7f22b7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22b815b0e0 0x7f22b815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:19.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.708+0000 7f22b6000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f22b80ccad0 0x7f22b806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:19.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.709+0000 7f22b6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22b80ca520 0x7f22b80cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35114/0 (socket says 172.21.15.6:35114) 2024-03-22T13:10:19.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.709+0000 7f22b6a00640 1 -- 172.21.15.6:0/2326711383 learned_addr learned my addr 172.21.15.6:0/2326711383 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.711+0000 7f22b6000640 1 -- 172.21.15.6:0/2326711383 >> v2:172.21.15.6:3300/0 conn(0x7f22b815b0e0 msgr2=0x7f22b815b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.711+0000 7f22b6000640 1 --2- 172.21.15.6:0/2326711383 >> v2:172.21.15.6:3300/0 conn(0x7f22b815b0e0 0x7f22b815b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.711+0000 7f22b6000640 1 -- 172.21.15.6:0/2326711383 >> v1:172.21.15.6:6789/0 conn(0x7f22b80ca520 legacy=0x7f22b80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:19.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.711+0000 7f22b6000640 1 -- 172.21.15.6:0/2326711383 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b806cd60 con 0x7f22b80ccad0 2024-03-22T13:10:19.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.715+0000 7f22b6000640 1 --2- 172.21.15.6:0/2326711383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ccad0 0x7f22b806d840 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f22b80b2490 tx=0x7f22a002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c4e1db9c7cc3b75 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:19.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.720+0000 7f22b5600640 1 -- 172.21.15.6:0/2326711383 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0035020 con 0x7f22b80ccad0 2024-03-22T13:10:19.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.722+0000 7f22b5600640 1 -- 172.21.15.6:0/2326711383 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a0031830 con 0x7f22b80ccad0 2024-03-22T13:10:19.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.722+0000 7f22b5600640 1 -- 172.21.15.6:0/2326711383 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0031c00 con 0x7f22b80ccad0 2024-03-22T13:10:19.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.722+0000 7f22bd800640 1 -- 172.21.15.6:0/2326711383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ccad0 msgr2=0x7f22b806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:19.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.723+0000 7f22bd800640 1 --2- 172.21.15.6:0/2326711383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ccad0 0x7f22b806d840 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f22b80b2490 tx=0x7f22a002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:10:19.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.723+0000 7f22bd800640 1 -- 172.21.15.6:0/2326711383 shutdown_connections 2024-03-22T13:10:19.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.723+0000 7f22bd800640 1 --2- 172.21.15.6:0/2326711383 >> v2:172.21.15.6:3300/0 conn(0x7f22b815b0e0 0x7f22b815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:19.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.723+0000 7f22bd800640 1 --2- 172.21.15.6:0/2326711383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ccad0 0x7f22b806d840 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:19.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.723+0000 7f22bd800640 1 -- 172.21.15.6:0/2326711383 >> 172.21.15.6:0/2326711383 conn(0x7f22b80b16b0 msgr2=0x7f22b80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:19.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.723+0000 7f22bd800640 1 -- 172.21.15.6:0/2326711383 shutdown_connections 2024-03-22T13:10:19.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.723+0000 7f22bd800640 1 -- 172.21.15.6:0/2326711383 wait complete. 2024-03-22T13:10:19.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.725+0000 7f22bd800640 1 Processor -- start 2024-03-22T13:10:19.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.725+0000 7f22bd800640 1 -- start start 2024-03-22T13:10:19.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.726+0000 7f22bd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ca520 0x7f22b80c4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:19.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.726+0000 7f22bd800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22b80ccad0 0x7f22b8153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:19.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.726+0000 7f22bd800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22b815b0e0 0x7f22b8158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:19.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.726+0000 7f22bd800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8154c20 con 0x7f22b80ccad0 2024-03-22T13:10:19.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.726+0000 7f22bd800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b806d320 con 0x7f22b80ca520 2024-03-22T13:10:19.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.726+0000 7f22bd800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b80ca900 con 0x7f22b815b0e0 2024-03-22T13:10:19.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.727+0000 7f22b6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22b80ccad0 0x7f22b8153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:19.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.727+0000 7f22b7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22b815b0e0 0x7f22b8158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:19.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.727+0000 7f22b6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22b80ccad0 0x7f22b8153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49318/0 (socket says 172.21.15.6:49318) 2024-03-22T13:10:19.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.727+0000 7f22b6000640 1 -- 172.21.15.6:0/4165582244 learned_addr learned my addr 172.21.15.6:0/4165582244 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:19.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.728+0000 7f22b6a00640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ca520 0x7f22b80c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:19.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.731+0000 7f22b7400640 1 -- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ca520 msgr2=0x7f22b80c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:19.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.731+0000 7f22b7400640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ca520 0x7f22b80c4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:19.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.731+0000 7f22b7400640 1 -- 172.21.15.6:0/4165582244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22b80ccad0 msgr2=0x7f22b8153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:19.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.731+0000 7f22b7400640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22b80ccad0 0x7f22b8153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:19.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.731+0000 7f22b7400640 1 -- 172.21.15.6:0/4165582244 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b80c5b60 con 0x7f22b815b0e0 2024-03-22T13:10:19.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.733+0000 7f22b7400640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22b815b0e0 0x7f22b8158a90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f22b000e820 tx=0x7f22b000ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:19.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.737+0000 7f22aea00640 1 -- 172.21.15.6:0/4165582244 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b0017070 con 0x7f22b815b0e0 2024-03-22T13:10:19.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.737+0000 7f22aea00640 1 -- 172.21.15.6:0/4165582244 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b000c730 con 0x7f22b815b0e0 2024-03-22T13:10:19.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.737+0000 7f22aea00640 1 -- 172.21.15.6:0/4165582244 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b000cb40 con 0x7f22b815b0e0 2024-03-22T13:10:19.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.737+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b81e67f0 con 0x7f22b815b0e0 2024-03-22T13:10:19.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.737+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b815a1a0 con 0x7f22b815b0e0 2024-03-22T13:10:19.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.738+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b80756c0 con 0x7f22b815b0e0 2024-03-22T13:10:19.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.759+0000 7f22aea00640 1 -- 172.21.15.6:0/4165582244 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f22b001a070 con 0x7f22b815b0e0 2024-03-22T13:10:19.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.760+0000 7f22aea00640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f229406bc50 0x7f229406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:19.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.760+0000 7f22b6a00640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f229406bc50 0x7f229406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:19.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:19.761+0000 7f22b6a00640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f229406bc50 0x7f229406e110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f22a80012e0 tx=0x7f22a8001a30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:20.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.000+0000 7f22aea00640 1 -- 172.21.15.6:0/4165582244 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f22b0056c70 con 0x7f22b815b0e0 2024-03-22T13:10:20.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.011+0000 7f22aea00640 1 -- 172.21.15.6:0/4165582244 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f22b0058bd0 con 0x7f22b815b0e0 2024-03-22T13:10:20.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.268+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22b80cb080 con 0x7f22b815b0e0 2024-03-22T13:10:20.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.290+0000 7f22aea00640 1 -- 172.21.15.6:0/4165582244 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f22b005cc70 con 0x7f22b815b0e0 2024-03-22T13:10:20.290 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:20.290 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:20.290 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.293+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f229406bc50 msgr2=0x7f229406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.293+0000 7f22bd800640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f229406bc50 0x7f229406e110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f22a80012e0 tx=0x7f22a8001a30 comp rx=0 tx=0).stop 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.293+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22b815b0e0 msgr2=0x7f22b8158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.293+0000 7f22bd800640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22b815b0e0 0x7f22b8158a90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f22b000e820 tx=0x7f22b000ece0 comp rx=0 tx=0).stop 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 shutdown_connections 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22b815b0e0 0x7f22b8158a90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22b80ccad0 0x7f22b8153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f229406bc50 0x7f229406e110 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 --2- 172.21.15.6:0/4165582244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22b80ca520 0x7f22b80c4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 >> 172.21.15.6:0/4165582244 conn(0x7f22b80b16b0 msgr2=0x7f22b815d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:20.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 shutdown_connections 2024-03-22T13:10:20.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:20.294+0000 7f22bd800640 1 -- 172.21.15.6:0/4165582244 wait complete. 2024-03-22T13:10:20.915 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:20.916 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:21.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:22.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.109+0000 7f5dfc400640 1 Processor -- start 2024-03-22T13:10:22.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.109+0000 7f5dfc400640 1 -- start start 2024-03-22T13:10:22.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.111+0000 7f5dfc400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df40c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:22.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.111+0000 7f5dfc400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5df40c2e50 0x7f5df40c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:22.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.111+0000 7f5dfc400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5df415b850 0x7f5df415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:22.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.111+0000 7f5dfc400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df4158ee0 con 0x7f5df40c2e50 2024-03-22T13:10:22.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.111+0000 7f5dfc400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df415c1e0 con 0x7f5df415b850 2024-03-22T13:10:22.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.111+0000 7f5dfc400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df4158010 con 0x7f5df40c9630 2024-03-22T13:10:22.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.112+0000 7f5df9a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df40c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:22.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.112+0000 7f5dfa400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5df415b850 0x7f5df415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:22.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.112+0000 7f5df9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5df40c2e50 0x7f5df40c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:22.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.112+0000 7f5df9a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df40c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46702/0 (socket says 172.21.15.6:46702) 2024-03-22T13:10:22.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.112+0000 7f5df9a00640 1 -- 172.21.15.6:0/4286819726 learned_addr learned my addr 172.21.15.6:0/4286819726 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.114+0000 7f5df9a00640 1 -- 172.21.15.6:0/4286819726 >> v2:172.21.15.92:3300/0 conn(0x7f5df415b850 msgr2=0x7f5df415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.114+0000 7f5df9a00640 1 --2- 172.21.15.6:0/4286819726 >> v2:172.21.15.92:3300/0 conn(0x7f5df415b850 0x7f5df415bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.114+0000 7f5df9a00640 1 -- 172.21.15.6:0/4286819726 >> v2:172.21.15.6:3300/0 conn(0x7f5df40c2e50 msgr2=0x7f5df40c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.114+0000 7f5df9a00640 1 --2- 172.21.15.6:0/4286819726 >> v2:172.21.15.6:3300/0 conn(0x7f5df40c2e50 0x7f5df40c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.114+0000 7f5df9a00640 1 -- 172.21.15.6:0/4286819726 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df41582d0 con 0x7f5df40c9630 2024-03-22T13:10:22.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.117+0000 7f5df9a00640 1 --2- 172.21.15.6:0/4286819726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df40c9a10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5df40cbad0 tx=0x7f5de802b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7facb7ef4a0e84f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:22.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.121+0000 7f5df3e00640 1 -- 172.21.15.6:0/4286819726 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8034070 con 0x7f5df40c9630 2024-03-22T13:10:22.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.121+0000 7f5df3e00640 1 -- 172.21.15.6:0/4286819726 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de8030850 con 0x7f5df40c9630 2024-03-22T13:10:22.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.121+0000 7f5df3e00640 1 -- 172.21.15.6:0/4286819726 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8030c20 con 0x7f5df40c9630 2024-03-22T13:10:22.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 -- 172.21.15.6:0/4286819726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 msgr2=0x7f5df40c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:22.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 --2- 172.21.15.6:0/4286819726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df40c9a10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5df40cbad0 tx=0x7f5de802b990 comp rx=0 tx=0).stop 2024-03-22T13:10:22.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 -- 172.21.15.6:0/4286819726 shutdown_connections 2024-03-22T13:10:22.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 --2- 172.21.15.6:0/4286819726 >> v2:172.21.15.92:3300/0 conn(0x7f5df415b850 0x7f5df415bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 --2- 172.21.15.6:0/4286819726 >> v2:172.21.15.6:3300/0 conn(0x7f5df40c2e50 0x7f5df40c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 --2- 172.21.15.6:0/4286819726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df40c9a10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 -- 172.21.15.6:0/4286819726 >> 172.21.15.6:0/4286819726 conn(0x7f5df40714f0 msgr2=0x7f5df4071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.122+0000 7f5dfc400640 1 -- 172.21.15.6:0/4286819726 shutdown_connections 2024-03-22T13:10:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.123+0000 7f5dfc400640 1 -- 172.21.15.6:0/4286819726 wait complete. 2024-03-22T13:10:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.124+0000 7f5dfc400640 1 Processor -- start 2024-03-22T13:10:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.125+0000 7f5dfc400640 1 -- start start 2024-03-22T13:10:22.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.125+0000 7f5dfc400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5df40c2e50 0x7f5df40cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:22.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5dfc400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:22.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5dfc400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df415b850 0x7f5df40aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5dfc400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df4158b70 con 0x7f5df415b850 2024-03-22T13:10:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5dfc400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df4158890 con 0x7f5df40c2e50 2024-03-22T13:10:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5dfc400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df40ca330 con 0x7f5df40c9630 2024-03-22T13:10:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5df9000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5df9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5df40c2e50 0x7f5df40cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:22.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.126+0000 7f5dfa400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df415b850 0x7f5df40aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:22.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.127+0000 7f5df9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5df40c2e50 0x7f5df40cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:41612/0 (socket says 172.21.15.6:41612) 2024-03-22T13:10:22.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.127+0000 7f5df9a00640 1 -- 172.21.15.6:0/1980354073 learned_addr learned my addr 172.21.15.6:0/1980354073 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.130+0000 7f5df9a00640 1 -- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 msgr2=0x7f5df406db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.130+0000 7f5df9a00640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df406db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.130+0000 7f5df9a00640 1 -- 172.21.15.6:0/1980354073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df415b850 msgr2=0x7f5df40aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.130+0000 7f5df9a00640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df415b850 0x7f5df40aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.130+0000 7f5df9a00640 1 -- 172.21.15.6:0/1980354073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df40af480 con 0x7f5df40c2e50 2024-03-22T13:10:22.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.133+0000 7f5df9a00640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5df40c2e50 0x7f5df40cc650 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5de802b9d0 tx=0x7f5de8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.136+0000 7f5df1600640 1 -- 172.21.15.6:0/1980354073 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8048070 con 0x7f5df40c2e50 2024-03-22T13:10:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.136+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df4156220 con 0x7f5df40c2e50 2024-03-22T13:10:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.136+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df4156780 con 0x7f5df40c2e50 2024-03-22T13:10:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.137+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df40c9d90 con 0x7f5df40c2e50 2024-03-22T13:10:22.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.345+0000 7f5df1600640 1 -- 172.21.15.6:0/1980354073 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de8030720 con 0x7f5df40c2e50 2024-03-22T13:10:22.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.418+0000 7f5df1600640 1 -- 172.21.15.6:0/1980354073 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8052060 con 0x7f5df40c2e50 2024-03-22T13:10:22.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.420+0000 7f5df1600640 1 -- 172.21.15.6:0/1980354073 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5de804b070 con 0x7f5df40c2e50 2024-03-22T13:10:22.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.422+0000 7f5df1600640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5dd406bbd0 0x7f5dd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:22.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.422+0000 7f5df9000640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5dd406bbd0 0x7f5dd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.423+0000 7f5df9000640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5dd406bbd0 0x7f5dd406e090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5df415b180 tx=0x7f5de0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:22.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.439+0000 7f5df1600640 1 -- 172.21.15.6:0/1980354073 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5de80792d0 con 0x7f5df40c2e50 2024-03-22T13:10:22.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.458+0000 7f5df1600640 1 -- 172.21.15.6:0/1980354073 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5df40c9d90 con 0x7f5df40c2e50 2024-03-22T13:10:22.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.711+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5df40cea50 con 0x7f5df40c2e50 2024-03-22T13:10:22.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.727+0000 7f5df1600640 1 -- 172.21.15.6:0/1980354073 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f5df40af480 con 0x7f5df40c2e50 2024-03-22T13:10:22.727 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:22.727 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:22.727 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:22.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5dd406bbd0 msgr2=0x7f5dd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:22.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5dd406bbd0 0x7f5dd406e090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5df415b180 tx=0x7f5de0009190 comp rx=0 tx=0).stop 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5df40c2e50 msgr2=0x7f5df40cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5df40c2e50 0x7f5df40cc650 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5de802b9d0 tx=0x7f5de8004730 comp rx=0 tx=0).stop 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 shutdown_connections 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df415b850 0x7f5df40aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5df40c9630 0x7f5df406db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5dd406bbd0 0x7f5dd406e090 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 --2- 172.21.15.6:0/1980354073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5df40c2e50 0x7f5df40cc650 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.730+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 >> 172.21.15.6:0/1980354073 conn(0x7f5df40714f0 msgr2=0x7f5df41572a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.731+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 shutdown_connections 2024-03-22T13:10:22.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:22.731+0000 7f5dfc400640 1 -- 172.21.15.6:0/1980354073 wait complete. 2024-03-22T13:10:23.352 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:23.352 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:24.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.570+0000 7f97c8e00640 1 Processor -- start 2024-03-22T13:10:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.571+0000 7f97c8e00640 1 -- start start 2024-03-22T13:10:24.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.573+0000 7f97c8e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97c406ede0 0x7f97c415abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.573+0000 7f97c8e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c40c91e0 0x7f97c40c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.573+0000 7f97c8e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c4070890 con 0x7f97c4154540 2024-03-22T13:10:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.573+0000 7f97c8e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c40c79e0 con 0x7f97c406ede0 2024-03-22T13:10:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.573+0000 7f97c8e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40c9bb0 con 0x7f97c40c91e0 2024-03-22T13:10:24.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.574+0000 7f97c3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c40c91e0 0x7f97c40c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:24.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.574+0000 7f97c3e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c40c91e0 0x7f97c40c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:32784/0 (socket says 172.21.15.6:32784) 2024-03-22T13:10:24.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.574+0000 7f97c3e00640 1 -- 172.21.15.6:0/2684088877 learned_addr learned my addr 172.21.15.6:0/2684088877 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.577+0000 7f97c3e00640 1 -- 172.21.15.6:0/2684088877 >> v2:172.21.15.6:3300/0 conn(0x7f97c406ede0 msgr2=0x7f97c415abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.577+0000 7f97c3e00640 1 --2- 172.21.15.6:0/2684088877 >> v2:172.21.15.6:3300/0 conn(0x7f97c406ede0 0x7f97c415abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:24.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.577+0000 7f97c3e00640 1 -- 172.21.15.6:0/2684088877 >> v1:172.21.15.6:6789/0 conn(0x7f97c4154540 legacy=0x7f97c415e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:24.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.577+0000 7f97c3e00640 1 -- 172.21.15.6:0/2684088877 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c4070b70 con 0x7f97c40c91e0 2024-03-22T13:10:24.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.579+0000 7f97c3e00640 1 --2- 172.21.15.6:0/2684088877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c40c91e0 0x7f97c40c9670 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009e50 tx=0x7f97bc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865b443c8a4f0459 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:24.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.581+0000 7f97c2000640 1 -- 172.21.15.6:0/2684088877 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc02cb00 con 0x7f97c40c91e0 2024-03-22T13:10:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.584+0000 7f97c2000640 1 -- 172.21.15.6:0/2684088877 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc02cca0 con 0x7f97c40c91e0 2024-03-22T13:10:24.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.584+0000 7f97c2000640 1 -- 172.21.15.6:0/2684088877 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc034610 con 0x7f97c40c91e0 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.584+0000 7f97c8e00640 1 -- 172.21.15.6:0/2684088877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c40c91e0 msgr2=0x7f97c40c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.584+0000 7f97c8e00640 1 --2- 172.21.15.6:0/2684088877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c40c91e0 0x7f97c40c9670 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009e50 tx=0x7f97bc02bf60 comp rx=0 tx=0).stop 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.585+0000 7f97c8e00640 1 -- 172.21.15.6:0/2684088877 shutdown_connections 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.585+0000 7f97c8e00640 1 --2- 172.21.15.6:0/2684088877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c40c91e0 0x7f97c40c9670 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.585+0000 7f97c8e00640 1 --2- 172.21.15.6:0/2684088877 >> v2:172.21.15.6:3300/0 conn(0x7f97c406ede0 0x7f97c415abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.585+0000 7f97c8e00640 1 -- 172.21.15.6:0/2684088877 >> 172.21.15.6:0/2684088877 conn(0x7f97c406c190 msgr2=0x7f97c406c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.585+0000 7f97c8e00640 1 -- 172.21.15.6:0/2684088877 shutdown_connections 2024-03-22T13:10:24.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.585+0000 7f97c8e00640 1 -- 172.21.15.6:0/2684088877 wait complete. 2024-03-22T13:10:24.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.587+0000 7f97c8e00640 1 Processor -- start 2024-03-22T13:10:24.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.587+0000 7f97c8e00640 1 -- start start 2024-03-22T13:10:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.588+0000 7f97c8e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c406ede0 0x7f97c414e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.588+0000 7f97c8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c40c91e0 0x7f97c41591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.588+0000 7f97c8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97c4154540 0x7f97c41f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.588+0000 7f97c8e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4154920 con 0x7f97c4154540 2024-03-22T13:10:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.588+0000 7f97c8e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c415a730 con 0x7f97c40c91e0 2024-03-22T13:10:24.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.588+0000 7f97c8e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4070e50 con 0x7f97c406ede0 2024-03-22T13:10:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.589+0000 7f97c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97c4154540 0x7f97c41f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.589+0000 7f97c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97c4154540 0x7f97c41f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60206/0 (socket says 172.21.15.6:60206) 2024-03-22T13:10:24.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.589+0000 7f97c3e00640 1 -- 172.21.15.6:0/369491693 learned_addr learned my addr 172.21.15.6:0/369491693 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:24.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.593+0000 7f97c3e00640 1 -- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c406ede0 msgr2=0x7f97c414e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:24.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.593+0000 7f97c3e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c406ede0 0x7f97c414e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:24.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.593+0000 7f97c3e00640 1 -- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c40c91e0 msgr2=0x7f97c41591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:24.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.593+0000 7f97c3e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c40c91e0 0x7f97c41591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:24.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.593+0000 7f97c3e00640 1 -- 172.21.15.6:0/369491693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c41f0c50 con 0x7f97c4154540 2024-03-22T13:10:24.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.596+0000 7f97c3e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97c4154540 0x7f97c41f3270 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009e50 tx=0x7f97bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:24.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.601+0000 7f97bb400640 1 -- 172.21.15.6:0/369491693 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc043020 con 0x7f97c4154540 2024-03-22T13:10:24.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.602+0000 7f97bb400640 1 -- 172.21.15.6:0/369491693 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc02ce50 con 0x7f97c4154540 2024-03-22T13:10:24.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.602+0000 7f97bb400640 1 -- 172.21.15.6:0/369491693 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc03e6b0 con 0x7f97c4154540 2024-03-22T13:10:24.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.602+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c4150300 con 0x7f97c4154540 2024-03-22T13:10:24.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.602+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c414e1a0 con 0x7f97c4154540 2024-03-22T13:10:24.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.603+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c4075650 con 0x7f97c4154540 2024-03-22T13:10:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.925+0000 7f97bb400640 1 -- 172.21.15.6:0/369491693 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f97bc046020 con 0x7f97c4154540 2024-03-22T13:10:24.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.929+0000 7f97bb400640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97a406bc50 0x7f97a406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:24.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.929+0000 7f97c3400640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97a406bc50 0x7f97a406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:24.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.930+0000 7f97c3400640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97a406bc50 0x7f97a406e110 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009650 tx=0x7f97b4006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:24.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.965+0000 7f97bb400640 1 -- 172.21.15.6:0/369491693 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f97bc079b90 con 0x7f97c4154540 2024-03-22T13:10:24.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:24.970+0000 7f97bb400640 1 -- 172.21.15.6:0/369491693 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f97bc04d030 con 0x7f97c4154540 2024-03-22T13:10:25.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.232+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97c406e4b0 con 0x7f97c4154540 2024-03-22T13:10:25.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.261+0000 7f97bb400640 1 -- 172.21.15.6:0/369491693 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f97bc07c1f0 con 0x7f97c4154540 2024-03-22T13:10:25.261 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:25.261 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:25.261 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.264+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97a406bc50 msgr2=0x7f97a406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.264+0000 7f97c8e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97a406bc50 0x7f97a406e110 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009650 tx=0x7f97b4006ba0 comp rx=0 tx=0).stop 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.264+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97c4154540 msgr2=0x7f97c41f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.264+0000 7f97c8e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97c4154540 0x7f97c41f3270 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009e50 tx=0x7f97bc004a40 comp rx=0 tx=0).stop 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 shutdown_connections 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97c4154540 0x7f97c41f3270 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97a406bc50 0x7f97a406e110 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c40c91e0 0x7f97c41591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 --2- 172.21.15.6:0/369491693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97c406ede0 0x7f97c414e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 >> 172.21.15.6:0/369491693 conn(0x7f97c406c190 msgr2=0x7f97c406fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 shutdown_connections 2024-03-22T13:10:25.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:25.265+0000 7f97c8e00640 1 -- 172.21.15.6:0/369491693 wait complete. 2024-03-22T13:10:25.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:25.888 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:26.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.084+0000 7fdba8600640 1 Processor -- start 2024-03-22T13:10:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.084+0000 7fdba8600640 1 -- start start 2024-03-22T13:10:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.086+0000 7fdba8600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba0154980 0x7fdba00ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.086+0000 7fdba8600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdba00cd300 0x7fdba00ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.086+0000 7fdba8600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdba015a9b0 0x7fdba015ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.086+0000 7fdba8600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba015a520 con 0x7fdba00cd300 2024-03-22T13:10:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.086+0000 7fdba8600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba015b3a0 con 0x7fdba015a9b0 2024-03-22T13:10:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.086+0000 7fdba8600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00cc380 con 0x7fdba0154980 2024-03-22T13:10:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.087+0000 7fdba5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba0154980 0x7fdba00ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.087+0000 7fdba5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdba00cd300 0x7fdba00ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.087+0000 7fdba5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdba00cd300 0x7fdba00ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60210/0 (socket says 172.21.15.6:60210) 2024-03-22T13:10:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.087+0000 7fdba5200640 1 -- 172.21.15.6:0/852417319 learned_addr learned my addr 172.21.15.6:0/852417319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.087+0000 7fdba5c00640 1 --2- 172.21.15.6:0/852417319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba0154980 0x7fdba00ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:32804/0 (socket says 172.21.15.6:32804) 2024-03-22T13:10:27.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.090+0000 7fdba5200640 1 -- 172.21.15.6:0/852417319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba0154980 msgr2=0x7fdba00ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:27.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.090+0000 7fdba5200640 1 --2- 172.21.15.6:0/852417319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba0154980 0x7fdba00ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.090+0000 7fdba5200640 1 -- 172.21.15.6:0/852417319 >> v2:172.21.15.92:3300/0 conn(0x7fdba015a9b0 msgr2=0x7fdba015ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:27.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.090+0000 7fdba5200640 1 --2- 172.21.15.6:0/852417319 >> v2:172.21.15.92:3300/0 conn(0x7fdba015a9b0 0x7fdba015ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.090+0000 7fdba5200640 1 -- 172.21.15.6:0/852417319 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba00cc660 con 0x7fdba00cd300 2024-03-22T13:10:27.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.109+0000 7fdba5200640 1 --2- 172.21.15.6:0/852417319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba00cd300 0x7fdba00ce840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdba015ea30 tx=0x7fdb8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85463361f2985494 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:27.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.118+0000 7fdb9fe00640 1 -- 172.21.15.6:0/852417319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c035020 con 0x7fdba00cd300 2024-03-22T13:10:27.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.118+0000 7fdb9fe00640 1 -- 172.21.15.6:0/852417319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c031870 con 0x7fdba00cd300 2024-03-22T13:10:27.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.118+0000 7fdb9fe00640 1 -- 172.21.15.6:0/852417319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c031c40 con 0x7fdba00cd300 2024-03-22T13:10:27.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.119+0000 7fdba8600640 1 -- 172.21.15.6:0/852417319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba00cd300 msgr2=0x7fdba00ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:27.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.119+0000 7fdba8600640 1 --2- 172.21.15.6:0/852417319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba00cd300 0x7fdba00ce840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdba015ea30 tx=0x7fdb8c02b910 comp rx=0 tx=0).stop 2024-03-22T13:10:27.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.120+0000 7fdba8600640 1 -- 172.21.15.6:0/852417319 shutdown_connections 2024-03-22T13:10:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.120+0000 7fdba8600640 1 --2- 172.21.15.6:0/852417319 >> v2:172.21.15.92:3300/0 conn(0x7fdba015a9b0 0x7fdba015ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.120+0000 7fdba8600640 1 --2- 172.21.15.6:0/852417319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba00cd300 0x7fdba00ce840 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.120+0000 7fdba8600640 1 --2- 172.21.15.6:0/852417319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba0154980 0x7fdba00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.120+0000 7fdba8600640 1 -- 172.21.15.6:0/852417319 >> 172.21.15.6:0/852417319 conn(0x7fdba00b16c0 msgr2=0x7fdba00b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.120+0000 7fdba8600640 1 -- 172.21.15.6:0/852417319 shutdown_connections 2024-03-22T13:10:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.120+0000 7fdba8600640 1 -- 172.21.15.6:0/852417319 wait complete. 2024-03-22T13:10:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.122+0000 7fdba8600640 1 Processor -- start 2024-03-22T13:10:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.123+0000 7fdba8600640 1 -- start start 2024-03-22T13:10:27.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.124+0000 7fdba8600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00cd300 0x7fdba01efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:27.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.124+0000 7fdba8600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba0154980 0x7fdba015dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:27.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.124+0000 7fdba8600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba015a9b0 0x7fdba014d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:27.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.124+0000 7fdba8600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00ccf00 con 0x7fdba015a9b0 2024-03-22T13:10:27.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.124+0000 7fdba8600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00ccc20 con 0x7fdba0154980 2024-03-22T13:10:27.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.124+0000 7fdba8600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba00ceea0 con 0x7fdba00cd300 2024-03-22T13:10:27.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.125+0000 7fdba5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba0154980 0x7fdba015dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:27.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.125+0000 7fdba5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00cd300 0x7fdba01efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:27.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.125+0000 7fdba6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba015a9b0 0x7fdba014d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:27.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.125+0000 7fdba5c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00cd300 0x7fdba01efde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:32812/0 (socket says 172.21.15.6:32812) 2024-03-22T13:10:27.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.125+0000 7fdba5c00640 1 -- 172.21.15.6:0/3604302546 learned_addr learned my addr 172.21.15.6:0/3604302546 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.129+0000 7fdba6600640 1 -- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00cd300 msgr2=0x7fdba01efde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.129+0000 7fdba6600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00cd300 0x7fdba01efde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.129+0000 7fdba6600640 1 -- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba0154980 msgr2=0x7fdba015dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.129+0000 7fdba6600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba0154980 0x7fdba015dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.129+0000 7fdba6600640 1 -- 172.21.15.6:0/3604302546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba015e780 con 0x7fdba015a9b0 2024-03-22T13:10:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.132+0000 7fdba6600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba015a9b0 0x7fdba014d950 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800be50 tx=0x7fdb9800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:27.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.137+0000 7fdb9d600640 1 -- 172.21.15.6:0/3604302546 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98019070 con 0x7fdba015a9b0 2024-03-22T13:10:27.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.137+0000 7fdb9d600640 1 -- 172.21.15.6:0/3604302546 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb98014440 con 0x7fdba015a9b0 2024-03-22T13:10:27.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.137+0000 7fdb9d600640 1 -- 172.21.15.6:0/3604302546 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98014830 con 0x7fdba015a9b0 2024-03-22T13:10:27.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.137+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba014ee50 con 0x7fdba015a9b0 2024-03-22T13:10:27.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.137+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba014f410 con 0x7fdba015a9b0 2024-03-22T13:10:27.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.138+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba0153ab0 con 0x7fdba015a9b0 2024-03-22T13:10:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.434+0000 7fdb9d600640 1 -- 172.21.15.6:0/3604302546 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdb9801c070 con 0x7fdba015a9b0 2024-03-22T13:10:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.435+0000 7fdb9d600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb8006bc20 0x7fdb8006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.436+0000 7fdba5c00640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb8006bc20 0x7fdb8006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.437+0000 7fdba5c00640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb8006bc20 0x7fdb8006e0e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009c00 tx=0x7fdb94009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:27.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.455+0000 7fdb9d600640 1 -- 172.21.15.6:0/3604302546 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdb98057300 con 0x7fdba015a9b0 2024-03-22T13:10:27.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.456+0000 7fdb9d600640 1 -- 172.21.15.6:0/3604302546 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdba0153ab0 con 0x7fdba015a9b0 2024-03-22T13:10:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.731+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba00756c0 con 0x7fdba015a9b0 2024-03-22T13:10:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.757+0000 7fdb9d600640 1 -- 172.21.15.6:0/3604302546 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fdb98023020 con 0x7fdba015a9b0 2024-03-22T13:10:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:27.758 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb8006bc20 msgr2=0x7fdb8006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb8006bc20 0x7fdb8006e0e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009c00 tx=0x7fdb94009190 comp rx=0 tx=0).stop 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba015a9b0 msgr2=0x7fdba014d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba015a9b0 0x7fdba014d950 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800be50 tx=0x7fdb9800bf40 comp rx=0 tx=0).stop 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 shutdown_connections 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba015a9b0 0x7fdba014d950 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb8006bc20 0x7fdb8006e0e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba0154980 0x7fdba015dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 --2- 172.21.15.6:0/3604302546 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00cd300 0x7fdba01efde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 >> 172.21.15.6:0/3604302546 conn(0x7fdba00b16c0 msgr2=0x7fdba00afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 shutdown_connections 2024-03-22T13:10:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:27.760+0000 7fdba8600640 1 -- 172.21.15.6:0/3604302546 wait complete. 2024-03-22T13:10:28.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:28.380 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:29.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:29.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.584+0000 7fda05400640 1 Processor -- start 2024-03-22T13:10:29.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.584+0000 7fda05400640 1 -- start start 2024-03-22T13:10:29.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.586+0000 7fda05400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda00154240 0x7fda0015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:29.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.586+0000 7fda05400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda000c4740 con 0x7fda000c68c0 2024-03-22T13:10:29.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.586+0000 7fda05400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda00154620 con 0x7fda000c7ee0 2024-03-22T13:10:29.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.586+0000 7fda05400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0015e130 con 0x7fda00154240 2024-03-22T13:10:29.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.587+0000 7fd9fe800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda00154240 0x7fda0015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.587+0000 7fd9ff200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fda000c68c0 0x7fda0006b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57442/0 (socket says 172.21.15.6:57442) 2024-03-22T13:10:29.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.587+0000 7fd9ff200640 1 -- 172.21.15.6:0/1742978730 learned_addr learned my addr 172.21.15.6:0/1742978730 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.590+0000 7fd9fe800640 1 -- 172.21.15.6:0/1742978730 >> v1:172.21.15.92:6789/0 conn(0x7fda000c7ee0 legacy=0x7fda000c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.590+0000 7fd9fe800640 1 -- 172.21.15.6:0/1742978730 >> v1:172.21.15.6:6789/0 conn(0x7fda000c68c0 legacy=0x7fda0006b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.590+0000 7fd9fe800640 1 -- 172.21.15.6:0/1742978730 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda000c6e00 con 0x7fda00154240 2024-03-22T13:10:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.590+0000 7fd9fde00640 1 -- 172.21.15.6:0/1742978730 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406305751 0 0) 0x7fda00154620 con 0x7fda000c7ee0 2024-03-22T13:10:29.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.592+0000 7fd9fe800640 1 --2- 172.21.15.6:0/1742978730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda00154240 0x7fda0015ac90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd9f00097d0 tx=0x7fd9f002b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb7a0729a30af55 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:29.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.596+0000 7fd9fde00640 1 -- 172.21.15.6:0/1742978730 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0034070 con 0x7fda00154240 2024-03-22T13:10:29.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.597+0000 7fd9fde00640 1 -- 172.21.15.6:0/1742978730 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f0030890 con 0x7fda00154240 2024-03-22T13:10:29.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.597+0000 7fd9fde00640 1 -- 172.21.15.6:0/1742978730 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0030c60 con 0x7fda00154240 2024-03-22T13:10:29.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.597+0000 7fda05400640 1 -- 172.21.15.6:0/1742978730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda00154240 msgr2=0x7fda0015ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:29.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.597+0000 7fda05400640 1 --2- 172.21.15.6:0/1742978730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda00154240 0x7fda0015ac90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd9f00097d0 tx=0x7fd9f002b9d0 comp rx=0 tx=0).stop 2024-03-22T13:10:29.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.597+0000 7fda05400640 1 -- 172.21.15.6:0/1742978730 shutdown_connections 2024-03-22T13:10:29.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.597+0000 7fda05400640 1 --2- 172.21.15.6:0/1742978730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda00154240 0x7fda0015ac90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:29.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.597+0000 7fda05400640 1 -- 172.21.15.6:0/1742978730 >> 172.21.15.6:0/1742978730 conn(0x7fda000cb930 msgr2=0x7fda000cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:29.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.598+0000 7fda05400640 1 -- 172.21.15.6:0/1742978730 shutdown_connections 2024-03-22T13:10:29.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.598+0000 7fda05400640 1 -- 172.21.15.6:0/1742978730 wait complete. 2024-03-22T13:10:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.599+0000 7fda05400640 1 Processor -- start 2024-03-22T13:10:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.599+0000 7fda05400640 1 -- start start 2024-03-22T13:10:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.600+0000 7fda05400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda000c68c0 0x7fda00151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.600+0000 7fda05400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda000c7ee0 0x7fda0014b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.600+0000 7fda05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda00154240 0x7fda0015d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:29.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.600+0000 7fda05400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0015dae0 con 0x7fda00154240 2024-03-22T13:10:29.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.600+0000 7fda05400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0015d800 con 0x7fda000c7ee0 2024-03-22T13:10:29.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.600+0000 7fda05400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0015ddc0 con 0x7fda000c68c0 2024-03-22T13:10:29.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.601+0000 7fd9fe800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda000c68c0 0x7fda00151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:29.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.601+0000 7fd9f7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda000c7ee0 0x7fda0014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:29.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.601+0000 7fd9ff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda00154240 0x7fda0015d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:29.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.601+0000 7fd9ff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda00154240 0x7fda0015d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60218/0 (socket says 172.21.15.6:60218) 2024-03-22T13:10:29.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.601+0000 7fd9fe800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda000c68c0 0x7fda00151210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:32844/0 (socket says 172.21.15.6:32844) 2024-03-22T13:10:29.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.601+0000 7fd9ff200640 1 -- 172.21.15.6:0/3278477373 learned_addr learned my addr 172.21.15.6:0/3278477373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.605+0000 7fd9f7e00640 1 -- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda000c68c0 msgr2=0x7fda00151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.605+0000 7fd9f7e00640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda000c68c0 0x7fda00151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.605+0000 7fd9f7e00640 1 -- 172.21.15.6:0/3278477373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda00154240 msgr2=0x7fda0015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.605+0000 7fd9f7e00640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda00154240 0x7fda0015d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:29.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.605+0000 7fd9f7e00640 1 -- 172.21.15.6:0/3278477373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0015c250 con 0x7fda000c7ee0 2024-03-22T13:10:29.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.607+0000 7fd9f7e00640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda000c7ee0 0x7fda0014b510 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800c430 tx=0x7fd9e800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:29.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.611+0000 7fd9f6a00640 1 -- 172.21.15.6:0/3278477373 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8017070 con 0x7fda000c7ee0 2024-03-22T13:10:29.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.611+0000 7fd9f6a00640 1 -- 172.21.15.6:0/3278477373 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e80035f0 con 0x7fda000c7ee0 2024-03-22T13:10:29.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.611+0000 7fd9f6a00640 1 -- 172.21.15.6:0/3278477373 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e80039e0 con 0x7fda000c7ee0 2024-03-22T13:10:29.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.611+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0015c530 con 0x7fda000c7ee0 2024-03-22T13:10:29.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.611+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda00150a10 con 0x7fda000c7ee0 2024-03-22T13:10:29.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.612+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda000c6050 con 0x7fda000c7ee0 2024-03-22T13:10:29.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.643+0000 7fd9f6a00640 1 -- 172.21.15.6:0/3278477373 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd9e801a070 con 0x7fda000c7ee0 2024-03-22T13:10:29.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.644+0000 7fd9f6a00640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd9dc06bc20 0x7fd9dc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:29.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.645+0000 7fd9fe800640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd9dc06bc20 0x7fd9dc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:29.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.646+0000 7fd9fe800640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd9dc06bc20 0x7fd9dc06e0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0036000 tx=0x7fd9f003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:29.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.649+0000 7fd9f6a00640 1 -- 172.21.15.6:0/3278477373 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd9e8056710 con 0x7fda000c7ee0 2024-03-22T13:10:29.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:29.907+0000 7fd9f6a00640 1 -- 172.21.15.6:0/3278477373 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fda000c6050 con 0x7fda000c7ee0 2024-03-22T13:10:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.189+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda00072f80 con 0x7fda000c7ee0 2024-03-22T13:10:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.210+0000 7fd9f6a00640 1 -- 172.21.15.6:0/3278477373 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fd9e8021020 con 0x7fda000c7ee0 2024-03-22T13:10:30.210 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:30.210 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd9dc06bc20 msgr2=0x7fd9dc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd9dc06bc20 0x7fd9dc06e0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0036000 tx=0x7fd9f003b040 comp rx=0 tx=0).stop 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda000c7ee0 msgr2=0x7fda0014b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda000c7ee0 0x7fda0014b510 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800c430 tx=0x7fd9e800c8f0 comp rx=0 tx=0).stop 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 shutdown_connections 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda00154240 0x7fda0015d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd9dc06bc20 0x7fd9dc06e0e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda000c7ee0 0x7fda0014b510 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 --2- 172.21.15.6:0/3278477373 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fda000c68c0 0x7fda00151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 >> 172.21.15.6:0/3278477373 conn(0x7fda000cb930 msgr2=0x7fda00152700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 shutdown_connections 2024-03-22T13:10:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:30.213+0000 7fda05400640 1 -- 172.21.15.6:0/3278477373 wait complete. 2024-03-22T13:10:30.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:30.838 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:31.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.041+0000 7f2beb200640 1 Processor -- start 2024-03-22T13:10:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.041+0000 7f2beb200640 1 -- start start 2024-03-22T13:10:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.043+0000 7f2beb200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2be406fea0 0x7f2be406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.043+0000 7f2beb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2be40c2ea0 0x7f2be40c3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.043+0000 7f2beb200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40c3d30 con 0x7f2be406b080 2024-03-22T13:10:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.043+0000 7f2beb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be40c3a50 con 0x7f2be40c2ea0 2024-03-22T13:10:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.043+0000 7f2beb200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be415a450 con 0x7f2be406fea0 2024-03-22T13:10:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.044+0000 7f2be3400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2be406fea0 0x7f2be406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.044+0000 7f2be9200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2be40c2ea0 0x7f2be40c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.044+0000 7f2be3400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2be406fea0 0x7f2be406b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:43822/0 (socket says 172.21.15.6:43822) 2024-03-22T13:10:32.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.044+0000 7f2be3400640 1 -- 172.21.15.6:0/2182571357 learned_addr learned my addr 172.21.15.6:0/2182571357 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:32.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.048+0000 7f2be3400640 1 -- 172.21.15.6:0/2182571357 >> v2:172.21.15.6:3300/0 conn(0x7f2be40c2ea0 msgr2=0x7f2be40c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:32.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.048+0000 7f2be3400640 1 --2- 172.21.15.6:0/2182571357 >> v2:172.21.15.6:3300/0 conn(0x7f2be40c2ea0 0x7f2be40c3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.048+0000 7f2be3400640 1 -- 172.21.15.6:0/2182571357 >> v1:172.21.15.92:6789/0 conn(0x7f2be406b080 legacy=0x7f2be415d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:32.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.048+0000 7f2be3400640 1 -- 172.21.15.6:0/2182571357 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be415a730 con 0x7f2be406fea0 2024-03-22T13:10:32.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.052+0000 7f2be3400640 1 --2- 172.21.15.6:0/2182571357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406fea0 0x7f2be406b460 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2be406fba0 tx=0x7f2bd002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42263c5eaff31944 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:32.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.058+0000 7f2be2a00640 1 -- 172.21.15.6:0/2182571357 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0034070 con 0x7f2be406fea0 2024-03-22T13:10:32.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.058+0000 7f2be2a00640 1 -- 172.21.15.6:0/2182571357 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd0030860 con 0x7f2be406fea0 2024-03-22T13:10:32.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.058+0000 7f2be2a00640 1 -- 172.21.15.6:0/2182571357 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0030c30 con 0x7f2be406fea0 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 -- 172.21.15.6:0/2182571357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406fea0 msgr2=0x7f2be406b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 --2- 172.21.15.6:0/2182571357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406fea0 0x7f2be406b460 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2be406fba0 tx=0x7f2bd002b9a0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 -- 172.21.15.6:0/2182571357 shutdown_connections 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 --2- 172.21.15.6:0/2182571357 >> v2:172.21.15.6:3300/0 conn(0x7f2be40c2ea0 0x7f2be40c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 --2- 172.21.15.6:0/2182571357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406fea0 0x7f2be406b460 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 -- 172.21.15.6:0/2182571357 >> 172.21.15.6:0/2182571357 conn(0x7f2be40ccb50 msgr2=0x7f2be40ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 -- 172.21.15.6:0/2182571357 shutdown_connections 2024-03-22T13:10:32.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.059+0000 7f2beb200640 1 -- 172.21.15.6:0/2182571357 wait complete. 2024-03-22T13:10:32.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.062+0000 7f2beb200640 1 Processor -- start 2024-03-22T13:10:32.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.062+0000 7f2beb200640 1 -- start start 2024-03-22T13:10:32.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.064+0000 7f2beb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406b080 0x7f2be414a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:32.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.064+0000 7f2beb200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2be406fea0 0x7f2be414dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:32.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.064+0000 7f2beb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be40c2ea0 0x7f2be415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:32.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.064+0000 7f2beb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be415e000 con 0x7f2be40c2ea0 2024-03-22T13:10:32.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.064+0000 7f2beb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be415acf0 con 0x7f2be406b080 2024-03-22T13:10:32.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.064+0000 7f2beb200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be415b210 con 0x7f2be406fea0 2024-03-22T13:10:32.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.064+0000 7f2be3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406b080 0x7f2be414a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:32.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.065+0000 7f2be9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be40c2ea0 0x7f2be415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:32.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.065+0000 7f2be3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2be406fea0 0x7f2be414dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:32.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.065+0000 7f2be3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406b080 0x7f2be414a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:43834/0 (socket says 172.21.15.6:43834) 2024-03-22T13:10:32.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.065+0000 7f2be3e00640 1 -- 172.21.15.6:0/3953863967 learned_addr learned my addr 172.21.15.6:0/3953863967 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:32.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.068+0000 7f2be3e00640 1 -- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2be406fea0 msgr2=0x7f2be414dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:32.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.068+0000 7f2be3e00640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2be406fea0 0x7f2be414dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.068+0000 7f2be3e00640 1 -- 172.21.15.6:0/3953863967 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be40c2ea0 msgr2=0x7f2be415b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:32.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.068+0000 7f2be3e00640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be40c2ea0 0x7f2be415b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.069+0000 7f2be3e00640 1 -- 172.21.15.6:0/3953863967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be415be70 con 0x7f2be406b080 2024-03-22T13:10:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.071+0000 7f2be3e00640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406b080 0x7f2be414a9d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400b420 tx=0x7f2bd400b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:32.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.075+0000 7f2bdbe00640 1 -- 172.21.15.6:0/3953863967 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4011830 con 0x7f2be406b080 2024-03-22T13:10:32.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.075+0000 7f2bdbe00640 1 -- 172.21.15.6:0/3953863967 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd4016070 con 0x7f2be406b080 2024-03-22T13:10:32.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.075+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be414c440 con 0x7f2be406b080 2024-03-22T13:10:32.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.075+0000 7f2bdbe00640 1 -- 172.21.15.6:0/3953863967 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4011c10 con 0x7f2be406b080 2024-03-22T13:10:32.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.075+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be414ca00 con 0x7f2be406b080 2024-03-22T13:10:32.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.077+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be40756c0 con 0x7f2be406b080 2024-03-22T13:10:32.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.095+0000 7f2bdbe00640 1 -- 172.21.15.6:0/3953863967 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2bd4004020 con 0x7f2be406b080 2024-03-22T13:10:32.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.097+0000 7f2bdbe00640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2bc406bc20 0x7f2bc406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:32.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.097+0000 7f2be3400640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2bc406bc20 0x7f2bc406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:32.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.098+0000 7f2be3400640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2bc406bc20 0x7f2bc406e0e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0036000 tx=0x7f2bd003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:32.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.100+0000 7f2bdbe00640 1 -- 172.21.15.6:0/3953863967 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f2bd4057330 con 0x7f2be406b080 2024-03-22T13:10:32.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.351+0000 7f2bdbe00640 1 -- 172.21.15.6:0/3953863967 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2bd4020020 con 0x7f2be406b080 2024-03-22T13:10:32.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.619+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2be406be50 con 0x7f2be406b080 2024-03-22T13:10:32.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.640+0000 7f2bdbe00640 1 -- 172.21.15.6:0/3953863967 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f2bd4059290 con 0x7f2be406b080 2024-03-22T13:10:32.640 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:32.640 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:32.641 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2bc406bc20 msgr2=0x7f2bc406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2bc406bc20 0x7f2bc406e0e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0036000 tx=0x7f2bd003b040 comp rx=0 tx=0).stop 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406b080 msgr2=0x7f2be414a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406b080 0x7f2be414a9d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400b420 tx=0x7f2bd400b8e0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 shutdown_connections 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2be40c2ea0 0x7f2be415b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2be406fea0 0x7f2be414dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2bc406bc20 0x7f2bc406e0e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 --2- 172.21.15.6:0/3953863967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2be406b080 0x7f2be414a9d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 >> 172.21.15.6:0/3953863967 conn(0x7f2be40ccb50 msgr2=0x7f2be4153150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:32.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 shutdown_connections 2024-03-22T13:10:32.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:32.643+0000 7f2beb200640 1 -- 172.21.15.6:0/3953863967 wait complete. 2024-03-22T13:10:33.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:33.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:33.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:10:33.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:10:34.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:34.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.468+0000 7f3652000640 1 Processor -- start 2024-03-22T13:10:34.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.468+0000 7f3652000640 1 -- start start 2024-03-22T13:10:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.470+0000 7f3652000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c0c5040 0x7f364c0c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.470+0000 7f3652000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c1590b0 con 0x7f364c06f8c0 2024-03-22T13:10:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.470+0000 7f3652000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0721f0 con 0x7f364c152400 2024-03-22T13:10:34.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.470+0000 7f3652000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0724d0 con 0x7f364c0c5040 2024-03-22T13:10:34.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.479+0000 7f364b400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f364c152400 0x7f364c15b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:35262/0 (socket says 172.21.15.6:35262) 2024-03-22T13:10:34.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.479+0000 7f364b400640 1 -- 172.21.15.6:0/3605670798 learned_addr learned my addr 172.21.15.6:0/3605670798 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:34.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.509+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924001383 0 0) 0x7f364c0721f0 con 0x7f364c152400 2024-03-22T13:10:34.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.509+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630003520 con 0x7f364c152400 2024-03-22T13:10:34.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.514+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125988612 0 0) 0x7f3630003520 con 0x7f364c152400 2024-03-22T13:10:34.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.514+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c0c5040 msgr2=0x7f364c0c5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:34.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.514+0000 7f364a000640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c0c5040 0x7f364c0c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:34.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.514+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 >> v1:172.21.15.6:6789/0 conn(0x7f364c06f8c0 legacy=0x7f364c0716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:34.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.514+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c1582f0 con 0x7f364c152400 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1630228424 0 0) 0x7f363c001670 con 0x7f364c152400 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 >> v1:172.21.15.92:6789/0 conn(0x7f364c152400 legacy=0x7f364c15b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c0c5040 0x7f36300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36300040c0 0x7f3630004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630004d90 0x7f3630005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0724d0 con 0x7f36300040c0 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363000bc90 con 0x7f3630004d90 2024-03-22T13:10:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.517+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363000be90 con 0x7f364c0c5040 2024-03-22T13:10:34.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.521+0000 7f364be00640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630004d90 0x7f3630005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:34.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.521+0000 7f364aa00640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36300040c0 0x7f3630004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.526+0000 7f364aa00640 1 -- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c0c5040 msgr2=0x7f36300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.526+0000 7f364aa00640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c0c5040 0x7f36300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.526+0000 7f364aa00640 1 -- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630004d90 msgr2=0x7f3630005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.526+0000 7f364aa00640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630004d90 0x7f3630005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:34.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.526+0000 7f364aa00640 1 -- 172.21.15.6:0/3605670798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630003520 con 0x7f36300040c0 2024-03-22T13:10:34.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.530+0000 7f364aa00640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36300040c0 0x7f3630004800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3638001750 tx=0x7f3630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0f4c025a53feee server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:34.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.534+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638001370 con 0x7f36300040c0 2024-03-22T13:10:34.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.537+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36380052d0 con 0x7f36300040c0 2024-03-22T13:10:34.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.537+0000 7f364a000640 1 -- 172.21.15.6:0/3605670798 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638003830 con 0x7f36300040c0 2024-03-22T13:10:34.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.538+0000 7f3652000640 1 -- 172.21.15.6:0/3605670798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36300040c0 msgr2=0x7f3630004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.538+0000 7f3652000640 1 --2- 172.21.15.6:0/3605670798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36300040c0 0x7f3630004800 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3638001750 tx=0x7f3630005830 comp rx=0 tx=0).stop 2024-03-22T13:10:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.538+0000 7f364be00640 1 -- 172.21.15.6:0/3605670798 reap_dead start 2024-03-22T13:10:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.539+0000 7f3652000640 1 -- 172.21.15.6:0/3605670798 shutdown_connections 2024-03-22T13:10:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.539+0000 7f3652000640 1 -- 172.21.15.6:0/3605670798 >> 172.21.15.6:0/3605670798 conn(0x7f364c0cb930 msgr2=0x7f364c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.539+0000 7f3652000640 1 -- 172.21.15.6:0/3605670798 shutdown_connections 2024-03-22T13:10:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.539+0000 7f3652000640 1 -- 172.21.15.6:0/3605670798 wait complete. 2024-03-22T13:10:34.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.542+0000 7f3652000640 1 Processor -- start 2024-03-22T13:10:34.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.542+0000 7f3652000640 1 -- start start 2024-03-22T13:10:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.543+0000 7f3652000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c144690 0x7f364c1f3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.543+0000 7f3652000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f364c157e50 0x7f364c14e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.543+0000 7f3652000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c15bf10 0x7f364c151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.543+0000 7f3652000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c158b90 con 0x7f364c157e50 2024-03-22T13:10:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.543+0000 7f3652000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c1588b0 con 0x7f364c15bf10 2024-03-22T13:10:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.543+0000 7f3652000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c1527e0 con 0x7f364c144690 2024-03-22T13:10:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.544+0000 7f364b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c144690 0x7f364c1f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.544+0000 7f364be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c15bf10 0x7f364c151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.544+0000 7f364b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c144690 0x7f364c1f3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46780/0 (socket says 172.21.15.6:46780) 2024-03-22T13:10:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.544+0000 7f364b400640 1 -- 172.21.15.6:0/727612541 learned_addr learned my addr 172.21.15.6:0/727612541 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.545+0000 7f364aa00640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f364c157e50 0x7f364c14e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:34.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.549+0000 7f364aa00640 1 -- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c144690 msgr2=0x7f364c1f3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:34.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.549+0000 7f364aa00640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c144690 0x7f364c1f3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:34.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.549+0000 7f364aa00640 1 -- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c15bf10 msgr2=0x7f364c151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:34.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.549+0000 7f364aa00640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c15bf10 0x7f364c151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:34.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.549+0000 7f364aa00640 1 -- 172.21.15.6:0/727612541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c1f3c40 con 0x7f364c157e50 2024-03-22T13:10:34.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.552+0000 7f364aa00640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f364c157e50 0x7f364c14e3f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3638004000 tx=0x7f3638001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.556+0000 7f3643400640 1 -- 172.21.15.6:0/727612541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638001370 con 0x7f364c157e50 2024-03-22T13:10:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.556+0000 7f3643400640 1 -- 172.21.15.6:0/727612541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3638042070 con 0x7f364c157e50 2024-03-22T13:10:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.556+0000 7f3643400640 1 -- 172.21.15.6:0/727612541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638007a30 con 0x7f364c157e50 2024-03-22T13:10:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.556+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c0cb630 con 0x7f364c157e50 2024-03-22T13:10:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.556+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c0cb350 con 0x7f364c157e50 2024-03-22T13:10:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.557+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c0756c0 con 0x7f364c157e50 2024-03-22T13:10:34.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.851+0000 7f3643400640 1 -- 172.21.15.6:0/727612541 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f363803b040 con 0x7f364c157e50 2024-03-22T13:10:34.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.853+0000 7f3643400640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f362c06bc50 0x7f362c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:34.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.853+0000 7f364b400640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f362c06bc50 0x7f362c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:34.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.854+0000 7f364b400640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f362c06bc50 0x7f362c06e110 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f363c004c10 tx=0x7f363c001a60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.879+0000 7f3643400640 1 -- 172.21.15.6:0/727612541 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3638078f90 con 0x7f364c157e50 2024-03-22T13:10:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:34.881+0000 7f3643400640 1 -- 172.21.15.6:0/727612541 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f363807aef0 con 0x7f364c157e50 2024-03-22T13:10:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.167+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f364c1501a0 con 0x7f364c157e50 2024-03-22T13:10:35.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.182+0000 7f3643400640 1 -- 172.21.15.6:0/727612541 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f363807ef90 con 0x7f364c157e50 2024-03-22T13:10:35.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:35.183 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:35.183 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f362c06bc50 msgr2=0x7f362c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f362c06bc50 0x7f362c06e110 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f363c004c10 tx=0x7f363c001a60 comp rx=0 tx=0).stop 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f364c157e50 msgr2=0x7f364c14e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f364c157e50 0x7f364c14e3f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3638004000 tx=0x7f3638001120 comp rx=0 tx=0).stop 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 shutdown_connections 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f362c06bc50 0x7f362c06e110 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c15bf10 0x7f364c151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f364c157e50 0x7f364c14e3f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 --2- 172.21.15.6:0/727612541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f364c144690 0x7f364c1f3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 >> 172.21.15.6:0/727612541 conn(0x7f364c0cb930 msgr2=0x7f364c0c1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 shutdown_connections 2024-03-22T13:10:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:35.185+0000 7f3652000640 1 -- 172.21.15.6:0/727612541 wait complete. 2024-03-22T13:10:35.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:35.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:36.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.017+0000 7ff446400640 1 Processor -- start 2024-03-22T13:10:37.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.017+0000 7ff446400640 1 -- start start 2024-03-22T13:10:37.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.019+0000 7ff446400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:37.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.019+0000 7ff446400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4400cf770 0x7ff4400cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:37.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.019+0000 7ff446400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4400cccd0 0x7ff44015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:37.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.019+0000 7ff446400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4400ce230 con 0x7ff4400cccd0 2024-03-22T13:10:37.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.019+0000 7ff446400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4400ce510 con 0x7ff4400cf770 2024-03-22T13:10:37.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.019+0000 7ff446400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400ce7f0 con 0x7ff4400cc3b0 2024-03-22T13:10:37.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.020+0000 7ff43fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4400cccd0 0x7ff44015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:37.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.020+0000 7ff43f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:37.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.020+0000 7ff43ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4400cf770 0x7ff4400cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:37.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.020+0000 7ff43ea00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4400cf770 0x7ff4400cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48018/0 (socket says 172.21.15.6:48018) 2024-03-22T13:10:37.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.020+0000 7ff43ea00640 1 -- 172.21.15.6:0/1534173020 learned_addr learned my addr 172.21.15.6:0/1534173020 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:37.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.023+0000 7ff43ea00640 1 -- 172.21.15.6:0/1534173020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 msgr2=0x7ff4400cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:37.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.023+0000 7ff43ea00640 1 --2- 172.21.15.6:0/1534173020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.023+0000 7ff43ea00640 1 -- 172.21.15.6:0/1534173020 >> v2:172.21.15.6:3300/0 conn(0x7ff4400cccd0 msgr2=0x7ff44015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.023+0000 7ff43ea00640 1 --2- 172.21.15.6:0/1534173020 >> v2:172.21.15.6:3300/0 conn(0x7ff4400cccd0 0x7ff44015ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.023+0000 7ff43ea00640 1 -- 172.21.15.6:0/1534173020 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400cead0 con 0x7ff4400cf770 2024-03-22T13:10:37.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.026+0000 7ff43ea00640 1 --2- 172.21.15.6:0/1534173020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cf770 0x7ff4400cc790 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff44008d840 tx=0x7ff42c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e1ac1dc24f0a23e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:37.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.030+0000 7ff43e000640 1 -- 172.21.15.6:0/1534173020 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c02e070 con 0x7ff4400cf770 2024-03-22T13:10:37.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.030+0000 7ff43e000640 1 -- 172.21.15.6:0/1534173020 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c037070 con 0x7ff4400cf770 2024-03-22T13:10:37.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.030+0000 7ff43e000640 1 -- 172.21.15.6:0/1534173020 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c031a00 con 0x7ff4400cf770 2024-03-22T13:10:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 -- 172.21.15.6:0/1534173020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cf770 msgr2=0x7ff4400cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 --2- 172.21.15.6:0/1534173020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cf770 0x7ff4400cc790 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff44008d840 tx=0x7ff42c02b880 comp rx=0 tx=0).stop 2024-03-22T13:10:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 -- 172.21.15.6:0/1534173020 shutdown_connections 2024-03-22T13:10:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 --2- 172.21.15.6:0/1534173020 >> v2:172.21.15.6:3300/0 conn(0x7ff4400cccd0 0x7ff44015ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 --2- 172.21.15.6:0/1534173020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cf770 0x7ff4400cc790 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 --2- 172.21.15.6:0/1534173020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 -- 172.21.15.6:0/1534173020 >> 172.21.15.6:0/1534173020 conn(0x7ff4400b16d0 msgr2=0x7ff4400b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:37.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 -- 172.21.15.6:0/1534173020 shutdown_connections 2024-03-22T13:10:37.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.031+0000 7ff446400640 1 -- 172.21.15.6:0/1534173020 wait complete. 2024-03-22T13:10:37.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.034+0000 7ff446400640 1 Processor -- start 2024-03-22T13:10:37.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.034+0000 7ff446400640 1 -- start start 2024-03-22T13:10:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.035+0000 7ff446400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.035+0000 7ff446400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cccd0 0x7ff4400c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.035+0000 7ff446400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4400cf770 0x7ff440148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.035+0000 7ff446400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44015a5d0 con 0x7ff4400cf770 2024-03-22T13:10:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.035+0000 7ff446400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44006dc10 con 0x7ff4400cccd0 2024-03-22T13:10:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.035+0000 7ff446400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff440154bb0 con 0x7ff4400cc3b0 2024-03-22T13:10:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.036+0000 7ff43ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cccd0 0x7ff4400c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.036+0000 7ff43f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.036+0000 7ff43fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4400cf770 0x7ff440148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.036+0000 7ff43ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cccd0 0x7ff4400c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48032/0 (socket says 172.21.15.6:48032) 2024-03-22T13:10:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.036+0000 7ff43ea00640 1 -- 172.21.15.6:0/3797542684 learned_addr learned my addr 172.21.15.6:0/3797542684 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.040+0000 7ff43ea00640 1 -- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 msgr2=0x7ff4400b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.040+0000 7ff43ea00640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.040+0000 7ff43ea00640 1 -- 172.21.15.6:0/3797542684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4400cf770 msgr2=0x7ff440148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.040+0000 7ff43ea00640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4400cf770 0x7ff440148df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.040+0000 7ff43ea00640 1 -- 172.21.15.6:0/3797542684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400c8bb0 con 0x7ff4400cccd0 2024-03-22T13:10:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.043+0000 7ff43ea00640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cccd0 0x7ff4400c8670 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff42c000c00 tx=0x7ff42c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.048+0000 7ff437400640 1 -- 172.21.15.6:0/3797542684 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c032670 con 0x7ff4400cccd0 2024-03-22T13:10:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.048+0000 7ff437400640 1 -- 172.21.15.6:0/3797542684 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c032810 con 0x7ff4400cccd0 2024-03-22T13:10:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.048+0000 7ff437400640 1 -- 172.21.15.6:0/3797542684 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c032c00 con 0x7ff4400cccd0 2024-03-22T13:10:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.048+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4400c7360 con 0x7ff4400cccd0 2024-03-22T13:10:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.048+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4400c7920 con 0x7ff4400cccd0 2024-03-22T13:10:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.050+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4400756c0 con 0x7ff4400cccd0 2024-03-22T13:10:37.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.329+0000 7ff437400640 1 -- 172.21.15.6:0/3797542684 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff42c04c060 con 0x7ff4400cccd0 2024-03-22T13:10:37.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.330+0000 7ff437400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff42006bc20 0x7ff42006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:37.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.331+0000 7ff437400640 1 -- 172.21.15.6:0/3797542684 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff42c0acad0 con 0x7ff4400cccd0 2024-03-22T13:10:37.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.331+0000 7ff43f400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff42006bc20 0x7ff42006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:37.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.332+0000 7ff43f400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff42006bc20 0x7ff42006e0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff430009ac0 tx=0x7ff430009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.342+0000 7ff437400640 1 -- 172.21.15.6:0/3797542684 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff42c040070 con 0x7ff4400cccd0 2024-03-22T13:10:37.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.601+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4400cac00 con 0x7ff4400cccd0 2024-03-22T13:10:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.622+0000 7ff437400640 1 -- 172.21.15.6:0/3797542684 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7ff42c07ad10 con 0x7ff4400cccd0 2024-03-22T13:10:37.623 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:37.623 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:37.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff42006bc20 msgr2=0x7ff42006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:37.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff42006bc20 0x7ff42006e0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff430009ac0 tx=0x7ff430009190 comp rx=0 tx=0).stop 2024-03-22T13:10:37.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cccd0 msgr2=0x7ff4400c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cccd0 0x7ff4400c8670 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff42c000c00 tx=0x7ff42c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 shutdown_connections 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4400cf770 0x7ff440148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff42006bc20 0x7ff42006e0e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4400cccd0 0x7ff4400c8670 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 --2- 172.21.15.6:0/3797542684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4400cc3b0 0x7ff4400b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 >> 172.21.15.6:0/3797542684 conn(0x7ff4400b16d0 msgr2=0x7ff44006cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 shutdown_connections 2024-03-22T13:10:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:37.626+0000 7ff446400640 1 -- 172.21.15.6:0/3797542684 wait complete. 2024-03-22T13:10:38.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:38.249 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:39.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:39.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.453+0000 7fac21c00640 1 Processor -- start 2024-03-22T13:10:39.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.453+0000 7fac21c00640 1 -- start start 2024-03-22T13:10:39.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.454+0000 7fac21c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:39.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.454+0000 7fac21c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fac1c0ce8a0 0x7fac1c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:39.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.454+0000 7fac21c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c0cee00 con 0x7fac1c0cc760 2024-03-22T13:10:39.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.454+0000 7fac21c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c0cf0c0 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.454+0000 7fac21c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0cf3a0 con 0x7fac1c0cc380 2024-03-22T13:10:39.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.455+0000 7fac1be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:39.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.455+0000 7fac1b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fac1c0ce8a0 0x7fac1c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:39.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.456+0000 7fac1be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c06c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46800/0 (socket says 172.21.15.6:46800) 2024-03-22T13:10:39.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.456+0000 7fac1be00640 1 -- 172.21.15.6:0/3307225408 learned_addr learned my addr 172.21.15.6:0/3307225408 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:39.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.458+0000 7fac1be00640 1 -- 172.21.15.6:0/3307225408 >> v2:172.21.15.6:3300/0 conn(0x7fac1c0ce8a0 msgr2=0x7fac1c15ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:39.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.458+0000 7fac1be00640 1 --2- 172.21.15.6:0/3307225408 >> v2:172.21.15.6:3300/0 conn(0x7fac1c0ce8a0 0x7fac1c15ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:39.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.458+0000 7fac1be00640 1 -- 172.21.15.6:0/3307225408 >> v1:172.21.15.6:6789/0 conn(0x7fac1c0cc760 legacy=0x7fac1c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:39.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.458+0000 7fac1be00640 1 -- 172.21.15.6:0/3307225408 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c0cf680 con 0x7fac1c0cc380 2024-03-22T13:10:39.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.461+0000 7fac1be00640 1 --2- 172.21.15.6:0/3307225408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c06c230 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009870 tx=0x7fac0c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=184735ee752efcff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:39.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.466+0000 7fac1aa00640 1 -- 172.21.15.6:0/3307225408 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c034070 con 0x7fac1c0cc380 2024-03-22T13:10:39.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.466+0000 7fac1aa00640 1 -- 172.21.15.6:0/3307225408 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c030840 con 0x7fac1c0cc380 2024-03-22T13:10:39.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.466+0000 7fac1aa00640 1 -- 172.21.15.6:0/3307225408 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c030c10 con 0x7fac1c0cc380 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.466+0000 7fac21c00640 1 -- 172.21.15.6:0/3307225408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 msgr2=0x7fac1c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.466+0000 7fac21c00640 1 --2- 172.21.15.6:0/3307225408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c06c230 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009870 tx=0x7fac0c02b980 comp rx=0 tx=0).stop 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.467+0000 7fac21c00640 1 -- 172.21.15.6:0/3307225408 shutdown_connections 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.467+0000 7fac21c00640 1 --2- 172.21.15.6:0/3307225408 >> v2:172.21.15.6:3300/0 conn(0x7fac1c0ce8a0 0x7fac1c15ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.467+0000 7fac21c00640 1 --2- 172.21.15.6:0/3307225408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c06c230 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.467+0000 7fac21c00640 1 -- 172.21.15.6:0/3307225408 >> 172.21.15.6:0/3307225408 conn(0x7fac1c0b1480 msgr2=0x7fac1c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.467+0000 7fac21c00640 1 -- 172.21.15.6:0/3307225408 shutdown_connections 2024-03-22T13:10:39.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.467+0000 7fac21c00640 1 -- 172.21.15.6:0/3307225408 wait complete. 2024-03-22T13:10:39.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.468+0000 7fac21c00640 1 Processor -- start 2024-03-22T13:10:39.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.469+0000 7fac21c00640 1 -- start start 2024-03-22T13:10:39.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.469+0000 7fac21c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c1f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:39.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.469+0000 7fac21c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac1c0cc760 0x7fac1c151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:39.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.469+0000 7fac21c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac1c0ce8a0 0x7fac1c0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:39.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.469+0000 7fac21c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c1549d0 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.469+0000 7fac21c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c15a910 con 0x7fac1c0cc760 2024-03-22T13:10:39.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.469+0000 7fac21c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0ccb90 con 0x7fac1c0cc380 2024-03-22T13:10:39.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.470+0000 7fac1be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c1f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:39.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.470+0000 7fac1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac1c0cc760 0x7fac1c151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:39.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.470+0000 7fac21200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac1c0ce8a0 0x7fac1c0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:39.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.471+0000 7fac21200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac1c0ce8a0 0x7fac1c0c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32816/0 (socket says 172.21.15.6:32816) 2024-03-22T13:10:39.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.471+0000 7fac21200640 1 -- 172.21.15.6:0/4066808037 learned_addr learned my addr 172.21.15.6:0/4066808037 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:39.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.471+0000 7fac1be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c1f0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:46802/0 (socket says 172.21.15.6:46802) 2024-03-22T13:10:39.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.475+0000 7fac21200640 1 -- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 msgr2=0x7fac1c1f0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:39.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.475+0000 7fac21200640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c1f0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:39.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.475+0000 7fac21200640 1 -- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac1c0cc760 msgr2=0x7fac1c151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:39.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.475+0000 7fac21200640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac1c0cc760 0x7fac1c151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:39.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.475+0000 7fac21200640 1 -- 172.21.15.6:0/4066808037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c15b760 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.477+0000 7fac21200640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac1c0ce8a0 0x7fac1c0c75f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c7d0 tx=0x7fac1400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.483+0000 7fac13e00640 1 -- 172.21.15.6:0/4066808037 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14003b70 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.483+0000 7fac13e00640 1 -- 172.21.15.6:0/4066808037 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac14003d30 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.483+0000 7fac13e00640 1 -- 172.21.15.6:0/4066808037 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400aa90 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.483+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c15ba40 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.483+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c0c6df0 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.485+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c075650 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.778+0000 7fac13e00640 1 -- 172.21.15.6:0/4066808037 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac1400ac30 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.779+0000 7fac13e00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabfc06bc50 0x7fabfc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.780+0000 7fac1be00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabfc06bc50 0x7fabfc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.781+0000 7fac1be00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabfc06bc50 0x7fabfc06e110 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fac0c036000 tx=0x7fac0c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:39.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.797+0000 7fac13e00640 1 -- 172.21.15.6:0/4066808037 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fac14057850 con 0x7fac1c0ce8a0 2024-03-22T13:10:39.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:39.798+0000 7fac13e00640 1 -- 172.21.15.6:0/4066808037 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac140597b0 con 0x7fac1c0ce8a0 2024-03-22T13:10:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.071+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac1c0c9990 con 0x7fac1c0ce8a0 2024-03-22T13:10:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.093+0000 7fac13e00640 1 -- 172.21.15.6:0/4066808037 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fac1405d850 con 0x7fac1c0ce8a0 2024-03-22T13:10:40.094 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:40.094 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabfc06bc50 msgr2=0x7fabfc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabfc06bc50 0x7fabfc06e110 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fac0c036000 tx=0x7fac0c03b040 comp rx=0 tx=0).stop 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac1c0ce8a0 msgr2=0x7fac1c0c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac1c0ce8a0 0x7fac1c0c75f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c7d0 tx=0x7fac1400cc90 comp rx=0 tx=0).stop 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 shutdown_connections 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac1c0ce8a0 0x7fac1c0c75f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fabfc06bc50 0x7fabfc06e110 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac1c0cc760 0x7fac1c151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 --2- 172.21.15.6:0/4066808037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac1c0cc380 0x7fac1c1f0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:40.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 >> 172.21.15.6:0/4066808037 conn(0x7fac1c0b1480 msgr2=0x7fac1c1592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:40.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 shutdown_connections 2024-03-22T13:10:40.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:40.097+0000 7fac21c00640 1 -- 172.21.15.6:0/4066808037 wait complete. 2024-03-22T13:10:40.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:40.719 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:41.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:41.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.917+0000 7ff830200640 1 Processor -- start 2024-03-22T13:10:41.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.918+0000 7ff830200640 1 -- start start 2024-03-22T13:10:41.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.919+0000 7ff830200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8280ca590 0x7ff8280ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:41.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.919+0000 7ff830200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff8280cc5f0 0x7ff8280cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:41.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.919+0000 7ff830200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8280ccf30 con 0x7ff8280cdfd0 2024-03-22T13:10:41.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.919+0000 7ff830200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8280cdb40 con 0x7ff8280cc5f0 2024-03-22T13:10:41.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.919+0000 7ff830200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82815a790 con 0x7ff8280ca590 2024-03-22T13:10:41.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.920+0000 7ff82d800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8280ca590 0x7ff8280ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:41.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.920+0000 7ff82d800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff8280ca590 0x7ff8280ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48058/0 (socket says 172.21.15.6:48058) 2024-03-22T13:10:41.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.920+0000 7ff82d800640 1 -- 172.21.15.6:0/1050718878 learned_addr learned my addr 172.21.15.6:0/1050718878 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.922+0000 7ff82d800640 1 -- 172.21.15.6:0/1050718878 >> v2:172.21.15.6:3300/0 conn(0x7ff8280cc5f0 msgr2=0x7ff8280cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.922+0000 7ff82d800640 1 --2- 172.21.15.6:0/1050718878 >> v2:172.21.15.6:3300/0 conn(0x7ff8280cc5f0 0x7ff8280cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.922+0000 7ff82d800640 1 -- 172.21.15.6:0/1050718878 >> v1:172.21.15.6:6789/0 conn(0x7ff8280cdfd0 legacy=0x7ff8280ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.922+0000 7ff82d800640 1 -- 172.21.15.6:0/1050718878 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82815aa70 con 0x7ff8280ca590 2024-03-22T13:10:41.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.925+0000 7ff82d800640 1 --2- 172.21.15.6:0/1050718878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280ca590 0x7ff8280ca970 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff82806bd10 tx=0x7ff81c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da11a1de0ef0d458 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:41.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.927+0000 7ff827e00640 1 -- 172.21.15.6:0/1050718878 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c034070 con 0x7ff8280ca590 2024-03-22T13:10:41.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.929+0000 7ff827e00640 1 -- 172.21.15.6:0/1050718878 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81c030860 con 0x7ff8280ca590 2024-03-22T13:10:41.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.929+0000 7ff827e00640 1 -- 172.21.15.6:0/1050718878 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c030c30 con 0x7ff8280ca590 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 -- 172.21.15.6:0/1050718878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280ca590 msgr2=0x7ff8280ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 --2- 172.21.15.6:0/1050718878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280ca590 0x7ff8280ca970 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff82806bd10 tx=0x7ff81c02b9a0 comp rx=0 tx=0).stop 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 -- 172.21.15.6:0/1050718878 shutdown_connections 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 --2- 172.21.15.6:0/1050718878 >> v2:172.21.15.6:3300/0 conn(0x7ff8280cc5f0 0x7ff8280cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 --2- 172.21.15.6:0/1050718878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280ca590 0x7ff8280ca970 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 -- 172.21.15.6:0/1050718878 >> 172.21.15.6:0/1050718878 conn(0x7ff8280b1720 msgr2=0x7ff8280b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 -- 172.21.15.6:0/1050718878 shutdown_connections 2024-03-22T13:10:41.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.930+0000 7ff830200640 1 -- 172.21.15.6:0/1050718878 wait complete. 2024-03-22T13:10:41.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.933+0000 7ff830200640 1 Processor -- start 2024-03-22T13:10:41.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.933+0000 7ff830200640 1 -- start start 2024-03-22T13:10:41.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.934+0000 7ff830200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8280ca590 0x7ff8280b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:41.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.934+0000 7ff830200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8280cc5f0 0x7ff82815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:41.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.934+0000 7ff830200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280cdfd0 0x7ff8281f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:41.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.934+0000 7ff830200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82815b310 con 0x7ff8280ca590 2024-03-22T13:10:41.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.934+0000 7ff830200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82815b030 con 0x7ff8280cdfd0 2024-03-22T13:10:41.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.934+0000 7ff830200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82815a480 con 0x7ff8280cc5f0 2024-03-22T13:10:41.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.935+0000 7ff82e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280cdfd0 0x7ff8281f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:41.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.935+0000 7ff82ce00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8280cc5f0 0x7ff82815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:41.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.936+0000 7ff82e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280cdfd0 0x7ff8281f3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:48072/0 (socket says 172.21.15.6:48072) 2024-03-22T13:10:41.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.936+0000 7ff82e200640 1 -- 172.21.15.6:0/1796119181 learned_addr learned my addr 172.21.15.6:0/1796119181 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:41.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.937+0000 7ff82d800640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8280ca590 0x7ff8280b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:41.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.941+0000 7ff82e200640 1 -- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8280cc5f0 msgr2=0x7ff82815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:41.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.941+0000 7ff82e200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8280cc5f0 0x7ff82815dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:41.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.941+0000 7ff82e200640 1 -- 172.21.15.6:0/1796119181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8280ca590 msgr2=0x7ff8280b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:41.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.941+0000 7ff82e200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8280ca590 0x7ff8280b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:41.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.941+0000 7ff82e200640 1 -- 172.21.15.6:0/1796119181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff828159620 con 0x7ff8280cdfd0 2024-03-22T13:10:41.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.947+0000 7ff82e200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280cdfd0 0x7ff8281f3ec0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff82000b990 tx=0x7ff82000bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:41.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.952+0000 7ff825600640 1 -- 172.21.15.6:0/1796119181 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820019070 con 0x7ff8280cdfd0 2024-03-22T13:10:41.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.952+0000 7ff825600640 1 -- 172.21.15.6:0/1796119181 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82000caa0 con 0x7ff8280cdfd0 2024-03-22T13:10:41.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.952+0000 7ff825600640 1 -- 172.21.15.6:0/1796119181 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820014440 con 0x7ff8280cdfd0 2024-03-22T13:10:41.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.952+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff828159900 con 0x7ff8280cdfd0 2024-03-22T13:10:41.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.952+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8281e5b30 con 0x7ff8280cdfd0 2024-03-22T13:10:41.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:41.953+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8280756c0 con 0x7ff8280cdfd0 2024-03-22T13:10:42.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.259+0000 7ff825600640 1 -- 172.21.15.6:0/1796119181 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff82001c070 con 0x7ff8280cdfd0 2024-03-22T13:10:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.261+0000 7ff825600640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff80806bc20 0x7ff80806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.262+0000 7ff82d800640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff80806bc20 0x7ff80806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:42.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.263+0000 7ff825600640 1 -- 172.21.15.6:0/1796119181 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff820057020 con 0x7ff8280cdfd0 2024-03-22T13:10:42.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.263+0000 7ff82d800640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff80806bc20 0x7ff80806e0e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff81c036000 tx=0x7ff81c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:42.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.274+0000 7ff825600640 1 -- 172.21.15.6:0/1796119181 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff820023020 con 0x7ff8280cdfd0 2024-03-22T13:10:42.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.541+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8280b58b0 con 0x7ff8280cdfd0 2024-03-22T13:10:42.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.564+0000 7ff825600640 1 -- 172.21.15.6:0/1796119181 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7ff820058f80 con 0x7ff8280cdfd0 2024-03-22T13:10:42.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:42.565 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:42.565 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:42.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff80806bc20 msgr2=0x7ff80806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:42.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff80806bc20 0x7ff80806e0e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff81c036000 tx=0x7ff81c008030 comp rx=0 tx=0).stop 2024-03-22T13:10:42.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280cdfd0 msgr2=0x7ff8281f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:42.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280cdfd0 0x7ff8281f3ec0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff82000b990 tx=0x7ff82000bd90 comp rx=0 tx=0).stop 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 shutdown_connections 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff80806bc20 0x7ff80806e0e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8280cdfd0 0x7ff8281f3ec0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8280cc5f0 0x7ff82815dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 --2- 172.21.15.6:0/1796119181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8280ca590 0x7ff8280b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 >> 172.21.15.6:0/1796119181 conn(0x7ff8280b1720 msgr2=0x7ff82806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 shutdown_connections 2024-03-22T13:10:42.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:42.568+0000 7ff830200640 1 -- 172.21.15.6:0/1796119181 wait complete. 2024-03-22T13:10:43.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:43.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:44.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.395+0000 7fd397200640 1 Processor -- start 2024-03-22T13:10:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.395+0000 7fd397200640 1 -- start start 2024-03-22T13:10:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.396+0000 7fd397200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd398154430 0x7fd39815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.396+0000 7fd397200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd398154810 0x7fd398072980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.396+0000 7fd397200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39815a410 con 0x7fd39815aba0 2024-03-22T13:10:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.396+0000 7fd397200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd398072ec0 con 0x7fd398154810 2024-03-22T13:10:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.396+0000 7fd397200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39815def0 con 0x7fd398154430 2024-03-22T13:10:44.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.421+0000 7fd395e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd398154430 0x7fd39815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:44.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.421+0000 7fd395e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd398154430 0x7fd39815d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:55070/0 (socket says 172.21.15.6:55070) 2024-03-22T13:10:44.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.421+0000 7fd395e00640 1 -- 172.21.15.6:0/2822629551 learned_addr learned my addr 172.21.15.6:0/2822629551 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:44.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.425+0000 7fd395e00640 1 -- 172.21.15.6:0/2822629551 >> v2:172.21.15.6:3300/0 conn(0x7fd398154810 msgr2=0x7fd398072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:44.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.425+0000 7fd395e00640 1 --2- 172.21.15.6:0/2822629551 >> v2:172.21.15.6:3300/0 conn(0x7fd398154810 0x7fd398072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:44.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.425+0000 7fd395e00640 1 -- 172.21.15.6:0/2822629551 >> v1:172.21.15.92:6789/0 conn(0x7fd39815aba0 legacy=0x7fd398071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:44.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.425+0000 7fd395e00640 1 -- 172.21.15.6:0/2822629551 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39815e1d0 con 0x7fd398154430 2024-03-22T13:10:44.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.430+0000 7fd395e00640 1 --2- 172.21.15.6:0/2822629551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 0x7fd39815d8f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd3980cc740 tx=0x7fd38802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ed42b1d1b66074f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:44.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.435+0000 7fd394a00640 1 -- 172.21.15.6:0/2822629551 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38802f850 con 0x7fd398154430 2024-03-22T13:10:44.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.435+0000 7fd394a00640 1 -- 172.21.15.6:0/2822629551 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38802f9f0 con 0x7fd398154430 2024-03-22T13:10:44.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.435+0000 7fd394a00640 1 -- 172.21.15.6:0/2822629551 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38802fdc0 con 0x7fd398154430 2024-03-22T13:10:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.436+0000 7fd397200640 1 -- 172.21.15.6:0/2822629551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 msgr2=0x7fd39815d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.436+0000 7fd397200640 1 --2- 172.21.15.6:0/2822629551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 0x7fd39815d8f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd3980cc740 tx=0x7fd38802ba30 comp rx=0 tx=0).stop 2024-03-22T13:10:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.437+0000 7fd397200640 1 -- 172.21.15.6:0/2822629551 shutdown_connections 2024-03-22T13:10:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.437+0000 7fd397200640 1 --2- 172.21.15.6:0/2822629551 >> v2:172.21.15.6:3300/0 conn(0x7fd398154810 0x7fd398072980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.437+0000 7fd397200640 1 --2- 172.21.15.6:0/2822629551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 0x7fd39815d8f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.437+0000 7fd397200640 1 -- 172.21.15.6:0/2822629551 >> 172.21.15.6:0/2822629551 conn(0x7fd3980cb890 msgr2=0x7fd3980cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.437+0000 7fd397200640 1 -- 172.21.15.6:0/2822629551 shutdown_connections 2024-03-22T13:10:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.437+0000 7fd397200640 1 -- 172.21.15.6:0/2822629551 wait complete. 2024-03-22T13:10:44.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.439+0000 7fd397200640 1 Processor -- start 2024-03-22T13:10:44.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.439+0000 7fd397200640 1 -- start start 2024-03-22T13:10:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.440+0000 7fd397200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 0x7fd3981f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.440+0000 7fd397200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398154810 0x7fd3981510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.440+0000 7fd397200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd39815aba0 0x7fd39814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.440+0000 7fd397200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980c6b90 con 0x7fd39815aba0 2024-03-22T13:10:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.440+0000 7fd397200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39815a680 con 0x7fd398154430 2024-03-22T13:10:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.440+0000 7fd397200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39815e790 con 0x7fd398154810 2024-03-22T13:10:44.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.441+0000 7fd396800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd39815aba0 0x7fd39814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:44.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.441+0000 7fd396800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd39815aba0 0x7fd39814b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38066/0 (socket says 172.21.15.6:38066) 2024-03-22T13:10:44.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.441+0000 7fd396800640 1 -- 172.21.15.6:0/3741136558 learned_addr learned my addr 172.21.15.6:0/3741136558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:44.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.445+0000 7fd396800640 1 -- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398154810 msgr2=0x7fd3981510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:44.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.445+0000 7fd396800640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398154810 0x7fd3981510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:44.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.445+0000 7fd396800640 1 -- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 msgr2=0x7fd3981f01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:44.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.445+0000 7fd396800640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 0x7fd3981f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:44.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.445+0000 7fd396800640 1 -- 172.21.15.6:0/3741136558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39815b6d0 con 0x7fd39815aba0 2024-03-22T13:10:44.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.448+0000 7fd396800640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd39815aba0 0x7fd39814b380 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c6e0 tx=0x7fd39000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.491+0000 7fd38e000640 1 -- 172.21.15.6:0/3741136558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390003b70 con 0x7fd39815aba0 2024-03-22T13:10:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.491+0000 7fd38e000640 1 -- 172.21.15.6:0/3741136558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd390003d30 con 0x7fd39815aba0 2024-03-22T13:10:44.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.491+0000 7fd38e000640 1 -- 172.21.15.6:0/3741136558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390005310 con 0x7fd39815aba0 2024-03-22T13:10:44.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.492+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3981599b0 con 0x7fd39815aba0 2024-03-22T13:10:44.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.492+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd398159f70 con 0x7fd39815aba0 2024-03-22T13:10:44.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.494+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd398075650 con 0x7fd39815aba0 2024-03-22T13:10:44.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.788+0000 7fd38e000640 1 -- 172.21.15.6:0/3741136558 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd3900054b0 con 0x7fd39815aba0 2024-03-22T13:10:44.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.789+0000 7fd38e000640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd37806bc20 0x7fd37806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:44.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.790+0000 7fd395e00640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd37806bc20 0x7fd37806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.791+0000 7fd395e00640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd37806bc20 0x7fd37806e0e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd388036000 tx=0x7fd388039040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:44.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.807+0000 7fd38e000640 1 -- 172.21.15.6:0/3741136558 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd3900057a0 con 0x7fd39815aba0 2024-03-22T13:10:44.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:44.808+0000 7fd38e000640 1 -- 172.21.15.6:0/3741136558 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd390059ed0 con 0x7fd39815aba0 2024-03-22T13:10:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.055+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd39815b190 con 0x7fd39815aba0 2024-03-22T13:10:45.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.083+0000 7fd38e000640 1 -- 172.21.15.6:0/3741136558 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fd3900597d0 con 0x7fd39815aba0 2024-03-22T13:10:45.084 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:45.084 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:45.084 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:45.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.087+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd37806bc20 msgr2=0x7fd37806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:45.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.087+0000 7fd397200640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd37806bc20 0x7fd37806e0e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd388036000 tx=0x7fd388039040 comp rx=0 tx=0).stop 2024-03-22T13:10:45.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.087+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd39815aba0 msgr2=0x7fd39814b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:45.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.087+0000 7fd397200640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd39815aba0 0x7fd39814b380 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c6e0 tx=0x7fd39000cba0 comp rx=0 tx=0).stop 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 shutdown_connections 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd39815aba0 0x7fd39814b380 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398154810 0x7fd3981510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd37806bc20 0x7fd37806e0e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 --2- 172.21.15.6:0/3741136558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398154430 0x7fd3981f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 >> 172.21.15.6:0/3741136558 conn(0x7fd3980cb890 msgr2=0x7fd3980c5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 shutdown_connections 2024-03-22T13:10:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:45.088+0000 7fd397200640 1 -- 172.21.15.6:0/3741136558 wait complete. 2024-03-22T13:10:45.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:45.718 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:46.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:46.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.923+0000 7f2511200640 1 Processor -- start 2024-03-22T13:10:46.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.923+0000 7f2511200640 1 -- start start 2024-03-22T13:10:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.925+0000 7f2511200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f250c06d840 0x7f250c15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.925+0000 7f2511200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c06c7a0 con 0x7f250c0ccad0 2024-03-22T13:10:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.925+0000 7f2511200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250c06ca80 con 0x7f250c0ca520 2024-03-22T13:10:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.925+0000 7f2511200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f250c0cd000 con 0x7f250c06d840 2024-03-22T13:10:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.926+0000 7f2509400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f250c0ccad0 0x7f250c162da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58206/0 (socket says 172.21.15.6:58206) 2024-03-22T13:10:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.926+0000 7f2509400640 1 -- 172.21.15.6:0/2240841712 learned_addr learned my addr 172.21.15.6:0/2240841712 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:46.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.926+0000 7f250a800640 1 --2- 172.21.15.6:0/2240841712 >> v2:172.21.15.92:3300/0 conn(0x7f250c06d840 0x7f250c15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:46.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.929+0000 7f2508a00640 1 -- 172.21.15.6:0/2240841712 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229513598 0 0) 0x7f250c06c7a0 con 0x7f250c0ccad0 2024-03-22T13:10:46.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.929+0000 7f2508a00640 1 -- 172.21.15.6:0/2240841712 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f0003520 con 0x7f250c0ccad0 2024-03-22T13:10:46.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.930+0000 7f250a800640 1 -- 172.21.15.6:0/2240841712 >> v1:172.21.15.92:6789/0 conn(0x7f250c0ca520 legacy=0x7f250c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:46.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.930+0000 7f250a800640 1 -- 172.21.15.6:0/2240841712 >> v1:172.21.15.6:6789/0 conn(0x7f250c0ccad0 legacy=0x7f250c162da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:46.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.930+0000 7f250a800640 1 -- 172.21.15.6:0/2240841712 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c06cd60 con 0x7f250c06d840 2024-03-22T13:10:46.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.937+0000 7f250a800640 1 --2- 172.21.15.6:0/2240841712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c06d840 0x7f250c15b0e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2504009d90 tx=0x7f250402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6457610137e73c75 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.940+0000 7f2508a00640 1 -- 172.21.15.6:0/2240841712 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504039070 con 0x7f250c06d840 2024-03-22T13:10:46.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.943+0000 7f2508a00640 1 -- 172.21.15.6:0/2240841712 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250402cd00 con 0x7f250c06d840 2024-03-22T13:10:46.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.943+0000 7f2508a00640 1 -- 172.21.15.6:0/2240841712 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504034640 con 0x7f250c06d840 2024-03-22T13:10:46.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.943+0000 7f2511200640 1 -- 172.21.15.6:0/2240841712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c06d840 msgr2=0x7f250c15b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:46.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.943+0000 7f2511200640 1 --2- 172.21.15.6:0/2240841712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c06d840 0x7f250c15b0e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2504009d90 tx=0x7f250402bf50 comp rx=0 tx=0).stop 2024-03-22T13:10:46.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.943+0000 7f2511200640 1 -- 172.21.15.6:0/2240841712 shutdown_connections 2024-03-22T13:10:46.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.943+0000 7f2511200640 1 --2- 172.21.15.6:0/2240841712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c06d840 0x7f250c15b0e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:46.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.944+0000 7f2511200640 1 -- 172.21.15.6:0/2240841712 >> 172.21.15.6:0/2240841712 conn(0x7f250c0b16b0 msgr2=0x7f250c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:46.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.944+0000 7f2511200640 1 -- 172.21.15.6:0/2240841712 shutdown_connections 2024-03-22T13:10:46.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.944+0000 7f2511200640 1 -- 172.21.15.6:0/2240841712 wait complete. 2024-03-22T13:10:46.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.946+0000 7f2511200640 1 Processor -- start 2024-03-22T13:10:46.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.946+0000 7f2511200640 1 -- start start 2024-03-22T13:10:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.947+0000 7f2511200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f250c06d840 0x7f250c153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.947+0000 7f2511200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f250c0ca520 0x7f250c0b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.947+0000 7f2511200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c0ccad0 0x7f250c1f0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.947+0000 7f2511200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c154c20 con 0x7f250c0ca520 2024-03-22T13:10:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.947+0000 7f2511200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c06d320 con 0x7f250c0ccad0 2024-03-22T13:10:46.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.947+0000 7f2511200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250c15b980 con 0x7f250c06d840 2024-03-22T13:10:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.948+0000 7f2509400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f250c0ca520 0x7f250c0b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.948+0000 7f2509e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f250c06d840 0x7f250c153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.949+0000 7f2509e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f250c06d840 0x7f250c153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47162/0 (socket says 172.21.15.6:47162) 2024-03-22T13:10:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.949+0000 7f2509e00640 1 -- 172.21.15.6:0/2732928945 learned_addr learned my addr 172.21.15.6:0/2732928945 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.949+0000 7f250a800640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c0ccad0 0x7f250c1f0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:46.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.953+0000 7f2509e00640 1 -- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c0ccad0 msgr2=0x7f250c1f0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.953+0000 7f2509e00640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c0ccad0 0x7f250c1f0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.953+0000 7f2509e00640 1 -- 172.21.15.6:0/2732928945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f250c0ca520 msgr2=0x7f250c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.953+0000 7f2509e00640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f250c0ca520 0x7f250c0b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.953+0000 7f2509e00640 1 -- 172.21.15.6:0/2732928945 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c0b4510 con 0x7f250c06d840 2024-03-22T13:10:46.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.956+0000 7f2509e00640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f250c06d840 0x7f250c153de0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00c480 tx=0x7f24fc00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:46.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.963+0000 7f2502000640 1 -- 172.21.15.6:0/2732928945 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc0052d0 con 0x7f250c06d840 2024-03-22T13:10:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.963+0000 7f2502000640 1 -- 172.21.15.6:0/2732928945 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc005470 con 0x7f250c06d840 2024-03-22T13:10:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.963+0000 7f2502000640 1 -- 172.21.15.6:0/2732928945 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc012870 con 0x7f250c06d840 2024-03-22T13:10:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.963+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250c0cf530 con 0x7f250c06d840 2024-03-22T13:10:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.964+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250c15cb00 con 0x7f250c06d840 2024-03-22T13:10:46.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:46.965+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c0756c0 con 0x7f250c06d840 2024-03-22T13:10:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.003+0000 7f2502000640 1 -- 172.21.15.6:0/2732928945 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f24fc012a10 con 0x7f250c06d840 2024-03-22T13:10:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.006+0000 7f2502000640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f24ec06bc20 0x7f24ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.006+0000 7f2509400640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f24ec06bc20 0x7f24ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.007+0000 7f2509400640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f24ec06bc20 0x7f24ec06e0e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f250c1f1580 tx=0x7f24f8007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:47.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.276+0000 7f2502000640 1 -- 172.21.15.6:0/2732928945 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f24fc0575a0 con 0x7f250c06d840 2024-03-22T13:10:47.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.295+0000 7f2502000640 1 -- 172.21.15.6:0/2732928945 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f250c0cf530 con 0x7f250c06d840 2024-03-22T13:10:47.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.555+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f250c0c3090 con 0x7f250c06d840 2024-03-22T13:10:47.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.577+0000 7f2502000640 1 -- 172.21.15.6:0/2732928945 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f24fc059500 con 0x7f250c06d840 2024-03-22T13:10:47.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:47.577 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:47.578 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f24ec06bc20 msgr2=0x7f24ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f24ec06bc20 0x7f24ec06e0e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f250c1f1580 tx=0x7f24f8007d30 comp rx=0 tx=0).stop 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f250c06d840 msgr2=0x7f250c153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f250c06d840 0x7f250c153de0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00c480 tx=0x7f24fc00c940 comp rx=0 tx=0).stop 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 shutdown_connections 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f24ec06bc20 0x7f24ec06e0e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f250c0ccad0 0x7f250c1f0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f250c0ca520 0x7f250c0b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:47.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 --2- 172.21.15.6:0/2732928945 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f250c06d840 0x7f250c153de0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 >> 172.21.15.6:0/2732928945 conn(0x7f250c0b16b0 msgr2=0x7f250c0b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 shutdown_connections 2024-03-22T13:10:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:47.580+0000 7f2511200640 1 -- 172.21.15.6:0/2732928945 wait complete. 2024-03-22T13:10:48.208 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:48.209 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:49.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.408+0000 7f116ca00640 1 Processor -- start 2024-03-22T13:10:49.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.408+0000 7f116ca00640 1 -- start start 2024-03-22T13:10:49.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.410+0000 7f116ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154810 0x7f116815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:49.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.410+0000 7f116ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116815a410 con 0x7f1168154430 2024-03-22T13:10:49.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.410+0000 7f116ca00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116815def0 con 0x7f116815aba0 2024-03-22T13:10:49.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.410+0000 7f116ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116815e1d0 con 0x7f1168154810 2024-03-22T13:10:49.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.410+0000 7f1166a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154810 0x7f116815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:49.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.411+0000 7f1167400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1168154430 0x7f1168072980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58214/0 (socket says 172.21.15.6:58214) 2024-03-22T13:10:49.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.411+0000 7f1167400640 1 -- 172.21.15.6:0/4281664714 learned_addr learned my addr 172.21.15.6:0/4281664714 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:49.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.414+0000 7f1166a00640 1 -- 172.21.15.6:0/4281664714 >> v1:172.21.15.92:6789/0 conn(0x7f116815aba0 legacy=0x7f1168071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:49.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.414+0000 7f1166a00640 1 -- 172.21.15.6:0/4281664714 >> v1:172.21.15.6:6789/0 conn(0x7f1168154430 legacy=0x7f1168072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:49.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.414+0000 7f1166a00640 1 -- 172.21.15.6:0/4281664714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116815e4b0 con 0x7f1168154810 2024-03-22T13:10:49.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.416+0000 7f1166a00640 1 --2- 172.21.15.6:0/4281664714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154810 0x7f116815d8f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f11540097d0 tx=0x7f115402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d70fe7e550a9ff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:49.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.419+0000 7f1166000640 1 -- 172.21.15.6:0/4281664714 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1154034070 con 0x7f1168154810 2024-03-22T13:10:49.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.421+0000 7f1166000640 1 -- 172.21.15.6:0/4281664714 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1154030880 con 0x7f1168154810 2024-03-22T13:10:49.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.421+0000 7f1166000640 1 -- 172.21.15.6:0/4281664714 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1154030c50 con 0x7f1168154810 2024-03-22T13:10:49.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.422+0000 7f116ca00640 1 -- 172.21.15.6:0/4281664714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154810 msgr2=0x7f116815d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:49.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.422+0000 7f116ca00640 1 --2- 172.21.15.6:0/4281664714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154810 0x7f116815d8f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f11540097d0 tx=0x7f115402b9c0 comp rx=0 tx=0).stop 2024-03-22T13:10:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.423+0000 7f116ca00640 1 -- 172.21.15.6:0/4281664714 shutdown_connections 2024-03-22T13:10:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.423+0000 7f116ca00640 1 --2- 172.21.15.6:0/4281664714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154810 0x7f116815d8f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.423+0000 7f116ca00640 1 -- 172.21.15.6:0/4281664714 >> 172.21.15.6:0/4281664714 conn(0x7f11680cb890 msgr2=0x7f11680cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.423+0000 7f116ca00640 1 -- 172.21.15.6:0/4281664714 shutdown_connections 2024-03-22T13:10:49.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.423+0000 7f116ca00640 1 -- 172.21.15.6:0/4281664714 wait complete. 2024-03-22T13:10:49.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.425+0000 7f116ca00640 1 Processor -- start 2024-03-22T13:10:49.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.426+0000 7f116ca00640 1 -- start start 2024-03-22T13:10:49.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.427+0000 7f116ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154430 0x7f116806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:49.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.427+0000 7f116ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1168154810 0x7f11681f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:49.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.427+0000 7f116ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f116815aba0 0x7f11681512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:49.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.427+0000 7f116ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815e790 con 0x7f116815aba0 2024-03-22T13:10:49.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.427+0000 7f116ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11680c6b90 con 0x7f1168154810 2024-03-22T13:10:49.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.427+0000 7f116ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1168071b90 con 0x7f1168154430 2024-03-22T13:10:49.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.428+0000 7f1167400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154430 0x7f116806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:49.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.428+0000 7f1166a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1168154810 0x7f11681f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:49.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.428+0000 7f1167e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f116815aba0 0x7f11681512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:49.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.428+0000 7f1167400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154430 0x7f116806ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:47184/0 (socket says 172.21.15.6:47184) 2024-03-22T13:10:49.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.428+0000 7f1167400640 1 -- 172.21.15.6:0/1472767756 learned_addr learned my addr 172.21.15.6:0/1472767756 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:49.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.432+0000 7f1167400640 1 -- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1168154810 msgr2=0x7f11681f0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:49.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.432+0000 7f1167400640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1168154810 0x7f11681f0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:49.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.432+0000 7f1167400640 1 -- 172.21.15.6:0/1472767756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f116815aba0 msgr2=0x7f11681512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:49.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.432+0000 7f1167400640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f116815aba0 0x7f11681512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:49.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.432+0000 7f1167400640 1 -- 172.21.15.6:0/1472767756 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168155890 con 0x7f1168154430 2024-03-22T13:10:49.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.435+0000 7f1167400640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154430 0x7f116806ee40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f115800c400 tx=0x7f115800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.437+0000 7f115f400640 1 -- 172.21.15.6:0/1472767756 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158017070 con 0x7f1168154430 2024-03-22T13:10:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.437+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11681f03a0 con 0x7f1168154430 2024-03-22T13:10:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.437+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168155e10 con 0x7f1168154430 2024-03-22T13:10:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.438+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1168075650 con 0x7f1168154430 2024-03-22T13:10:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.438+0000 7f115f400640 1 -- 172.21.15.6:0/1472767756 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1158005ce0 con 0x7f1168154430 2024-03-22T13:10:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.439+0000 7f115f400640 1 -- 172.21.15.6:0/1472767756 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158021060 con 0x7f1168154430 2024-03-22T13:10:49.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.731+0000 7f115f400640 1 -- 172.21.15.6:0/1472767756 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f115801a070 con 0x7f1168154430 2024-03-22T13:10:49.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.733+0000 7f115f400640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f114806bbd0 0x7f114806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:49.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.733+0000 7f115f400640 1 -- 172.21.15.6:0/1472767756 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1158056b10 con 0x7f1168154430 2024-03-22T13:10:49.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.733+0000 7f1166a00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f114806bbd0 0x7f114806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:49.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.735+0000 7f1166a00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f114806bbd0 0x7f114806e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1154036000 tx=0x7f115403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:49.752+0000 7f115f400640 1 -- 172.21.15.6:0/1472767756 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1158058a70 con 0x7f1168154430 2024-03-22T13:10:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.007+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1168153c50 con 0x7f1168154430 2024-03-22T13:10:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.031+0000 7f115f400640 1 -- 172.21.15.6:0/1472767756 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f115805cb10 con 0x7f1168154430 2024-03-22T13:10:50.031 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:50.031 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.033+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f114806bbd0 msgr2=0x7f114806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.033+0000 7f116ca00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f114806bbd0 0x7f114806e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1154036000 tx=0x7f115403b040 comp rx=0 tx=0).stop 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.033+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154430 msgr2=0x7f116806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.033+0000 7f116ca00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154430 0x7f116806ee40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f115800c400 tx=0x7f115800c8c0 comp rx=0 tx=0).stop 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 shutdown_connections 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f116815aba0 0x7f11681512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f114806bbd0 0x7f114806e090 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1168154810 0x7f11681f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 --2- 172.21.15.6:0/1472767756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1168154430 0x7f116806ee40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 >> 172.21.15.6:0/1472767756 conn(0x7f11680cb890 msgr2=0x7f1168151de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 shutdown_connections 2024-03-22T13:10:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:50.034+0000 7f116ca00640 1 -- 172.21.15.6:0/1472767756 wait complete. 2024-03-22T13:10:50.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:50.655 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:51.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.856+0000 7f1c04a00640 1 Processor -- start 2024-03-22T13:10:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.856+0000 7f1c04a00640 1 -- start start 2024-03-22T13:10:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.857+0000 7f1c04a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c000ccad0 0x7f1c0006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.857+0000 7f1c04a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0006c7a0 con 0x7f1c000ca520 2024-03-22T13:10:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.857+0000 7f1c04a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0006ca80 con 0x7f1c0015b0e0 2024-03-22T13:10:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.857+0000 7f1c04a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c000cd000 con 0x7f1c000ccad0 2024-03-22T13:10:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.858+0000 7f1bfe800640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1c0015b0e0 0x7f1c00164db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:37028/0 (socket says 172.21.15.6:37028) 2024-03-22T13:10:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.858+0000 7f1bfe800640 1 -- 172.21.15.6:0/4001713222 learned_addr learned my addr 172.21.15.6:0/4001713222 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.858+0000 7f1bfd400640 1 --2- 172.21.15.6:0/4001713222 >> v2:172.21.15.6:3300/0 conn(0x7f1c000ccad0 0x7f1c0006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:51.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.861+0000 7f1bfca00640 1 -- 172.21.15.6:0/4001713222 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964271465 0 0) 0x7f1c0006ca80 con 0x7f1c0015b0e0 2024-03-22T13:10:51.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.861+0000 7f1bfca00640 1 -- 172.21.15.6:0/4001713222 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be4003520 con 0x7f1c0015b0e0 2024-03-22T13:10:51.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.862+0000 7f1bfd400640 1 -- 172.21.15.6:0/4001713222 >> v1:172.21.15.92:6789/0 conn(0x7f1c0015b0e0 legacy=0x7f1c00164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:51.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.862+0000 7f1bfd400640 1 -- 172.21.15.6:0/4001713222 >> v1:172.21.15.6:6789/0 conn(0x7f1c000ca520 legacy=0x7f1c000cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:51.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.862+0000 7f1bfd400640 1 -- 172.21.15.6:0/4001713222 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0006cd60 con 0x7f1c000ccad0 2024-03-22T13:10:51.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.865+0000 7f1bfd400640 1 --2- 172.21.15.6:0/4001713222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c000ccad0 0x7f1c0006d840 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1be80097a0 tx=0x7f1be802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806b8a4526d99637 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.873+0000 7f1bfca00640 1 -- 172.21.15.6:0/4001713222 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8035020 con 0x7f1c000ccad0 2024-03-22T13:10:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.873+0000 7f1bfca00640 1 -- 172.21.15.6:0/4001713222 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be8031830 con 0x7f1c000ccad0 2024-03-22T13:10:51.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.873+0000 7f1bfca00640 1 -- 172.21.15.6:0/4001713222 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8031c00 con 0x7f1c000ccad0 2024-03-22T13:10:51.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.873+0000 7f1c04a00640 1 -- 172.21.15.6:0/4001713222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c000ccad0 msgr2=0x7f1c0006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:51.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.873+0000 7f1c04a00640 1 --2- 172.21.15.6:0/4001713222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c000ccad0 0x7f1c0006d840 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1be80097a0 tx=0x7f1be802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:10:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.874+0000 7f1c04a00640 1 -- 172.21.15.6:0/4001713222 shutdown_connections 2024-03-22T13:10:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.874+0000 7f1c04a00640 1 --2- 172.21.15.6:0/4001713222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c000ccad0 0x7f1c0006d840 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.874+0000 7f1c04a00640 1 -- 172.21.15.6:0/4001713222 >> 172.21.15.6:0/4001713222 conn(0x7f1c000b16b0 msgr2=0x7f1c000b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.874+0000 7f1c04a00640 1 -- 172.21.15.6:0/4001713222 shutdown_connections 2024-03-22T13:10:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.874+0000 7f1c04a00640 1 -- 172.21.15.6:0/4001713222 wait complete. 2024-03-22T13:10:51.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.876+0000 7f1c04a00640 1 Processor -- start 2024-03-22T13:10:51.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.876+0000 7f1c04a00640 1 -- start start 2024-03-22T13:10:51.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.877+0000 7f1c04a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c000ca520 0x7f1c0015c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:51.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.877+0000 7f1c04a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0015b0e0 0x7f1c00151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:51.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.877+0000 7f1c04a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c001e4a90 0x7f1c00152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:51.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.877+0000 7f1c04a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00154c20 con 0x7f1c0015b0e0 2024-03-22T13:10:51.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.877+0000 7f1c04a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0006d320 con 0x7f1c001e4a90 2024-03-22T13:10:51.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.877+0000 7f1c04a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0015b980 con 0x7f1c000ca520 2024-03-22T13:10:51.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.878+0000 7f1bfe800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c001e4a90 0x7f1c00152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:51.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.878+0000 7f1bfd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0015b0e0 0x7f1c00151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:51.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.878+0000 7f1bfde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c000ca520 0x7f1c0015c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:51.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.879+0000 7f1bfe800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c001e4a90 0x7f1c00152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:55104/0 (socket says 172.21.15.6:55104) 2024-03-22T13:10:51.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.879+0000 7f1bfe800640 1 -- 172.21.15.6:0/851674729 learned_addr learned my addr 172.21.15.6:0/851674729 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.883+0000 7f1bfde00640 1 -- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c001e4a90 msgr2=0x7f1c00152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.883+0000 7f1bfde00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c001e4a90 0x7f1c00152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.883+0000 7f1bfde00640 1 -- 172.21.15.6:0/851674729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0015b0e0 msgr2=0x7f1c00151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.883+0000 7f1bfde00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0015b0e0 0x7f1c00151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.883+0000 7f1bfde00640 1 -- 172.21.15.6:0/851674729 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c001f06e0 con 0x7f1c000ca520 2024-03-22T13:10:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.886+0000 7f1bfde00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c000ca520 0x7f1c0015c020 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000c700 tx=0x7f1bf000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.889+0000 7f1bf6000640 1 -- 172.21.15.6:0/851674729 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0017070 con 0x7f1c000ca520 2024-03-22T13:10:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.889+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c001f09a0 con 0x7f1c000ca520 2024-03-22T13:10:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.889+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c001f0f60 con 0x7f1c000ca520 2024-03-22T13:10:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.890+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c000756c0 con 0x7f1c000ca520 2024-03-22T13:10:51.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.916+0000 7f1bf6000640 1 -- 172.21.15.6:0/851674729 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf000ce10 con 0x7f1c000ca520 2024-03-22T13:10:51.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.916+0000 7f1bf6000640 1 -- 172.21.15.6:0/851674729 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0021060 con 0x7f1c000ca520 2024-03-22T13:10:51.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.918+0000 7f1bf6000640 1 -- 172.21.15.6:0/851674729 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1bf001a070 con 0x7f1c000ca520 2024-03-22T13:10:51.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.919+0000 7f1bf6000640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1bdc06bc00 0x7f1bdc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:51.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.920+0000 7f1bfd400640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1bdc06bc00 0x7f1bdc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:51.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:51.921+0000 7f1bfd400640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1bdc06bc00 0x7f1bdc06e0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1be8036000 tx=0x7f1be8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:52.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.215+0000 7f1bf6000640 1 -- 172.21.15.6:0/851674729 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1bf0056d70 con 0x7f1c000ca520 2024-03-22T13:10:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.234+0000 7f1bf6000640 1 -- 172.21.15.6:0/851674729 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1bf0058cd0 con 0x7f1c000ca520 2024-03-22T13:10:52.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.488+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c000cb080 con 0x7f1c000ca520 2024-03-22T13:10:52.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.549+0000 7f1bf6000640 1 -- 172.21.15.6:0/851674729 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f1bf005cd70 con 0x7f1c000ca520 2024-03-22T13:10:52.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:52.549 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.552+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1bdc06bc00 msgr2=0x7f1bdc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.552+0000 7f1c04a00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1bdc06bc00 0x7f1bdc06e0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1be8036000 tx=0x7f1be8008030 comp rx=0 tx=0).stop 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.552+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c000ca520 msgr2=0x7f1c0015c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.552+0000 7f1c04a00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c000ca520 0x7f1c0015c020 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000c700 tx=0x7f1bf000cbc0 comp rx=0 tx=0).stop 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 shutdown_connections 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1bdc06bc00 0x7f1bdc06e0c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c001e4a90 0x7f1c00152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c0015b0e0 0x7f1c00151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 --2- 172.21.15.6:0/851674729 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c000ca520 0x7f1c0015c020 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 >> 172.21.15.6:0/851674729 conn(0x7f1c000b16b0 msgr2=0x7f1c000b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:52.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 shutdown_connections 2024-03-22T13:10:52.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:52.553+0000 7f1c04a00640 1 -- 172.21.15.6:0/851674729 wait complete. 2024-03-22T13:10:53.182 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:53.183 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:54.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:54.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.387+0000 7f4d9c000640 1 Processor -- start 2024-03-22T13:10:54.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.387+0000 7f4d9c000640 1 -- start start 2024-03-22T13:10:54.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.388+0000 7f4d9c000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:54.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.389+0000 7f4d9c000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4d9415b0e0 0x7f4d9415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:54.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.389+0000 7f4d9c000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9406c7a0 con 0x7f4d940ca520 2024-03-22T13:10:54.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.389+0000 7f4d9c000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d940cd000 con 0x7f4d9415b0e0 2024-03-22T13:10:54.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.389+0000 7f4d9c000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9406ca80 con 0x7f4d940ccad0 2024-03-22T13:10:54.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.389+0000 7f4d99600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4d940ca520 0x7f4d940cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52644/0 (socket says 172.21.15.6:52644) 2024-03-22T13:10:54.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.389+0000 7f4d99600640 1 -- 172.21.15.6:0/2816157270 learned_addr learned my addr 172.21.15.6:0/2816157270 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:54.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.391+0000 7f4d9a000640 1 --2- 172.21.15.6:0/2816157270 >> v2:172.21.15.92:3300/0 conn(0x7f4d9415b0e0 0x7f4d9415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:54.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.392+0000 7f4d98c00640 1 --2- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:54.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.393+0000 7f4d93e00640 1 -- 172.21.15.6:0/2816157270 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582516294 0 0) 0x7f4d9406c7a0 con 0x7f4d940ca520 2024-03-22T13:10:54.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.393+0000 7f4d93e00640 1 -- 172.21.15.6:0/2816157270 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d7c003520 con 0x7f4d940ca520 2024-03-22T13:10:54.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.395+0000 7f4d9a000640 1 -- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 msgr2=0x7f4d9406d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:54.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.395+0000 7f4d9a000640 1 --2- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9406d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:54.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.395+0000 7f4d9a000640 1 -- 172.21.15.6:0/2816157270 >> v1:172.21.15.6:6789/0 conn(0x7f4d940ca520 legacy=0x7f4d940cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:54.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.395+0000 7f4d9a000640 1 -- 172.21.15.6:0/2816157270 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9406cd60 con 0x7f4d9415b0e0 2024-03-22T13:10:54.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.398+0000 7f4d9a000640 1 --2- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d9415b0e0 0x7f4d9415b980 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009d90 tx=0x7f4d8c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d1e848c48a7feb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.403+0000 7f4d93e00640 1 -- 172.21.15.6:0/2816157270 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c039070 con 0x7f4d9415b0e0 2024-03-22T13:10:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.403+0000 7f4d93e00640 1 -- 172.21.15.6:0/2816157270 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c02cc60 con 0x7f4d9415b0e0 2024-03-22T13:10:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.403+0000 7f4d93e00640 1 -- 172.21.15.6:0/2816157270 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c034660 con 0x7f4d9415b0e0 2024-03-22T13:10:54.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 -- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d9415b0e0 msgr2=0x7f4d9415b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:54.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d9415b0e0 0x7f4d9415b980 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009d90 tx=0x7f4d8c02beb0 comp rx=0 tx=0).stop 2024-03-22T13:10:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 -- 172.21.15.6:0/2816157270 shutdown_connections 2024-03-22T13:10:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d9415b0e0 0x7f4d9415b980 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2816157270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9406d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 -- 172.21.15.6:0/2816157270 >> 172.21.15.6:0/2816157270 conn(0x7f4d940b16b0 msgr2=0x7f4d940b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 -- 172.21.15.6:0/2816157270 shutdown_connections 2024-03-22T13:10:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.404+0000 7f4d9c000640 1 -- 172.21.15.6:0/2816157270 wait complete. 2024-03-22T13:10:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.407+0000 7f4d9c000640 1 Processor -- start 2024-03-22T13:10:54.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.407+0000 7f4d9c000640 1 -- start start 2024-03-22T13:10:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.409+0000 7f4d9c000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d940ca520 0x7f4d940c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.409+0000 7f4d9c000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9415a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.409+0000 7f4d9c000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9415b0e0 0x7f4d9415b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.409+0000 7f4d9c000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d94154c20 con 0x7f4d9415b0e0 2024-03-22T13:10:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.409+0000 7f4d9c000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9406d320 con 0x7f4d940ca520 2024-03-22T13:10:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.409+0000 7f4d9c000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940ca900 con 0x7f4d940ccad0 2024-03-22T13:10:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.409+0000 7f4d98c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9415a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.410+0000 7f4d99600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d940ca520 0x7f4d940c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.410+0000 7f4d99600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d940ca520 0x7f4d940c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:33314/0 (socket says 172.21.15.6:33314) 2024-03-22T13:10:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.410+0000 7f4d99600640 1 -- 172.21.15.6:0/2326625049 learned_addr learned my addr 172.21.15.6:0/2326625049 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.414+0000 7f4d99600640 1 -- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 msgr2=0x7f4d9415a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.414+0000 7f4d99600640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9415a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.414+0000 7f4d99600640 1 -- 172.21.15.6:0/2326625049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9415b0e0 msgr2=0x7f4d9415b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.414+0000 7f4d99600640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9415b0e0 0x7f4d9415b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:54.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.414+0000 7f4d99600640 1 -- 172.21.15.6:0/2326625049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d941f1550 con 0x7f4d940ca520 2024-03-22T13:10:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.416+0000 7f4d99600640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d940ca520 0x7f4d940c86d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800c430 tx=0x7f4d8800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.445+0000 7f4d91600640 1 -- 172.21.15.6:0/2326625049 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88017070 con 0x7f4d940ca520 2024-03-22T13:10:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.445+0000 7f4d91600640 1 -- 172.21.15.6:0/2326625049 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d88005ce0 con 0x7f4d940ca520 2024-03-22T13:10:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.445+0000 7f4d91600640 1 -- 172.21.15.6:0/2326625049 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88007310 con 0x7f4d940ca520 2024-03-22T13:10:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.445+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d940b53c0 con 0x7f4d940ca520 2024-03-22T13:10:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.446+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d941f0a70 con 0x7f4d940ca520 2024-03-22T13:10:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.447+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9406c7a0 con 0x7f4d940ca520 2024-03-22T13:10:54.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.473+0000 7f4d91600640 1 -- 172.21.15.6:0/2326625049 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d8801a070 con 0x7f4d940ca520 2024-03-22T13:10:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.475+0000 7f4d91600640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d7406bc20 0x7f4d7406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.475+0000 7f4d98c00640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d7406bc20 0x7f4d7406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.476+0000 7f4d98c00640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d7406bc20 0x7f4d7406e0e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4d940b4c80 tx=0x7f4d80009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:54.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.761+0000 7f4d91600640 1 -- 172.21.15.6:0/2326625049 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4d880567d0 con 0x7f4d940ca520 2024-03-22T13:10:54.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:54.780+0000 7f4d91600640 1 -- 172.21.15.6:0/2326625049 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4d9406c7a0 con 0x7f4d940ca520 2024-03-22T13:10:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.034+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d940b5910 con 0x7f4d940ca520 2024-03-22T13:10:55.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.058+0000 7f4d91600640 1 -- 172.21.15.6:0/2326625049 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f4d940b5910 con 0x7f4d940ca520 2024-03-22T13:10:55.058 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:55.058 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:55.058 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.061+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d7406bc20 msgr2=0x7f4d7406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.061+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d7406bc20 0x7f4d7406e0e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4d940b4c80 tx=0x7f4d80009190 comp rx=0 tx=0).stop 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d940ca520 msgr2=0x7f4d940c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d940ca520 0x7f4d940c86d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800c430 tx=0x7f4d8800c8f0 comp rx=0 tx=0).stop 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 shutdown_connections 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d9415b0e0 0x7f4d9415b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d940ccad0 0x7f4d9415a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d7406bc20 0x7f4d7406e0e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 --2- 172.21.15.6:0/2326625049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d940ca520 0x7f4d940c86d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 >> 172.21.15.6:0/2326625049 conn(0x7f4d940b16b0 msgr2=0x7f4d940b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 shutdown_connections 2024-03-22T13:10:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:55.062+0000 7f4d9c000640 1 -- 172.21.15.6:0/2326625049 wait complete. 2024-03-22T13:10:55.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:55.682 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:56.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:56.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.885+0000 7fa321800640 1 Processor -- start 2024-03-22T13:10:56.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.885+0000 7fa321800640 1 -- start start 2024-03-22T13:10:56.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.886+0000 7fa321800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa31c0ca520 0x7fa31c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:56.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.886+0000 7fa321800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c15b0e0 0x7fa31c15b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.886+0000 7fa321800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31c06c7a0 con 0x7fa31c0ca900 2024-03-22T13:10:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.886+0000 7fa321800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31c0cd000 con 0x7fa31c0ca520 2024-03-22T13:10:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.886+0000 7fa321800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c06ca80 con 0x7fa31c15b0e0 2024-03-22T13:10:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.887+0000 7fa312000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c15b0e0 0x7fa31c15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:56.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.887+0000 7fa31b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa31c0ca900 0x7fa31c0cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52656/0 (socket says 172.21.15.6:52656) 2024-03-22T13:10:56.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.887+0000 7fa31b400640 1 -- 172.21.15.6:0/3473490936 learned_addr learned my addr 172.21.15.6:0/3473490936 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:56.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.888+0000 7fa31aa00640 1 --2- 172.21.15.6:0/3473490936 >> v2:172.21.15.92:3300/0 conn(0x7fa31c0ca520 0x7fa31c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:56.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.890+0000 7fa312000640 1 -- 172.21.15.6:0/3473490936 >> v2:172.21.15.92:3300/0 conn(0x7fa31c0ca520 msgr2=0x7fa31c06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:56.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.890+0000 7fa312000640 1 --2- 172.21.15.6:0/3473490936 >> v2:172.21.15.92:3300/0 conn(0x7fa31c0ca520 0x7fa31c06d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:56.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.891+0000 7fa312000640 1 -- 172.21.15.6:0/3473490936 >> v1:172.21.15.6:6789/0 conn(0x7fa31c0ca900 legacy=0x7fa31c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:56.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.891+0000 7fa312000640 1 -- 172.21.15.6:0/3473490936 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c06cd60 con 0x7fa31c15b0e0 2024-03-22T13:10:56.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.893+0000 7fa312000640 1 --2- 172.21.15.6:0/3473490936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c15b0e0 0x7fa31c15b980 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa3040097a0 tx=0x7fa30402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4651e6da3d03e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:56.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.896+0000 7fa31a000640 1 -- 172.21.15.6:0/3473490936 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304035020 con 0x7fa31c15b0e0 2024-03-22T13:10:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.898+0000 7fa31a000640 1 -- 172.21.15.6:0/3473490936 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa304031830 con 0x7fa31c15b0e0 2024-03-22T13:10:56.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.898+0000 7fa31a000640 1 -- 172.21.15.6:0/3473490936 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa304031c00 con 0x7fa31c15b0e0 2024-03-22T13:10:56.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.898+0000 7fa321800640 1 -- 172.21.15.6:0/3473490936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c15b0e0 msgr2=0x7fa31c15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.898+0000 7fa321800640 1 --2- 172.21.15.6:0/3473490936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c15b0e0 0x7fa31c15b980 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa3040097a0 tx=0x7fa30402b8d0 comp rx=0 tx=0).stop 2024-03-22T13:10:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.899+0000 7fa321800640 1 -- 172.21.15.6:0/3473490936 shutdown_connections 2024-03-22T13:10:56.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.899+0000 7fa321800640 1 --2- 172.21.15.6:0/3473490936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c15b0e0 0x7fa31c15b980 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:56.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.899+0000 7fa321800640 1 --2- 172.21.15.6:0/3473490936 >> v2:172.21.15.92:3300/0 conn(0x7fa31c0ca520 0x7fa31c06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:56.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.899+0000 7fa321800640 1 -- 172.21.15.6:0/3473490936 >> 172.21.15.6:0/3473490936 conn(0x7fa31c0b16b0 msgr2=0x7fa31c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:56.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.899+0000 7fa321800640 1 -- 172.21.15.6:0/3473490936 shutdown_connections 2024-03-22T13:10:56.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.899+0000 7fa321800640 1 -- 172.21.15.6:0/3473490936 wait complete. 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.901+0000 7fa321800640 1 Processor -- start 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.901+0000 7fa321800640 1 -- start start 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.902+0000 7fa321800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa31c0ca520 0x7fa31c1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.902+0000 7fa321800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c0ca900 0x7fa31c1f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.902+0000 7fa321800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa31c15b0e0 0x7fa31c155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.902+0000 7fa321800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0cad60 con 0x7fa31c15b0e0 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.902+0000 7fa321800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c06d320 con 0x7fa31c0ca520 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.902+0000 7fa321800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c154c20 con 0x7fa31c0ca900 2024-03-22T13:10:56.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.903+0000 7fa312000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c0ca900 0x7fa31c1f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.903+0000 7fa31aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa31c0ca520 0x7fa31c1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.903+0000 7fa31b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa31c15b0e0 0x7fa31c155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.903+0000 7fa31aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa31c0ca520 0x7fa31c1f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:33346/0 (socket says 172.21.15.6:33346) 2024-03-22T13:10:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.903+0000 7fa31aa00640 1 -- 172.21.15.6:0/1226195074 learned_addr learned my addr 172.21.15.6:0/1226195074 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:56.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.907+0000 7fa31aa00640 1 -- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c0ca900 msgr2=0x7fa31c1f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:56.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.907+0000 7fa31aa00640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c0ca900 0x7fa31c1f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:56.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.907+0000 7fa31aa00640 1 -- 172.21.15.6:0/1226195074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa31c15b0e0 msgr2=0x7fa31c155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:56.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.907+0000 7fa31aa00640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa31c15b0e0 0x7fa31c155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:56.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.907+0000 7fa31aa00640 1 -- 172.21.15.6:0/1226195074 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c157550 con 0x7fa31c0ca520 2024-03-22T13:10:56.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.910+0000 7fa31aa00640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa31c0ca520 0x7fa31c1f3a70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00e8a0 tx=0x7fa30c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:56.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.914+0000 7fa313400640 1 -- 172.21.15.6:0/1226195074 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c017070 con 0x7fa31c0ca520 2024-03-22T13:10:56.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.914+0000 7fa313400640 1 -- 172.21.15.6:0/1226195074 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00c940 con 0x7fa31c0ca520 2024-03-22T13:10:56.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.914+0000 7fa313400640 1 -- 172.21.15.6:0/1226195074 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00cd30 con 0x7fa31c0ca520 2024-03-22T13:10:56.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.914+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c157830 con 0x7fa31c0ca520 2024-03-22T13:10:56.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.915+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c155200 con 0x7fa31c0ca520 2024-03-22T13:10:56.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:56.916+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c0756c0 con 0x7fa31c0ca520 2024-03-22T13:10:57.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.188+0000 7fa313400640 1 -- 172.21.15.6:0/1226195074 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa30c01a070 con 0x7fa31c0ca520 2024-03-22T13:10:57.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.190+0000 7fa313400640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa2f806bc20 0x7fa2f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:57.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.191+0000 7fa312000640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa2f806bc20 0x7fa2f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:57.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.192+0000 7fa312000640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa2f806bc20 0x7fa2f806e0e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa304036000 tx=0x7fa30403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:57.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.194+0000 7fa313400640 1 -- 172.21.15.6:0/1226195074 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fa30c0575a0 con 0x7fa31c0ca520 2024-03-22T13:10:57.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.211+0000 7fa313400640 1 -- 172.21.15.6:0/1226195074 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa30c059500 con 0x7fa31c0ca520 2024-03-22T13:10:57.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.464+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa31c0cb080 con 0x7fa31c0ca520 2024-03-22T13:10:57.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.488+0000 7fa313400640 1 -- 172.21.15.6:0/1226195074 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fa30c05d5a0 con 0x7fa31c0ca520 2024-03-22T13:10:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:57.489 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:57.489 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:10:57.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa2f806bc20 msgr2=0x7fa2f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:57.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa2f806bc20 0x7fa2f806e0e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa304036000 tx=0x7fa30403c040 comp rx=0 tx=0).stop 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa31c0ca520 msgr2=0x7fa31c1f3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa31c0ca520 0x7fa31c1f3a70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00e8a0 tx=0x7fa30c00ed60 comp rx=0 tx=0).stop 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 shutdown_connections 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa31c15b0e0 0x7fa31c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa31c0ca900 0x7fa31c1f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fa2f806bc20 0x7fa2f806e0e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 --2- 172.21.15.6:0/1226195074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa31c0ca520 0x7fa31c1f3a70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.491+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 >> 172.21.15.6:0/1226195074 conn(0x7fa31c0b16b0 msgr2=0x7fa31c153db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.492+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 shutdown_connections 2024-03-22T13:10:57.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:57.492+0000 7fa321800640 1 -- 172.21.15.6:0/1226195074 wait complete. 2024-03-22T13:10:58.112 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:10:58.112 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:10:59.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:10:59.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.305+0000 7f4c46e00640 1 Processor -- start 2024-03-22T13:10:59.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.305+0000 7f4c46e00640 1 -- start start 2024-03-22T13:10:59.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.306+0000 7f4c46e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400cc380 0x7f4c4006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:59.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.306+0000 7f4c46e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4c400ce8a0 0x7f4c4015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:59.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.306+0000 7f4c46e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c400cee00 con 0x7f4c400cc760 2024-03-22T13:10:59.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.306+0000 7f4c46e00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c400cf0c0 con 0x7f4c400ce8a0 2024-03-22T13:10:59.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.306+0000 7f4c46e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400cf3a0 con 0x7f4c400cc380 2024-03-22T13:10:59.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.307+0000 7f4c45a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400cc380 0x7f4c4006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:59.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.307+0000 7f4c45000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4c400ce8a0 0x7f4c4015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:59.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.308+0000 7f4c46400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4c400cc760 0x7f4c400cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52662/0 (socket says 172.21.15.6:52662) 2024-03-22T13:10:59.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.308+0000 7f4c46400640 1 -- 172.21.15.6:0/3210621917 learned_addr learned my addr 172.21.15.6:0/3210621917 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:10:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.310+0000 7f4c45000640 1 -- 172.21.15.6:0/3210621917 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400cc380 msgr2=0x7f4c4006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:10:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.310+0000 7f4c45000640 1 --2- 172.21.15.6:0/3210621917 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400cc380 0x7f4c4006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.310+0000 7f4c45000640 1 -- 172.21.15.6:0/3210621917 >> v1:172.21.15.6:6789/0 conn(0x7f4c400cc760 legacy=0x7f4c400cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.310+0000 7f4c45000640 1 -- 172.21.15.6:0/3210621917 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400cf680 con 0x7f4c400ce8a0 2024-03-22T13:10:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.312+0000 7f4c45000640 1 --2- 172.21.15.6:0/3210621917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400ce8a0 0x7f4c4015ae30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0097a0 tx=0x7f4c2c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bbee67f1522f201 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:59.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.318+0000 7f4c3fe00640 1 -- 172.21.15.6:0/3210621917 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c035020 con 0x7f4c400ce8a0 2024-03-22T13:10:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.318+0000 7f4c3fe00640 1 -- 172.21.15.6:0/3210621917 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c031810 con 0x7f4c400ce8a0 2024-03-22T13:10:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.318+0000 7f4c3fe00640 1 -- 172.21.15.6:0/3210621917 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c031be0 con 0x7f4c400ce8a0 2024-03-22T13:10:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.318+0000 7f4c46e00640 1 -- 172.21.15.6:0/3210621917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400ce8a0 msgr2=0x7f4c4015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.318+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3210621917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400ce8a0 0x7f4c4015ae30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0097a0 tx=0x7f4c2c02b8b0 comp rx=0 tx=0).stop 2024-03-22T13:10:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.319+0000 7f4c46e00640 1 -- 172.21.15.6:0/3210621917 shutdown_connections 2024-03-22T13:10:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.319+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3210621917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400ce8a0 0x7f4c4015ae30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.319+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3210621917 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400cc380 0x7f4c4006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.319+0000 7f4c46e00640 1 -- 172.21.15.6:0/3210621917 >> 172.21.15.6:0/3210621917 conn(0x7f4c400b1480 msgr2=0x7f4c400b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:10:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.319+0000 7f4c46e00640 1 -- 172.21.15.6:0/3210621917 shutdown_connections 2024-03-22T13:10:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.319+0000 7f4c46e00640 1 -- 172.21.15.6:0/3210621917 wait complete. 2024-03-22T13:10:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.320+0000 7f4c46e00640 1 Processor -- start 2024-03-22T13:10:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.321+0000 7f4c46e00640 1 -- start start 2024-03-22T13:10:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c46e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400cc380 0x7f4c401f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c46e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c400cc760 0x7f4c400c3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c46e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400ce8a0 0x7f4c4015d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c46e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401549d0 con 0x7f4c400cc760 2024-03-22T13:10:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c46e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4015a910 con 0x7f4c400cc380 2024-03-22T13:10:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c46e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400ccb90 con 0x7f4c400ce8a0 2024-03-22T13:10:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c45000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c400cc760 0x7f4c400c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.322+0000 7f4c46400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400ce8a0 0x7f4c4015d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.323+0000 7f4c45000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c400cc760 0x7f4c400c3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47444/0 (socket says 172.21.15.6:47444) 2024-03-22T13:10:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.323+0000 7f4c45000640 1 -- 172.21.15.6:0/3762354797 learned_addr learned my addr 172.21.15.6:0/3762354797 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:10:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.324+0000 7f4c45a00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400cc380 0x7f4c401f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.327+0000 7f4c45000640 1 -- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400ce8a0 msgr2=0x7f4c4015d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.327+0000 7f4c45000640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400ce8a0 0x7f4c4015d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.327+0000 7f4c45000640 1 -- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400cc380 msgr2=0x7f4c401f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:10:59.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.327+0000 7f4c45000640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400cc380 0x7f4c401f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:10:59.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.327+0000 7f4c45000640 1 -- 172.21.15.6:0/3762354797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400c5170 con 0x7f4c400cc760 2024-03-22T13:10:59.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.330+0000 7f4c45000640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c400cc760 0x7f4c400c3710 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c02bec0 tx=0x7f4c2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.333+0000 7f4c3d600640 1 -- 172.21.15.6:0/3762354797 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c041070 con 0x7f4c400cc760 2024-03-22T13:10:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.333+0000 7f4c3d600640 1 -- 172.21.15.6:0/3762354797 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c03ad00 con 0x7f4c400cc760 2024-03-22T13:10:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.333+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c400c4310 con 0x7f4c400cc760 2024-03-22T13:10:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.334+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c400c48d0 con 0x7f4c400cc760 2024-03-22T13:10:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.334+0000 7f4c3d600640 1 -- 172.21.15.6:0/3762354797 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c035040 con 0x7f4c400cc760 2024-03-22T13:10:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.335+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c005290 con 0x7f4c400cc760 2024-03-22T13:10:59.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.363+0000 7f4c3d600640 1 -- 172.21.15.6:0/3762354797 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c2c04c060 con 0x7f4c400cc760 2024-03-22T13:10:59.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.363+0000 7f4c3d600640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4c2006bc20 0x7f4c2006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:10:59.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.364+0000 7f4c45a00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4c2006bc20 0x7f4c2006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:10:59.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.365+0000 7f4c45a00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4c2006bc20 0x7f4c2006e0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4c34009aa0 tx=0x7f4c34009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:10:59.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.370+0000 7f4c3d600640 1 -- 172.21.15.6:0/3762354797 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4c2c078e40 con 0x7f4c400cc760 2024-03-22T13:10:59.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.666+0000 7f4c3d600640 1 -- 172.21.15.6:0/3762354797 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4c2c07ada0 con 0x7f4c400cc760 2024-03-22T13:10:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.928+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c1c005580 con 0x7f4c400cc760 2024-03-22T13:10:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:10:59.998+0000 7f4c3d600640 1 -- 172.21.15.6:0/3762354797 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f4c2c07ee40 con 0x7f4c400cc760 2024-03-22T13:10:59.999 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:10:59.999 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:10:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4c2006bc20 msgr2=0x7f4c2006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4c2006bc20 0x7f4c2006e0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4c34009aa0 tx=0x7f4c34009190 comp rx=0 tx=0).stop 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c400cc760 msgr2=0x7f4c400c3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c400cc760 0x7f4c400c3710 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c02bec0 tx=0x7f4c2c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 shutdown_connections 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c400ce8a0 0x7f4c4015d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c400cc760 0x7f4c400c3710 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4c2006bc20 0x7f4c2006e0e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 --2- 172.21.15.6:0/3762354797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c400cc380 0x7f4c401f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 >> 172.21.15.6:0/3762354797 conn(0x7f4c400b1480 msgr2=0x7f4c400cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.002+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 shutdown_connections 2024-03-22T13:11:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:00.003+0000 7f4c46e00640 1 -- 172.21.15.6:0/3762354797 wait complete. 2024-03-22T13:11:00.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:00.633 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:01.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:01.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.841+0000 7f1cf6600640 1 Processor -- start 2024-03-22T13:11:01.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.842+0000 7f1cf6600640 1 -- start start 2024-03-22T13:11:01.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.843+0000 7f1cf6600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf006f8c0 0x7f1cf00716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:01.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.843+0000 7f1cf6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf01590b0 con 0x7f1cf0071c10 2024-03-22T13:11:01.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.843+0000 7f1cf6600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf00721f0 con 0x7f1cf0152400 2024-03-22T13:11:01.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.843+0000 7f1cf6600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf00724d0 con 0x7f1cf006f8c0 2024-03-22T13:11:01.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.844+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf006f8c0 0x7f1cf00716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:01.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.844+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf006f8c0 0x7f1cf00716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:37134/0 (socket says 172.21.15.6:37134) 2024-03-22T13:11:01.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.844+0000 7f1ceea00640 1 -- 172.21.15.6:0/1552815136 learned_addr learned my addr 172.21.15.6:0/1552815136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:01.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.847+0000 7f1ceea00640 1 -- 172.21.15.6:0/1552815136 >> v1:172.21.15.92:6789/0 conn(0x7f1cf0152400 legacy=0x7f1cf015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:01.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.847+0000 7f1ceea00640 1 -- 172.21.15.6:0/1552815136 >> v1:172.21.15.6:6789/0 conn(0x7f1cf0071c10 legacy=0x7f1cf00c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:01.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.847+0000 7f1ceea00640 1 -- 172.21.15.6:0/1552815136 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf01582f0 con 0x7f1cf006f8c0 2024-03-22T13:11:01.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.849+0000 7f1ceea00640 1 --2- 172.21.15.6:0/1552815136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf006f8c0 0x7f1cf00716d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0072650 tx=0x7f1cdc02ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34b6f1525a5fb2b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:01.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.854+0000 7f1cee000640 1 -- 172.21.15.6:0/1552815136 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc02d850 con 0x7f1cf006f8c0 2024-03-22T13:11:01.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.854+0000 7f1cee000640 1 -- 172.21.15.6:0/1552815136 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc02d9f0 con 0x7f1cf006f8c0 2024-03-22T13:11:01.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.854+0000 7f1cee000640 1 -- 172.21.15.6:0/1552815136 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc02ddc0 con 0x7f1cf006f8c0 2024-03-22T13:11:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.855+0000 7f1cf6600640 1 -- 172.21.15.6:0/1552815136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf006f8c0 msgr2=0x7f1cf00716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.855+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1552815136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf006f8c0 0x7f1cf00716d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0072650 tx=0x7f1cdc02ba90 comp rx=0 tx=0).stop 2024-03-22T13:11:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.855+0000 7f1cf6600640 1 -- 172.21.15.6:0/1552815136 shutdown_connections 2024-03-22T13:11:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.855+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1552815136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf006f8c0 0x7f1cf00716d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.855+0000 7f1cf6600640 1 -- 172.21.15.6:0/1552815136 >> 172.21.15.6:0/1552815136 conn(0x7f1cf00cb930 msgr2=0x7f1cf00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.855+0000 7f1cf6600640 1 -- 172.21.15.6:0/1552815136 shutdown_connections 2024-03-22T13:11:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.855+0000 7f1cf6600640 1 -- 172.21.15.6:0/1552815136 wait complete. 2024-03-22T13:11:01.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.857+0000 7f1cf6600640 1 Processor -- start 2024-03-22T13:11:01.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.857+0000 7f1cf6600640 1 -- start start 2024-03-22T13:11:01.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.858+0000 7f1cf6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf006f8c0 0x7f1cf014bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:01.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.858+0000 7f1cf6600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf0071c10 0x7f1cf00c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:01.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.858+0000 7f1cf6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0152400 0x7f1cf00ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:01.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.858+0000 7f1cf6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf0158b90 con 0x7f1cf0152400 2024-03-22T13:11:01.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.858+0000 7f1cf6600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf01588b0 con 0x7f1cf006f8c0 2024-03-22T13:11:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.858+0000 7f1cf6600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf01527e0 con 0x7f1cf0071c10 2024-03-22T13:11:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.859+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf0071c10 0x7f1cf00c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.859+0000 7f1cef400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf006f8c0 0x7f1cf014bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.859+0000 7f1ceea00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf0071c10 0x7f1cf00c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:37148/0 (socket says 172.21.15.6:37148) 2024-03-22T13:11:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.859+0000 7f1ceea00640 1 -- 172.21.15.6:0/1793032443 learned_addr learned my addr 172.21.15.6:0/1793032443 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.859+0000 7f1cefe00640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0152400 0x7f1cf00ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:01.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.863+0000 7f1ceea00640 1 -- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf006f8c0 msgr2=0x7f1cf014bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:01.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.863+0000 7f1ceea00640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf006f8c0 0x7f1cf014bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:01.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.863+0000 7f1ceea00640 1 -- 172.21.15.6:0/1793032443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0152400 msgr2=0x7f1cf00ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:01.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.863+0000 7f1ceea00640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0152400 0x7f1cf00ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:01.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.863+0000 7f1ceea00640 1 -- 172.21.15.6:0/1793032443 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf00c9fc0 con 0x7f1cf0071c10 2024-03-22T13:11:01.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.866+0000 7f1ceea00640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf0071c10 0x7f1cf00c9730 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009990 tx=0x7f1cdc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.871+0000 7f1ce7400640 1 -- 172.21.15.6:0/1793032443 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc047070 con 0x7f1cf0071c10 2024-03-22T13:11:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.871+0000 7f1ce7400640 1 -- 172.21.15.6:0/1793032443 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc037a40 con 0x7f1cf0071c10 2024-03-22T13:11:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.871+0000 7f1ce7400640 1 -- 172.21.15.6:0/1793032443 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc037e10 con 0x7f1cf0071c10 2024-03-22T13:11:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.871+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf01f1920 con 0x7f1cf0071c10 2024-03-22T13:11:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.871+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf01f1ee0 con 0x7f1cf0071c10 2024-03-22T13:11:01.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.872+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf00756c0 con 0x7f1cf0071c10 2024-03-22T13:11:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.898+0000 7f1ce7400640 1 -- 172.21.15.6:0/1793032443 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1cdc04c480 con 0x7f1cf0071c10 2024-03-22T13:11:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.899+0000 7f1ce7400640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cd006bc50 0x7f1cd006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.900+0000 7f1cef400640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cd006bc50 0x7f1cd006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:01.901+0000 7f1cef400640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cd006bc50 0x7f1cd006e110 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0000f60 tx=0x7f1ce0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:02.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.180+0000 7f1ce7400640 1 -- 172.21.15.6:0/1793032443 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1cdc078f20 con 0x7f1cf0071c10 2024-03-22T13:11:02.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.193+0000 7f1ce7400640 1 -- 172.21.15.6:0/1793032443 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1cdc07ae80 con 0x7f1cf0071c10 2024-03-22T13:11:02.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.442+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cf0140a10 con 0x7f1cf0071c10 2024-03-22T13:11:02.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.457+0000 7f1ce7400640 1 -- 172.21.15.6:0/1793032443 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f1cdc07ef20 con 0x7f1cf0071c10 2024-03-22T13:11:02.457 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:02.458 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:02.458 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:02.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cd006bc50 msgr2=0x7f1cd006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:02.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cd006bc50 0x7f1cd006e110 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0000f60 tx=0x7f1ce0009040 comp rx=0 tx=0).stop 2024-03-22T13:11:02.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf0071c10 msgr2=0x7f1cf00c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:02.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf0071c10 0x7f1cf00c9730 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009990 tx=0x7f1cdc004730 comp rx=0 tx=0).stop 2024-03-22T13:11:02.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 shutdown_connections 2024-03-22T13:11:02.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0152400 0x7f1cf00ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:02.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf0071c10 0x7f1cf00c9730 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:02.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1cd006bc50 0x7f1cd006e110 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:02.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 --2- 172.21.15.6:0/1793032443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf006f8c0 0x7f1cf014bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:02.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 >> 172.21.15.6:0/1793032443 conn(0x7f1cf00cb930 msgr2=0x7f1cf00c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:02.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 shutdown_connections 2024-03-22T13:11:02.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:02.460+0000 7f1cf6600640 1 -- 172.21.15.6:0/1793032443 wait complete. 2024-03-22T13:11:03.083 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:03.083 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:03.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:11:03.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:11:04.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.277+0000 7fd891600640 1 Processor -- start 2024-03-22T13:11:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.277+0000 7fd891600640 1 -- start start 2024-03-22T13:11:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.279+0000 7fd891600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c0ce8a0 0x7fd88c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.279+0000 7fd891600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd88c15ae30 0x7fd88c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.279+0000 7fd891600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88c15b260 con 0x7fd88c0cc380 2024-03-22T13:11:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.279+0000 7fd891600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88c0cee00 con 0x7fd88c15ae30 2024-03-22T13:11:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.279+0000 7fd891600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c0cf0c0 con 0x7fd88c0ce8a0 2024-03-22T13:11:04.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.280+0000 7fd883400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c0ce8a0 0x7fd88c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:04.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.280+0000 7fd883400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c0ce8a0 0x7fd88c06c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50450/0 (socket says 172.21.15.6:50450) 2024-03-22T13:11:04.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.280+0000 7fd883400640 1 -- 172.21.15.6:0/908816799 learned_addr learned my addr 172.21.15.6:0/908816799 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:04.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.281+0000 7fd890c00640 1 --2- 172.21.15.6:0/908816799 >> v2:172.21.15.92:3300/0 conn(0x7fd88c15ae30 0x7fd88c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.283+0000 7fd883400640 1 -- 172.21.15.6:0/908816799 >> v2:172.21.15.92:3300/0 conn(0x7fd88c15ae30 msgr2=0x7fd88c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.283+0000 7fd883400640 1 --2- 172.21.15.6:0/908816799 >> v2:172.21.15.92:3300/0 conn(0x7fd88c15ae30 0x7fd88c0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.283+0000 7fd883400640 1 -- 172.21.15.6:0/908816799 >> v1:172.21.15.6:6789/0 conn(0x7fd88c0cc380 legacy=0x7fd88c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.283+0000 7fd883400640 1 -- 172.21.15.6:0/908816799 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c0cf3a0 con 0x7fd88c0ce8a0 2024-03-22T13:11:04.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.288+0000 7fd883400640 1 --2- 172.21.15.6:0/908816799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c0ce8a0 0x7fd88c06c230 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd874009870 tx=0x7fd87402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fd20ee4b058e615 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:04.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.293+0000 7fd88b400640 1 -- 172.21.15.6:0/908816799 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874034070 con 0x7fd88c0ce8a0 2024-03-22T13:11:04.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.293+0000 7fd88b400640 1 -- 172.21.15.6:0/908816799 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd874030840 con 0x7fd88c0ce8a0 2024-03-22T13:11:04.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.293+0000 7fd88b400640 1 -- 172.21.15.6:0/908816799 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874030c10 con 0x7fd88c0ce8a0 2024-03-22T13:11:04.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 -- 172.21.15.6:0/908816799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c0ce8a0 msgr2=0x7fd88c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:04.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 --2- 172.21.15.6:0/908816799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c0ce8a0 0x7fd88c06c230 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd874009870 tx=0x7fd87402b980 comp rx=0 tx=0).stop 2024-03-22T13:11:04.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 -- 172.21.15.6:0/908816799 shutdown_connections 2024-03-22T13:11:04.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 --2- 172.21.15.6:0/908816799 >> v2:172.21.15.92:3300/0 conn(0x7fd88c15ae30 0x7fd88c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 --2- 172.21.15.6:0/908816799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c0ce8a0 0x7fd88c06c230 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 -- 172.21.15.6:0/908816799 >> 172.21.15.6:0/908816799 conn(0x7fd88c0b1480 msgr2=0x7fd88c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:04.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 -- 172.21.15.6:0/908816799 shutdown_connections 2024-03-22T13:11:04.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.294+0000 7fd891600640 1 -- 172.21.15.6:0/908816799 wait complete. 2024-03-22T13:11:04.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.297+0000 7fd891600640 1 Processor -- start 2024-03-22T13:11:04.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.297+0000 7fd891600640 1 -- start start 2024-03-22T13:11:04.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd891600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd88c0cc380 0x7fd88c1532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:04.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd891600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c0ce8a0 0x7fd88c1e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:04.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd891600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c15ae30 0x7fd88c1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:04.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd891600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c15a910 con 0x7fd88c0ce8a0 2024-03-22T13:11:04.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd891600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c15a630 con 0x7fd88c0cc380 2024-03-22T13:11:04.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd891600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c1549d0 con 0x7fd88c15ae30 2024-03-22T13:11:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd890c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c15ae30 0x7fd88c1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.299+0000 7fd88be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd88c0cc380 0x7fd88c1532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.300+0000 7fd883400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c0ce8a0 0x7fd88c1e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.300+0000 7fd890c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c15ae30 0x7fd88c1570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50452/0 (socket says 172.21.15.6:50452) 2024-03-22T13:11:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.300+0000 7fd890c00640 1 -- 172.21.15.6:0/3631441703 learned_addr learned my addr 172.21.15.6:0/3631441703 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.305+0000 7fd890c00640 1 -- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd88c0cc380 msgr2=0x7fd88c1532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.305+0000 7fd890c00640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd88c0cc380 0x7fd88c1532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.305+0000 7fd890c00640 1 -- 172.21.15.6:0/3631441703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c0ce8a0 msgr2=0x7fd88c1e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.305+0000 7fd890c00640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c0ce8a0 0x7fd88c1e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.305+0000 7fd890c00640 1 -- 172.21.15.6:0/3631441703 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c1e3650 con 0x7fd88c15ae30 2024-03-22T13:11:04.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.308+0000 7fd890c00640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c15ae30 0x7fd88c1570e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd884007e70 tx=0x7fd88400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:04.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.312+0000 7fd888c00640 1 -- 172.21.15.6:0/3631441703 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884014620 con 0x7fd88c15ae30 2024-03-22T13:11:04.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.312+0000 7fd888c00640 1 -- 172.21.15.6:0/3631441703 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884019070 con 0x7fd88c15ae30 2024-03-22T13:11:04.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.312+0000 7fd888c00640 1 -- 172.21.15.6:0/3631441703 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884014a20 con 0x7fd88c15ae30 2024-03-22T13:11:04.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.312+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c1e3930 con 0x7fd88c15ae30 2024-03-22T13:11:04.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.312+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c1568e0 con 0x7fd88c15ae30 2024-03-22T13:11:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.313+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88c075650 con 0x7fd88c15ae30 2024-03-22T13:11:04.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.343+0000 7fd888c00640 1 -- 172.21.15.6:0/3631441703 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd884014bc0 con 0x7fd88c15ae30 2024-03-22T13:11:04.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.345+0000 7fd888c00640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd86806bc20 0x7fd86806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:04.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.345+0000 7fd88be00640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd86806bc20 0x7fd86806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:04.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.346+0000 7fd88be00640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd86806bc20 0x7fd86806e0e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0016e0 tx=0x7fd87c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.347+0000 7fd888c00640 1 -- 172.21.15.6:0/3631441703 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd884058630 con 0x7fd88c15ae30 2024-03-22T13:11:04.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.635+0000 7fd888c00640 1 -- 172.21.15.6:0/3631441703 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd88405a590 con 0x7fd88c15ae30 2024-03-22T13:11:04.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.886+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd88c159670 con 0x7fd88c15ae30 2024-03-22T13:11:04.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.899+0000 7fd888c00640 1 -- 172.21.15.6:0/3631441703 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fd88405e630 con 0x7fd88c15ae30 2024-03-22T13:11:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:04.899 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:04.899 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd86806bc20 msgr2=0x7fd86806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd86806bc20 0x7fd86806e0e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0016e0 tx=0x7fd87c009040 comp rx=0 tx=0).stop 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c15ae30 msgr2=0x7fd88c1570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c15ae30 0x7fd88c1570e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd884007e70 tx=0x7fd88400e4b0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 shutdown_connections 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd88c15ae30 0x7fd88c1570e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88c0ce8a0 0x7fd88c1e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd86806bc20 0x7fd86806e0e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 --2- 172.21.15.6:0/3631441703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd88c0cc380 0x7fd88c1532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 >> 172.21.15.6:0/3631441703 conn(0x7fd88c0b1480 msgr2=0x7fd88c157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 shutdown_connections 2024-03-22T13:11:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:04.902+0000 7fd891600640 1 -- 172.21.15.6:0/3631441703 wait complete. 2024-03-22T13:11:05.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:05.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:06.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:06.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.720+0000 7f9813600640 1 Processor -- start 2024-03-22T13:11:06.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.721+0000 7f9813600640 1 -- start start 2024-03-22T13:11:06.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.722+0000 7f9813600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f980c06b080 0x7f980c06b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:06.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.722+0000 7f9813600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06fea0 0x7f980c15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:06.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.722+0000 7f9813600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f980c15de30 0x7f980c0c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:06.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.722+0000 7f9813600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c0c3d30 con 0x7f980c15de30 2024-03-22T13:11:06.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.722+0000 7f9813600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c0c3a50 con 0x7f980c06b080 2024-03-22T13:11:06.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.722+0000 7f9813600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c15a450 con 0x7f980c06fea0 2024-03-22T13:11:06.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.723+0000 7f9811600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f980c15de30 0x7f980c0c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:06.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.723+0000 7f9810c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f980c06b080 0x7f980c06b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:06.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.723+0000 7f980be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06fea0 0x7f980c15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:06.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.723+0000 7f9810c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f980c06b080 0x7f980c06b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52244/0 (socket says 172.21.15.6:52244) 2024-03-22T13:11:06.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.723+0000 7f9810c00640 1 -- 172.21.15.6:0/1346152294 learned_addr learned my addr 172.21.15.6:0/1346152294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:06.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.725+0000 7f9810c00640 1 -- 172.21.15.6:0/1346152294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06fea0 msgr2=0x7f980c15d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:06.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.725+0000 7f9810c00640 1 --2- 172.21.15.6:0/1346152294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06fea0 0x7f980c15d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.725+0000 7f9810c00640 1 -- 172.21.15.6:0/1346152294 >> v2:172.21.15.6:3300/0 conn(0x7f980c15de30 msgr2=0x7f980c0c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:06.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.725+0000 7f9810c00640 1 --2- 172.21.15.6:0/1346152294 >> v2:172.21.15.6:3300/0 conn(0x7f980c15de30 0x7f980c0c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.725+0000 7f9810c00640 1 -- 172.21.15.6:0/1346152294 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c15a730 con 0x7f980c06b080 2024-03-22T13:11:06.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.727+0000 7f9810c00640 1 --2- 172.21.15.6:0/1346152294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06b080 0x7f980c06b460 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f980c0504c0 tx=0x7f980002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=200abdccc23b26c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:06.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.730+0000 7f980b400640 1 -- 172.21.15.6:0/1346152294 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800034070 con 0x7f980c06b080 2024-03-22T13:11:06.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.732+0000 7f980b400640 1 -- 172.21.15.6:0/1346152294 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9800030860 con 0x7f980c06b080 2024-03-22T13:11:06.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.732+0000 7f980b400640 1 -- 172.21.15.6:0/1346152294 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800030c30 con 0x7f980c06b080 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 -- 172.21.15.6:0/1346152294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06b080 msgr2=0x7f980c06b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 --2- 172.21.15.6:0/1346152294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06b080 0x7f980c06b460 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f980c0504c0 tx=0x7f980002b9a0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 -- 172.21.15.6:0/1346152294 shutdown_connections 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 --2- 172.21.15.6:0/1346152294 >> v2:172.21.15.6:3300/0 conn(0x7f980c15de30 0x7f980c0c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 --2- 172.21.15.6:0/1346152294 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06fea0 0x7f980c15d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 --2- 172.21.15.6:0/1346152294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06b080 0x7f980c06b460 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 -- 172.21.15.6:0/1346152294 >> 172.21.15.6:0/1346152294 conn(0x7f980c0ccb50 msgr2=0x7f980c0ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 -- 172.21.15.6:0/1346152294 shutdown_connections 2024-03-22T13:11:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.733+0000 7f9813600640 1 -- 172.21.15.6:0/1346152294 wait complete. 2024-03-22T13:11:06.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.736+0000 7f9813600640 1 Processor -- start 2024-03-22T13:11:06.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.736+0000 7f9813600640 1 -- start start 2024-03-22T13:11:06.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9813600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06b080 0x7f980c14f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:06.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9813600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06fea0 0x7f980c1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9813600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f980c15de30 0x7f980c1f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9813600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c15b210 con 0x7f980c15de30 2024-03-22T13:11:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9813600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c15acf0 con 0x7f980c06fea0 2024-03-22T13:11:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9813600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c154930 con 0x7f980c06b080 2024-03-22T13:11:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9810c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06b080 0x7f980c14f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:06.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f9811600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f980c15de30 0x7f980c1f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:06.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.738+0000 7f980be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06fea0 0x7f980c1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:06.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.739+0000 7f9810c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06b080 0x7f980c14f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:50472/0 (socket says 172.21.15.6:50472) 2024-03-22T13:11:06.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.739+0000 7f9810c00640 1 -- 172.21.15.6:0/3870317479 learned_addr learned my addr 172.21.15.6:0/3870317479 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:06.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.743+0000 7f9811600640 1 -- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06b080 msgr2=0x7f980c14f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:06.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.743+0000 7f9811600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06b080 0x7f980c14f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.743+0000 7f9811600640 1 -- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06fea0 msgr2=0x7f980c1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:06.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.743+0000 7f9811600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06fea0 0x7f980c1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:06.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.743+0000 7f9811600640 1 -- 172.21.15.6:0/3870317479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c152720 con 0x7f980c15de30 2024-03-22T13:11:06.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.746+0000 7f9811600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f980c15de30 0x7f980c1f0f50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f980400eba0 tx=0x7f980400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:06.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.751+0000 7f9808c00640 1 -- 172.21.15.6:0/3870317479 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804018020 con 0x7f980c15de30 2024-03-22T13:11:06.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.751+0000 7f9808c00640 1 -- 172.21.15.6:0/3870317479 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980400cd20 con 0x7f980c15de30 2024-03-22T13:11:06.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.751+0000 7f9808c00640 1 -- 172.21.15.6:0/3870317479 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98040146c0 con 0x7f980c15de30 2024-03-22T13:11:06.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.751+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c1f0790 con 0x7f980c15de30 2024-03-22T13:11:06.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.751+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c157860 con 0x7f980c15de30 2024-03-22T13:11:06.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.753+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c0756c0 con 0x7f980c15de30 2024-03-22T13:11:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.776+0000 7f9808c00640 1 -- 172.21.15.6:0/3870317479 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9804005d10 con 0x7f980c15de30 2024-03-22T13:11:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.778+0000 7f9808c00640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97ec06bc20 0x7f97ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.778+0000 7f9810c00640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97ec06bc20 0x7f97ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:06.779+0000 7f9810c00640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97ec06bc20 0x7f97ec06e0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9800009630 tx=0x7f9800008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:07.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.082+0000 7f9808c00640 1 -- 172.21.15.6:0/3870317479 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f9804058040 con 0x7f980c15de30 2024-03-22T13:11:07.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.083+0000 7f9808c00640 1 -- 172.21.15.6:0/3870317479 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9804022030 con 0x7f980c15de30 2024-03-22T13:11:07.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.380+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f980c06be50 con 0x7f980c15de30 2024-03-22T13:11:07.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.402+0000 7f9808c00640 1 -- 172.21.15.6:0/3870317479 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f9804010040 con 0x7f980c15de30 2024-03-22T13:11:07.402 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:07.403 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:07.403 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97ec06bc20 msgr2=0x7f97ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97ec06bc20 0x7f97ec06e0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9800009630 tx=0x7f9800008030 comp rx=0 tx=0).stop 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f980c15de30 msgr2=0x7f980c1f0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f980c15de30 0x7f980c1f0f50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f980400eba0 tx=0x7f980400c490 comp rx=0 tx=0).stop 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 shutdown_connections 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f980c15de30 0x7f980c1f0f50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f97ec06bc20 0x7f97ec06e0e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f980c06fea0 0x7f980c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 --2- 172.21.15.6:0/3870317479 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f980c06b080 0x7f980c14f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.405+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 >> 172.21.15.6:0/3870317479 conn(0x7f980c0ccb50 msgr2=0x7f980c1516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.406+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 shutdown_connections 2024-03-22T13:11:07.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:07.406+0000 7f9813600640 1 -- 172.21.15.6:0/3870317479 wait complete. 2024-03-22T13:11:08.029 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:08.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:09.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.211+0000 7f560dc00640 1 Processor -- start 2024-03-22T13:11:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.211+0000 7f560dc00640 1 -- start start 2024-03-22T13:11:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.212+0000 7f560dc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56081589f0 0x7f560815f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.213+0000 7f560dc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56080c7640 0x7f5608162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.213+0000 7f560dc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f56081628c0 0x7f560806b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.213+0000 7f560dc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5608161c10 con 0x7f56081589f0 2024-03-22T13:11:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.213+0000 7f560dc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5608158dd0 con 0x7f56081628c0 2024-03-22T13:11:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.213+0000 7f560dc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56080c7b40 con 0x7f56080c7640 2024-03-22T13:11:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.214+0000 7f5607400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f56081628c0 0x7f560806b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:09.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.214+0000 7f5606000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56080c7640 0x7f5608162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:09.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.214+0000 7f5606a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56081589f0 0x7f560815f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:09.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.214+0000 7f5607400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f56081628c0 0x7f560806b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52276/0 (socket says 172.21.15.6:52276) 2024-03-22T13:11:09.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.214+0000 7f5607400640 1 -- 172.21.15.6:0/312380932 learned_addr learned my addr 172.21.15.6:0/312380932 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.216+0000 7f5607400640 1 -- 172.21.15.6:0/312380932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56080c7640 msgr2=0x7f5608162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.216+0000 7f5607400640 1 --2- 172.21.15.6:0/312380932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56080c7640 0x7f5608162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.216+0000 7f5607400640 1 -- 172.21.15.6:0/312380932 >> v2:172.21.15.6:3300/0 conn(0x7f56081589f0 msgr2=0x7f560815f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.216+0000 7f5607400640 1 --2- 172.21.15.6:0/312380932 >> v2:172.21.15.6:3300/0 conn(0x7f56081589f0 0x7f560815f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.216+0000 7f5607400640 1 -- 172.21.15.6:0/312380932 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56080c6b10 con 0x7f56081628c0 2024-03-22T13:11:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.218+0000 7f5607400640 1 --2- 172.21.15.6:0/312380932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56081628c0 0x7f560806b080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f560806c1e0 tx=0x7f560002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4ba04874e1a20c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.223+0000 7f5605600640 1 -- 172.21.15.6:0/312380932 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560002cb00 con 0x7f56081628c0 2024-03-22T13:11:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.223+0000 7f5605600640 1 -- 172.21.15.6:0/312380932 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560002cca0 con 0x7f56081628c0 2024-03-22T13:11:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.223+0000 7f5605600640 1 -- 172.21.15.6:0/312380932 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600034610 con 0x7f56081628c0 2024-03-22T13:11:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 -- 172.21.15.6:0/312380932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56081628c0 msgr2=0x7f560806b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 --2- 172.21.15.6:0/312380932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56081628c0 0x7f560806b080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f560806c1e0 tx=0x7f560002bf60 comp rx=0 tx=0).stop 2024-03-22T13:11:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 -- 172.21.15.6:0/312380932 shutdown_connections 2024-03-22T13:11:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 --2- 172.21.15.6:0/312380932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56081628c0 0x7f560806b080 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 --2- 172.21.15.6:0/312380932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56080c7640 0x7f5608162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 --2- 172.21.15.6:0/312380932 >> v2:172.21.15.6:3300/0 conn(0x7f56081589f0 0x7f560815f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 -- 172.21.15.6:0/312380932 >> 172.21.15.6:0/312380932 conn(0x7f56080cb930 msgr2=0x7f56080cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 -- 172.21.15.6:0/312380932 shutdown_connections 2024-03-22T13:11:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.224+0000 7f560dc00640 1 -- 172.21.15.6:0/312380932 wait complete. 2024-03-22T13:11:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.226+0000 7f560dc00640 1 Processor -- start 2024-03-22T13:11:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.226+0000 7f560dc00640 1 -- start start 2024-03-22T13:11:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.227+0000 7f560dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56080c7640 0x7f5608153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.227+0000 7f560dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56081589f0 0x7f56081e82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.227+0000 7f560dc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56081628c0 0x7f560814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.227+0000 7f560dc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5608161e60 con 0x7f56081589f0 2024-03-22T13:11:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.227+0000 7f560dc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56080c70d0 con 0x7f56080c7640 2024-03-22T13:11:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.227+0000 7f560dc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5608072380 con 0x7f56081628c0 2024-03-22T13:11:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.228+0000 7f5606000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56081589f0 0x7f56081e82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.228+0000 7f5606a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56080c7640 0x7f5608153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.228+0000 7f5607400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56081628c0 0x7f560814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.228+0000 7f5606000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56081589f0 0x7f56081e82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33340/0 (socket says 172.21.15.6:33340) 2024-03-22T13:11:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.228+0000 7f5606000640 1 -- 172.21.15.6:0/414400480 learned_addr learned my addr 172.21.15.6:0/414400480 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.232+0000 7f5607400640 1 -- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56080c7640 msgr2=0x7f5608153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.232+0000 7f5607400640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56080c7640 0x7f5608153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.232+0000 7f5607400640 1 -- 172.21.15.6:0/414400480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56081589f0 msgr2=0x7f56081e82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.232+0000 7f5607400640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56081589f0 0x7f56081e82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.232+0000 7f5607400640 1 -- 172.21.15.6:0/414400480 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56080c57c0 con 0x7f56081628c0 2024-03-22T13:11:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.236+0000 7f5607400640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56081628c0 0x7f560814c800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5608161d90 tx=0x7f5600004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.241+0000 7f55fea00640 1 -- 172.21.15.6:0/414400480 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600043020 con 0x7f56081628c0 2024-03-22T13:11:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.241+0000 7f55fea00640 1 -- 172.21.15.6:0/414400480 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560002ce50 con 0x7f56081628c0 2024-03-22T13:11:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.241+0000 7f55fea00640 1 -- 172.21.15.6:0/414400480 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560003e670 con 0x7f56081628c0 2024-03-22T13:11:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.241+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56081f4300 con 0x7f56081628c0 2024-03-22T13:11:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.241+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56081f48c0 con 0x7f56081628c0 2024-03-22T13:11:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.243+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56080756c0 con 0x7f56081628c0 2024-03-22T13:11:09.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.552+0000 7f55fea00640 1 -- 172.21.15.6:0/414400480 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5600046020 con 0x7f56081628c0 2024-03-22T13:11:09.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.554+0000 7f55fea00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55e806bc20 0x7f55e806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:09.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.554+0000 7f5606a00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55e806bc20 0x7f55e806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:09.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.555+0000 7f5606a00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55e806bc20 0x7f55e806e0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009aa0 tx=0x7f55f8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:09.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.557+0000 7f55fea00640 1 -- 172.21.15.6:0/414400480 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f56000798b0 con 0x7f56081628c0 2024-03-22T13:11:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.576+0000 7f55fea00640 1 -- 172.21.15.6:0/414400480 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f560004d030 con 0x7f56081628c0 2024-03-22T13:11:09.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.819+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f560806be50 con 0x7f56081628c0 2024-03-22T13:11:09.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.833+0000 7f55fea00640 1 -- 172.21.15.6:0/414400480 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f560007b810 con 0x7f56081628c0 2024-03-22T13:11:09.834 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:09.834 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:09.835 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55e806bc20 msgr2=0x7f55e806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55e806bc20 0x7f55e806e0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009aa0 tx=0x7f55f8009190 comp rx=0 tx=0).stop 2024-03-22T13:11:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56081628c0 msgr2=0x7f560814c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56081628c0 0x7f560814c800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5608161d90 tx=0x7f5600004a40 comp rx=0 tx=0).stop 2024-03-22T13:11:09.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 shutdown_connections 2024-03-22T13:11:09.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56081628c0 0x7f560814c800 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56081589f0 0x7f56081e82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f55e806bc20 0x7f55e806e0e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 --2- 172.21.15.6:0/414400480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56080c7640 0x7f5608153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:09.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 >> 172.21.15.6:0/414400480 conn(0x7f56080cb930 msgr2=0x7f560806f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 shutdown_connections 2024-03-22T13:11:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:09.836+0000 7f560dc00640 1 -- 172.21.15.6:0/414400480 wait complete. 2024-03-22T13:11:10.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:10.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:11.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:11.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.653+0000 7f1b49000640 1 Processor -- start 2024-03-22T13:11:11.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.654+0000 7f1b49000640 1 -- start start 2024-03-22T13:11:11.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.655+0000 7f1b49000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b440c35b0 0x7f1b4406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:11.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.655+0000 7f1b49000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440c3d30 con 0x7f1b4406b080 2024-03-22T13:11:11.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.655+0000 7f1b49000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440c3a50 con 0x7f1b4406fea0 2024-03-22T13:11:11.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.655+0000 7f1b49000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4415a450 con 0x7f1b440c35b0 2024-03-22T13:11:11.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.656+0000 7f1b41400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1b4406fea0 0x7f1b440c2ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:51136/0 (socket says 172.21.15.6:51136) 2024-03-22T13:11:11.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.656+0000 7f1b42800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b440c35b0 0x7f1b4406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:11.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.656+0000 7f1b41400640 1 -- 172.21.15.6:0/2218056864 learned_addr learned my addr 172.21.15.6:0/2218056864 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:11.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.659+0000 7f1b40a00640 1 -- 172.21.15.6:0/2218056864 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968047141 0 0) 0x7f1b440c3a50 con 0x7f1b4406fea0 2024-03-22T13:11:11.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.660+0000 7f1b40a00640 1 -- 172.21.15.6:0/2218056864 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28003520 con 0x7f1b4406fea0 2024-03-22T13:11:11.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.660+0000 7f1b42800640 1 -- 172.21.15.6:0/2218056864 >> v1:172.21.15.92:6789/0 conn(0x7f1b4406fea0 legacy=0x7f1b440c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:11.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.660+0000 7f1b42800640 1 -- 172.21.15.6:0/2218056864 >> v1:172.21.15.6:6789/0 conn(0x7f1b4406b080 legacy=0x7f1b4415d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:11.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.660+0000 7f1b42800640 1 -- 172.21.15.6:0/2218056864 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4415a730 con 0x7f1b440c35b0 2024-03-22T13:11:11.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.663+0000 7f1b42800640 1 --2- 172.21.15.6:0/2218056864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b440c35b0 0x7f1b4406b460 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009dc0 tx=0x7f1b3c02bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c6b04ca1a977ff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.668+0000 7f1b40a00640 1 -- 172.21.15.6:0/2218056864 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c039070 con 0x7f1b440c35b0 2024-03-22T13:11:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.668+0000 7f1b40a00640 1 -- 172.21.15.6:0/2218056864 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c02cd60 con 0x7f1b440c35b0 2024-03-22T13:11:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.668+0000 7f1b40a00640 1 -- 172.21.15.6:0/2218056864 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c034660 con 0x7f1b440c35b0 2024-03-22T13:11:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.669+0000 7f1b49000640 1 -- 172.21.15.6:0/2218056864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b440c35b0 msgr2=0x7f1b4406b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.669+0000 7f1b49000640 1 --2- 172.21.15.6:0/2218056864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b440c35b0 0x7f1b4406b460 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009dc0 tx=0x7f1b3c02bfb0 comp rx=0 tx=0).stop 2024-03-22T13:11:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.669+0000 7f1b49000640 1 -- 172.21.15.6:0/2218056864 shutdown_connections 2024-03-22T13:11:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.669+0000 7f1b49000640 1 --2- 172.21.15.6:0/2218056864 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b440c35b0 0x7f1b4406b460 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.669+0000 7f1b49000640 1 -- 172.21.15.6:0/2218056864 >> 172.21.15.6:0/2218056864 conn(0x7f1b440ccb50 msgr2=0x7f1b440ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:11.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.669+0000 7f1b49000640 1 -- 172.21.15.6:0/2218056864 shutdown_connections 2024-03-22T13:11:11.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.669+0000 7f1b49000640 1 -- 172.21.15.6:0/2218056864 wait complete. 2024-03-22T13:11:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.671+0000 7f1b49000640 1 Processor -- start 2024-03-22T13:11:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.671+0000 7f1b49000640 1 -- start start 2024-03-22T13:11:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.672+0000 7f1b49000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4406b080 0x7f1b44074730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.672+0000 7f1b49000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4406fea0 0x7f1b441e50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.672+0000 7f1b49000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b440c35b0 0x7f1b441e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.672+0000 7f1b49000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4415e000 con 0x7f1b440c35b0 2024-03-22T13:11:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.672+0000 7f1b49000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4415acf0 con 0x7f1b4406b080 2024-03-22T13:11:11.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.672+0000 7f1b49000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4415b210 con 0x7f1b4406fea0 2024-03-22T13:11:11.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.673+0000 7f1b41e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4406b080 0x7f1b44074730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:11.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.673+0000 7f1b41400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4406fea0 0x7f1b441e50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:11.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.673+0000 7f1b42800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b440c35b0 0x7f1b441e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:11.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.673+0000 7f1b41e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4406b080 0x7f1b44074730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:52296/0 (socket says 172.21.15.6:52296) 2024-03-22T13:11:11.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.673+0000 7f1b41e00640 1 -- 172.21.15.6:0/2271965671 learned_addr learned my addr 172.21.15.6:0/2271965671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:11.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.677+0000 7f1b41e00640 1 -- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4406fea0 msgr2=0x7f1b441e50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:11.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.677+0000 7f1b41e00640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4406fea0 0x7f1b441e50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:11.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.677+0000 7f1b41e00640 1 -- 172.21.15.6:0/2271965671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b440c35b0 msgr2=0x7f1b441e55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:11.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.677+0000 7f1b41e00640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b440c35b0 0x7f1b441e55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:11.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.677+0000 7f1b41e00640 1 -- 172.21.15.6:0/2271965671 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b441e4760 con 0x7f1b4406b080 2024-03-22T13:11:11.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.679+0000 7f1b41e00640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4406b080 0x7f1b44074730 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400c3f0 tx=0x7f1b3400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.683+0000 7f1b3a000640 1 -- 172.21.15.6:0/2271965671 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b340052d0 con 0x7f1b4406b080 2024-03-22T13:11:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.683+0000 7f1b3a000640 1 -- 172.21.15.6:0/2271965671 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b34005490 con 0x7f1b4406b080 2024-03-22T13:11:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.683+0000 7f1b3a000640 1 -- 172.21.15.6:0/2271965671 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b34007260 con 0x7f1b4406b080 2024-03-22T13:11:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.683+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b44157ec0 con 0x7f1b4406b080 2024-03-22T13:11:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.683+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b441efd30 con 0x7f1b4406b080 2024-03-22T13:11:11.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.684+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b440756c0 con 0x7f1b4406b080 2024-03-22T13:11:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.709+0000 7f1b3a000640 1 -- 172.21.15.6:0/2271965671 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b34007400 con 0x7f1b4406b080 2024-03-22T13:11:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.710+0000 7f1b3a000640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2406bc50 0x7f1b2406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.710+0000 7f1b41400640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2406bc50 0x7f1b2406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:11.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.711+0000 7f1b41400640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2406bc50 0x7f1b2406e110 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1b441e6210 tx=0x7f1b30007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:11.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.967+0000 7f1b3a000640 1 -- 172.21.15.6:0/2271965671 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1b34056bc0 con 0x7f1b4406b080 2024-03-22T13:11:11.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:11.978+0000 7f1b3a000640 1 -- 172.21.15.6:0/2271965671 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b34020070 con 0x7f1b4406b080 2024-03-22T13:11:12.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.275+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b4406be50 con 0x7f1b4406b080 2024-03-22T13:11:12.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.351+0000 7f1b3a000640 1 -- 172.21.15.6:0/2271965671 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f1b34058b20 con 0x7f1b4406b080 2024-03-22T13:11:12.352 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:12.353 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:12.353 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:12.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2406bc50 msgr2=0x7f1b2406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:12.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2406bc50 0x7f1b2406e110 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1b441e6210 tx=0x7f1b30007d30 comp rx=0 tx=0).stop 2024-03-22T13:11:12.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4406b080 msgr2=0x7f1b44074730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:12.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4406b080 0x7f1b44074730 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400c3f0 tx=0x7f1b3400c8b0 comp rx=0 tx=0).stop 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 shutdown_connections 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b440c35b0 0x7f1b441e55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b4406fea0 0x7f1b441e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b2406bc50 0x7f1b2406e110 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 --2- 172.21.15.6:0/2271965671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b4406b080 0x7f1b44074730 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 >> 172.21.15.6:0/2271965671 conn(0x7f1b440ccb50 msgr2=0x7f1b44152a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 shutdown_connections 2024-03-22T13:11:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:12.356+0000 7f1b49000640 1 -- 172.21.15.6:0/2271965671 wait complete. 2024-03-22T13:11:12.979 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:12.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:13.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:14.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.171+0000 7fdb5cc00640 1 Processor -- start 2024-03-22T13:11:14.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.171+0000 7fdb5cc00640 1 -- start start 2024-03-22T13:11:14.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.173+0000 7fdb5cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb58150f50 0x7fdb580c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:14.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.173+0000 7fdb5cc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdb58157420 0x7fdb580cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:14.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.173+0000 7fdb5cc00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb580c4a20 con 0x7fdb58156f90 2024-03-22T13:11:14.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.173+0000 7fdb5cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb580c67c0 con 0x7fdb58150f50 2024-03-22T13:11:14.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.173+0000 7fdb5cc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb580c6aa0 con 0x7fdb58157420 2024-03-22T13:11:14.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.174+0000 7fdb57e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdb58157420 0x7fdb580cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:14.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.174+0000 7fdb57400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb58150f50 0x7fdb580c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:14.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.175+0000 7fdb57400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdb58150f50 0x7fdb580c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58120/0 (socket says 172.21.15.6:58120) 2024-03-22T13:11:14.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.175+0000 7fdb57400640 1 -- 172.21.15.6:0/1660365720 learned_addr learned my addr 172.21.15.6:0/1660365720 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:14.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.178+0000 7fdb57400640 1 -- 172.21.15.6:0/1660365720 >> v2:172.21.15.92:3300/0 conn(0x7fdb58157420 msgr2=0x7fdb580cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:14.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.178+0000 7fdb57400640 1 --2- 172.21.15.6:0/1660365720 >> v2:172.21.15.92:3300/0 conn(0x7fdb58157420 0x7fdb580cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:14.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.178+0000 7fdb57400640 1 -- 172.21.15.6:0/1660365720 >> v1:172.21.15.92:6789/0 conn(0x7fdb58156f90 legacy=0x7fdb58072d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:14.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.178+0000 7fdb57400640 1 -- 172.21.15.6:0/1660365720 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb580c3f40 con 0x7fdb58150f50 2024-03-22T13:11:14.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.181+0000 7fdb57400640 1 --2- 172.21.15.6:0/1660365720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb580c60a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdb580c9260 tx=0x7fdb4802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f984d2d91853b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:14.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.186+0000 7fdb56000640 1 -- 172.21.15.6:0/1660365720 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4802f850 con 0x7fdb58150f50 2024-03-22T13:11:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.187+0000 7fdb56000640 1 -- 172.21.15.6:0/1660365720 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4802f9f0 con 0x7fdb58150f50 2024-03-22T13:11:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.187+0000 7fdb56000640 1 -- 172.21.15.6:0/1660365720 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4802fdc0 con 0x7fdb58150f50 2024-03-22T13:11:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.187+0000 7fdb5cc00640 1 -- 172.21.15.6:0/1660365720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 msgr2=0x7fdb580c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.187+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/1660365720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb580c60a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdb580c9260 tx=0x7fdb4802ba30 comp rx=0 tx=0).stop 2024-03-22T13:11:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.188+0000 7fdb5cc00640 1 -- 172.21.15.6:0/1660365720 shutdown_connections 2024-03-22T13:11:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.188+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/1660365720 >> v2:172.21.15.92:3300/0 conn(0x7fdb58157420 0x7fdb580cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.188+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/1660365720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb580c60a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.188+0000 7fdb5cc00640 1 -- 172.21.15.6:0/1660365720 >> 172.21.15.6:0/1660365720 conn(0x7fdb58070140 msgr2=0x7fdb58070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.188+0000 7fdb5cc00640 1 -- 172.21.15.6:0/1660365720 shutdown_connections 2024-03-22T13:11:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.188+0000 7fdb5cc00640 1 -- 172.21.15.6:0/1660365720 wait complete. 2024-03-22T13:11:14.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.189+0000 7fdb5cc00640 1 Processor -- start 2024-03-22T13:11:14.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.190+0000 7fdb5cc00640 1 -- start start 2024-03-22T13:11:14.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.190+0000 7fdb5cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb5815af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:14.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb5cc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb58156f90 0x7fdb58151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb5cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb58157420 0x7fdb58154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb5cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb58151330 con 0x7fdb58150f50 2024-03-22T13:11:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb5cc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb58073410 con 0x7fdb58157420 2024-03-22T13:11:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb5cc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb581514b0 con 0x7fdb58156f90 2024-03-22T13:11:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb5815af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb57e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb58157420 0x7fdb58154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.191+0000 7fdb56a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb58156f90 0x7fdb58151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.192+0000 7fdb57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb5815af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58128/0 (socket says 172.21.15.6:58128) 2024-03-22T13:11:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.192+0000 7fdb57400640 1 -- 172.21.15.6:0/3029229518 learned_addr learned my addr 172.21.15.6:0/3029229518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.195+0000 7fdb57400640 1 -- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb58156f90 msgr2=0x7fdb58151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.195+0000 7fdb57400640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb58156f90 0x7fdb58151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.195+0000 7fdb57400640 1 -- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb58157420 msgr2=0x7fdb58154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.195+0000 7fdb57400640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb58157420 0x7fdb58154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.196+0000 7fdb57400640 1 -- 172.21.15.6:0/3029229518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb58153eb0 con 0x7fdb58150f50 2024-03-22T13:11:14.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.203+0000 7fdb57400640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb5815af10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdb4802ba70 tx=0x7fdb48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.209+0000 7fdb4f400640 1 -- 172.21.15.6:0/3029229518 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48037d10 con 0x7fdb58150f50 2024-03-22T13:11:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.209+0000 7fdb4f400640 1 -- 172.21.15.6:0/3029229518 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb48046070 con 0x7fdb58150f50 2024-03-22T13:11:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.209+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5814e8a0 con 0x7fdb58150f50 2024-03-22T13:11:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.209+0000 7fdb4f400640 1 -- 172.21.15.6:0/3029229518 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48041640 con 0x7fdb58150f50 2024-03-22T13:11:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.209+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb581590c0 con 0x7fdb58150f50 2024-03-22T13:11:14.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.211+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb580c55e0 con 0x7fdb58150f50 2024-03-22T13:11:14.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.642+0000 7fdb4f400640 1 -- 172.21.15.6:0/3029229518 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdb48040070 con 0x7fdb58150f50 2024-03-22T13:11:14.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.645+0000 7fdb4f400640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb3806bc50 0x7fdb3806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:14.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.645+0000 7fdb56a00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb3806bc50 0x7fdb3806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:14.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.647+0000 7fdb56a00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb3806bc50 0x7fdb3806e110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdb580cc110 tx=0x7fdb44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:14.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.681+0000 7fdb4f400640 1 -- 172.21.15.6:0/3029229518 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdb48079fc0 con 0x7fdb58150f50 2024-03-22T13:11:14.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.683+0000 7fdb4f400640 1 -- 172.21.15.6:0/3029229518 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdb580c55e0 con 0x7fdb58150f50 2024-03-22T13:11:14.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.959+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb58150a90 con 0x7fdb58150f50 2024-03-22T13:11:14.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:14.996+0000 7fdb4f400640 1 -- 172.21.15.6:0/3029229518 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fdb48033030 con 0x7fdb58150f50 2024-03-22T13:11:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:14.997 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:14.997 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb3806bc50 msgr2=0x7fdb3806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb3806bc50 0x7fdb3806e110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdb580cc110 tx=0x7fdb44009040 comp rx=0 tx=0).stop 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 msgr2=0x7fdb5815af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb5815af10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdb4802ba70 tx=0x7fdb48004730 comp rx=0 tx=0).stop 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 shutdown_connections 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdb3806bc50 0x7fdb3806e110 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb58157420 0x7fdb58154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb58156f90 0x7fdb58151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:15.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 --2- 172.21.15.6:0/3029229518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb58150f50 0x7fdb5815af10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:15.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 >> 172.21.15.6:0/3029229518 conn(0x7fdb58070140 msgr2=0x7fdb5814eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:15.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 shutdown_connections 2024-03-22T13:11:15.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:15.000+0000 7fdb5cc00640 1 -- 172.21.15.6:0/3029229518 wait complete. 2024-03-22T13:11:15.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:15.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:16.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.832+0000 7fe83bc00640 1 Processor -- start 2024-03-22T13:11:16.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.833+0000 7fe83bc00640 1 -- start start 2024-03-22T13:11:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.834+0000 7fe83bc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe83406f8c0 0x7fe8340716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.834+0000 7fe83bc00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe834071c10 0x7fe8340c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.834+0000 7fe83bc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8341590b0 con 0x7fe834152400 2024-03-22T13:11:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.834+0000 7fe83bc00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8340724d0 con 0x7fe834071c10 2024-03-22T13:11:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.834+0000 7fe83bc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340721f0 con 0x7fe83406f8c0 2024-03-22T13:11:16.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.835+0000 7fe839200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe834152400 0x7fe83415b740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48982/0 (socket says 172.21.15.6:48982) 2024-03-22T13:11:16.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.835+0000 7fe839200640 1 -- 172.21.15.6:0/2883816550 learned_addr learned my addr 172.21.15.6:0/2883816550 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:16.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.835+0000 7fe839c00640 1 --2- 172.21.15.6:0/2883816550 >> v2:172.21.15.92:3300/0 conn(0x7fe834071c10 0x7fe8340c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:16.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.835+0000 7fe833e00640 1 --2- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe83406f8c0 0x7fe8340716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:16.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.838+0000 7fe839c00640 1 -- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe83406f8c0 msgr2=0x7fe8340716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:16.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.838+0000 7fe839c00640 1 --2- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe83406f8c0 0x7fe8340716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:16.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.838+0000 7fe839c00640 1 -- 172.21.15.6:0/2883816550 >> v1:172.21.15.6:6789/0 conn(0x7fe834152400 legacy=0x7fe83415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:16.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.838+0000 7fe839c00640 1 -- 172.21.15.6:0/2883816550 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8341582f0 con 0x7fe834071c10 2024-03-22T13:11:16.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.841+0000 7fe839c00640 1 --2- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8340c5040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009d90 tx=0x7fe82c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=301dbdb2a0abe1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:16.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.843+0000 7fe833400640 1 -- 172.21.15.6:0/2883816550 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c039070 con 0x7fe834071c10 2024-03-22T13:11:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.845+0000 7fe833400640 1 -- 172.21.15.6:0/2883816550 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c02cce0 con 0x7fe834071c10 2024-03-22T13:11:16.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.845+0000 7fe833400640 1 -- 172.21.15.6:0/2883816550 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c034660 con 0x7fe834071c10 2024-03-22T13:11:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.846+0000 7fe83bc00640 1 -- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 msgr2=0x7fe8340c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.846+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8340c5040 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009d90 tx=0x7fe82c02bf30 comp rx=0 tx=0).stop 2024-03-22T13:11:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.846+0000 7fe83bc00640 1 -- 172.21.15.6:0/2883816550 shutdown_connections 2024-03-22T13:11:16.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.846+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8340c5040 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:16.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.846+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2883816550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe83406f8c0 0x7fe8340716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:16.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.846+0000 7fe83bc00640 1 -- 172.21.15.6:0/2883816550 >> 172.21.15.6:0/2883816550 conn(0x7fe8340cb930 msgr2=0x7fe8340cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:16.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.846+0000 7fe83bc00640 1 -- 172.21.15.6:0/2883816550 shutdown_connections 2024-03-22T13:11:16.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.847+0000 7fe83bc00640 1 -- 172.21.15.6:0/2883816550 wait complete. 2024-03-22T13:11:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.848+0000 7fe83bc00640 1 Processor -- start 2024-03-22T13:11:16.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.848+0000 7fe83bc00640 1 -- start start 2024-03-22T13:11:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.849+0000 7fe83bc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe83406f8c0 0x7fe8341432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.849+0000 7fe83bc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8341e5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.849+0000 7fe83bc00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe834152400 0x7fe8341e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.849+0000 7fe83bc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe834158b90 con 0x7fe83406f8c0 2024-03-22T13:11:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.849+0000 7fe83bc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341588b0 con 0x7fe834071c10 2024-03-22T13:11:16.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.849+0000 7fe83bc00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341527e0 con 0x7fe834152400 2024-03-22T13:11:16.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.850+0000 7fe839200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe83406f8c0 0x7fe8341432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:16.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.850+0000 7fe839c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe834152400 0x7fe8341e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:16.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.850+0000 7fe833e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8341e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:16.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.850+0000 7fe833e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8341e5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:53394/0 (socket says 172.21.15.6:53394) 2024-03-22T13:11:16.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.850+0000 7fe833e00640 1 -- 172.21.15.6:0/2415665485 learned_addr learned my addr 172.21.15.6:0/2415665485 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.854+0000 7fe833e00640 1 -- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe834152400 msgr2=0x7fe8341e5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:16.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.854+0000 7fe833e00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe834152400 0x7fe8341e5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.854+0000 7fe833e00640 1 -- 172.21.15.6:0/2415665485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe83406f8c0 msgr2=0x7fe8341432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.854+0000 7fe833e00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe83406f8c0 0x7fe8341432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:16.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.854+0000 7fe833e00640 1 -- 172.21.15.6:0/2415665485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83414e0b0 con 0x7fe834071c10 2024-03-22T13:11:16.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.856+0000 7fe833e00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8341e5930 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe82000e9b0 tx=0x7fe82000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.860+0000 7fe830c00640 1 -- 172.21.15.6:0/2415665485 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820017070 con 0x7fe834071c10 2024-03-22T13:11:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.860+0000 7fe830c00640 1 -- 172.21.15.6:0/2415665485 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82000c9a0 con 0x7fe834071c10 2024-03-22T13:11:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.860+0000 7fe830c00640 1 -- 172.21.15.6:0/2415665485 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82000cd90 con 0x7fe834071c10 2024-03-22T13:11:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.860+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83414e350 con 0x7fe834071c10 2024-03-22T13:11:16.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.860+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83414e910 con 0x7fe834071c10 2024-03-22T13:11:16.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.861+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8340756c0 con 0x7fe834071c10 2024-03-22T13:11:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.892+0000 7fe830c00640 1 -- 172.21.15.6:0/2415665485 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe82001a070 con 0x7fe834071c10 2024-03-22T13:11:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.894+0000 7fe830c00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe81406bc20 0x7fe81406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.894+0000 7fe839200640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe81406bc20 0x7fe81406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.895+0000 7fe839200640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe81406bc20 0x7fe81406e0e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe828007fd0 tx=0x7fe828007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:16.898+0000 7fe830c00640 1 -- 172.21.15.6:0/2415665485 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe820057740 con 0x7fe834071c10 2024-03-22T13:11:17.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.156+0000 7fe830c00640 1 -- 172.21.15.6:0/2415665485 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe8200596a0 con 0x7fe834071c10 2024-03-22T13:11:17.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.408+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8340c6bc0 con 0x7fe834071c10 2024-03-22T13:11:17.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.429+0000 7fe830c00640 1 -- 172.21.15.6:0/2415665485 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fe82005d740 con 0x7fe834071c10 2024-03-22T13:11:17.430 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:17.431 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:17.431 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe81406bc20 msgr2=0x7fe81406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe81406bc20 0x7fe81406e0e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe828007fd0 tx=0x7fe828007e80 comp rx=0 tx=0).stop 2024-03-22T13:11:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 msgr2=0x7fe8341e5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8341e5930 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe82000e9b0 tx=0x7fe82000ee70 comp rx=0 tx=0).stop 2024-03-22T13:11:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 shutdown_connections 2024-03-22T13:11:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe834152400 0x7fe8341e5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe81406bc20 0x7fe81406e0e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:17.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe834071c10 0x7fe8341e5930 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:17.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 --2- 172.21.15.6:0/2415665485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe83406f8c0 0x7fe8341432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:17.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 >> 172.21.15.6:0/2415665485 conn(0x7fe8340cb930 msgr2=0x7fe8340c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:17.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 shutdown_connections 2024-03-22T13:11:17.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:17.433+0000 7fe83bc00640 1 -- 172.21.15.6:0/2415665485 wait complete. 2024-03-22T13:11:18.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:18.056 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:19.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:19.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.248+0000 7ff335000640 1 Processor -- start 2024-03-22T13:11:19.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.248+0000 7ff335000640 1 -- start start 2024-03-22T13:11:19.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.250+0000 7ff335000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3300cc380 0x7ff33006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:19.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.250+0000 7ff335000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff3300ce8a0 0x7ff33015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:19.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.250+0000 7ff335000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3300cee00 con 0x7ff3300cc760 2024-03-22T13:11:19.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.250+0000 7ff335000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3300cf0c0 con 0x7ff3300cc380 2024-03-22T13:11:19.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.250+0000 7ff335000640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3300cf3a0 con 0x7ff3300ce8a0 2024-03-22T13:11:19.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.251+0000 7ff32f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3300cc380 0x7ff33006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:19.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.251+0000 7ff326a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff3300ce8a0 0x7ff33015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:19.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.251+0000 7ff32fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff3300cc760 0x7ff3300cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48996/0 (socket says 172.21.15.6:48996) 2024-03-22T13:11:19.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.251+0000 7ff32fe00640 1 -- 172.21.15.6:0/3703205275 learned_addr learned my addr 172.21.15.6:0/3703205275 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:19.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.255+0000 7ff326a00640 1 -- 172.21.15.6:0/3703205275 >> v2:172.21.15.6:3300/0 conn(0x7ff3300cc380 msgr2=0x7ff33006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:19.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.255+0000 7ff326a00640 1 --2- 172.21.15.6:0/3703205275 >> v2:172.21.15.6:3300/0 conn(0x7ff3300cc380 0x7ff33006c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.255+0000 7ff326a00640 1 -- 172.21.15.6:0/3703205275 >> v1:172.21.15.6:6789/0 conn(0x7ff3300cc760 legacy=0x7ff3300cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:19.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.255+0000 7ff326a00640 1 -- 172.21.15.6:0/3703205275 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3300cf680 con 0x7ff3300ce8a0 2024-03-22T13:11:19.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.255+0000 7ff32f400640 1 --2- 172.21.15.6:0/3703205275 >> v2:172.21.15.6:3300/0 conn(0x7ff3300cc380 0x7ff33006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:11:19.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.258+0000 7ff326a00640 1 --2- 172.21.15.6:0/3703205275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300ce8a0 0x7ff33015ae30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff3300cb7f0 tx=0x7ff31802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed105e93e724d816 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:19.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.261+0000 7ff32ea00640 1 -- 172.21.15.6:0/3703205275 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff318035020 con 0x7ff3300ce8a0 2024-03-22T13:11:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.263+0000 7ff32ea00640 1 -- 172.21.15.6:0/3703205275 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff318031830 con 0x7ff3300ce8a0 2024-03-22T13:11:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.263+0000 7ff32ea00640 1 -- 172.21.15.6:0/3703205275 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff318031c00 con 0x7ff3300ce8a0 2024-03-22T13:11:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 -- 172.21.15.6:0/3703205275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300ce8a0 msgr2=0x7ff33015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 --2- 172.21.15.6:0/3703205275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300ce8a0 0x7ff33015ae30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff3300cb7f0 tx=0x7ff31802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 -- 172.21.15.6:0/3703205275 shutdown_connections 2024-03-22T13:11:19.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 --2- 172.21.15.6:0/3703205275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300ce8a0 0x7ff33015ae30 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 --2- 172.21.15.6:0/3703205275 >> v2:172.21.15.6:3300/0 conn(0x7ff3300cc380 0x7ff33006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 -- 172.21.15.6:0/3703205275 >> 172.21.15.6:0/3703205275 conn(0x7ff3300b1480 msgr2=0x7ff3300b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:19.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 -- 172.21.15.6:0/3703205275 shutdown_connections 2024-03-22T13:11:19.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.264+0000 7ff335000640 1 -- 172.21.15.6:0/3703205275 wait complete. 2024-03-22T13:11:19.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.267+0000 7ff335000640 1 Processor -- start 2024-03-22T13:11:19.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.267+0000 7ff335000640 1 -- start start 2024-03-22T13:11:19.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.269+0000 7ff335000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3300cc380 0x7ff330152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:19.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.269+0000 7ff335000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300cc760 0x7ff33015b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:19.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.269+0000 7ff335000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3300ce8a0 0x7ff3300b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:19.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.269+0000 7ff335000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3301549d0 con 0x7ff3300cc380 2024-03-22T13:11:19.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.269+0000 7ff335000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33015a910 con 0x7ff3300cc760 2024-03-22T13:11:19.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.269+0000 7ff335000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300ccb70 con 0x7ff3300ce8a0 2024-03-22T13:11:19.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.270+0000 7ff326a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300cc760 0x7ff33015b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:19.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.270+0000 7ff32fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3300ce8a0 0x7ff3300b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:19.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.270+0000 7ff32f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3300cc380 0x7ff330152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:19.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.270+0000 7ff32fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3300ce8a0 0x7ff3300b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:35084/0 (socket says 172.21.15.6:35084) 2024-03-22T13:11:19.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.270+0000 7ff32fe00640 1 -- 172.21.15.6:0/3562810843 learned_addr learned my addr 172.21.15.6:0/3562810843 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.274+0000 7ff32f400640 1 -- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3300ce8a0 msgr2=0x7ff3300b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.274+0000 7ff32f400640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3300ce8a0 0x7ff3300b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.274+0000 7ff32f400640 1 -- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300cc760 msgr2=0x7ff33015b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.274+0000 7ff32f400640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300cc760 0x7ff33015b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.274+0000 7ff32f400640 1 -- 172.21.15.6:0/3562810843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33015bd40 con 0x7ff3300cc380 2024-03-22T13:11:19.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.277+0000 7ff32f400640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3300cc380 0x7ff330152d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff320009e60 tx=0x7ff32000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:19.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.283+0000 7ff327e00640 1 -- 172.21.15.6:0/3562810843 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320018020 con 0x7ff3300cc380 2024-03-22T13:11:19.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.283+0000 7ff327e00640 1 -- 172.21.15.6:0/3562810843 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff320014680 con 0x7ff3300cc380 2024-03-22T13:11:19.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.283+0000 7ff327e00640 1 -- 172.21.15.6:0/3562810843 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320014a70 con 0x7ff3300cc380 2024-03-22T13:11:19.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.283+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33015cc80 con 0x7ff3300cc380 2024-03-22T13:11:19.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.283+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33015d240 con 0x7ff3300cc380 2024-03-22T13:11:19.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.285+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff330075650 con 0x7ff3300cc380 2024-03-22T13:11:19.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.318+0000 7ff327e00640 1 -- 172.21.15.6:0/3562810843 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff320005d10 con 0x7ff3300cc380 2024-03-22T13:11:19.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.319+0000 7ff327e00640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff30c06bc50 0x7ff30c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:19.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.320+0000 7ff326a00640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff30c06bc50 0x7ff30c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:19.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.321+0000 7ff326a00640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff30c06bc50 0x7ff30c06e110 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff318036000 tx=0x7ff31803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:19.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.323+0000 7ff327e00640 1 -- 172.21.15.6:0/3562810843 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff320058400 con 0x7ff3300cc380 2024-03-22T13:11:19.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.650+0000 7ff327e00640 1 -- 172.21.15.6:0/3562810843 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff320022020 con 0x7ff3300cc380 2024-03-22T13:11:19.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.904+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff330068150 con 0x7ff3300cc380 2024-03-22T13:11:19.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.957+0000 7ff327e00640 1 -- 172.21.15.6:0/3562810843 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7ff320010040 con 0x7ff3300cc380 2024-03-22T13:11:19.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:19.958 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:19.959 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.960+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff30c06bc50 msgr2=0x7ff30c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.960+0000 7ff335000640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff30c06bc50 0x7ff30c06e110 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff318036000 tx=0x7ff31803c040 comp rx=0 tx=0).stop 2024-03-22T13:11:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.960+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3300cc380 msgr2=0x7ff330152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.960+0000 7ff335000640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3300cc380 0x7ff330152d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff320009e60 tx=0x7ff32000c490 comp rx=0 tx=0).stop 2024-03-22T13:11:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 shutdown_connections 2024-03-22T13:11:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3300ce8a0 0x7ff3300b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff30c06bc50 0x7ff30c06e110 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3300cc760 0x7ff33015b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 --2- 172.21.15.6:0/3562810843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3300cc380 0x7ff330152d80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 >> 172.21.15.6:0/3562810843 conn(0x7ff3300b1480 msgr2=0x7ff3300cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 shutdown_connections 2024-03-22T13:11:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:19.961+0000 7ff335000640 1 -- 172.21.15.6:0/3562810843 wait complete. 2024-03-22T13:11:20.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:20.585 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:21.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.783+0000 7f0d87200640 1 Processor -- start 2024-03-22T13:11:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.783+0000 7f0d87200640 1 -- start start 2024-03-22T13:11:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.784+0000 7f0d87200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0d80152400 0x7f0d8015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.785+0000 7f0d87200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d801590b0 con 0x7f0d8006f8c0 2024-03-22T13:11:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.785+0000 7f0d87200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d800721f0 con 0x7f0d8015bc80 2024-03-22T13:11:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.785+0000 7f0d87200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d800724d0 con 0x7f0d80152400 2024-03-22T13:11:21.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.785+0000 7f0d85200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0d8006f8c0 0x7f0d800c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49010/0 (socket says 172.21.15.6:49010) 2024-03-22T13:11:21.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.785+0000 7f0d85200640 1 -- 172.21.15.6:0/689563268 learned_addr learned my addr 172.21.15.6:0/689563268 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:21.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.787+0000 7f0d7fe00640 1 --2- 172.21.15.6:0/689563268 >> v2:172.21.15.92:3300/0 conn(0x7f0d80152400 0x7f0d8015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:21.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.789+0000 7f0d7f400640 1 -- 172.21.15.6:0/689563268 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858380090 0 0) 0x7f0d801590b0 con 0x7f0d8006f8c0 2024-03-22T13:11:21.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.789+0000 7f0d7f400640 1 -- 172.21.15.6:0/689563268 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d64003520 con 0x7f0d8006f8c0 2024-03-22T13:11:21.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.790+0000 7f0d7fe00640 1 -- 172.21.15.6:0/689563268 >> v1:172.21.15.92:6789/0 conn(0x7f0d8015bc80 legacy=0x7f0d800716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:21.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.790+0000 7f0d7fe00640 1 -- 172.21.15.6:0/689563268 >> v1:172.21.15.6:6789/0 conn(0x7f0d8006f8c0 legacy=0x7f0d800c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:21.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.790+0000 7f0d7fe00640 1 -- 172.21.15.6:0/689563268 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d801582f0 con 0x7f0d80152400 2024-03-22T13:11:21.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.794+0000 7f0d7fe00640 1 --2- 172.21.15.6:0/689563268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d8015b740 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0d700097a0 tx=0x7f0d7002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67ac802d3d75c698 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.799+0000 7f0d7f400640 1 -- 172.21.15.6:0/689563268 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7002f810 con 0x7f0d80152400 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.799+0000 7f0d7f400640 1 -- 172.21.15.6:0/689563268 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7002f9b0 con 0x7f0d80152400 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.799+0000 7f0d7f400640 1 -- 172.21.15.6:0/689563268 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7002fd80 con 0x7f0d80152400 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.800+0000 7f0d87200640 1 -- 172.21.15.6:0/689563268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 msgr2=0x7f0d8015b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.800+0000 7f0d87200640 1 --2- 172.21.15.6:0/689563268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d8015b740 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0d700097a0 tx=0x7f0d7002b9f0 comp rx=0 tx=0).stop 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.800+0000 7f0d87200640 1 -- 172.21.15.6:0/689563268 shutdown_connections 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.800+0000 7f0d87200640 1 --2- 172.21.15.6:0/689563268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d8015b740 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.800+0000 7f0d87200640 1 -- 172.21.15.6:0/689563268 >> 172.21.15.6:0/689563268 conn(0x7f0d800cb930 msgr2=0x7f0d800cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.800+0000 7f0d87200640 1 -- 172.21.15.6:0/689563268 shutdown_connections 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.800+0000 7f0d87200640 1 -- 172.21.15.6:0/689563268 wait complete. 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.802+0000 7f0d87200640 1 Processor -- start 2024-03-22T13:11:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.802+0000 7f0d87200640 1 -- start start 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.803+0000 7f0d87200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8006f8c0 0x7f0d801effb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.803+0000 7f0d87200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d80157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.803+0000 7f0d87200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d8015bc80 0x7f0d801597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.803+0000 7f0d87200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d80158b90 con 0x7f0d8006f8c0 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.803+0000 7f0d87200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d801588b0 con 0x7f0d80152400 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.803+0000 7f0d87200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d801527e0 con 0x7f0d8015bc80 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.804+0000 7f0d7fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8006f8c0 0x7f0d801effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.804+0000 7f0d77400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d80157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.804+0000 7f0d85200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d8015bc80 0x7f0d801597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.804+0000 7f0d7fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8006f8c0 0x7f0d801effb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58168/0 (socket says 172.21.15.6:58168) 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.804+0000 7f0d77400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d80157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:53434/0 (socket says 172.21.15.6:53434) 2024-03-22T13:11:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.804+0000 7f0d7fe00640 1 -- 172.21.15.6:0/3200936365 learned_addr learned my addr 172.21.15.6:0/3200936365 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:21.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.808+0000 7f0d7fe00640 1 -- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d8015bc80 msgr2=0x7f0d801597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:21.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.808+0000 7f0d7fe00640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d8015bc80 0x7f0d801597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:21.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.808+0000 7f0d7fe00640 1 -- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 msgr2=0x7f0d80157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:21.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.808+0000 7f0d7fe00640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d80157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:21.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.808+0000 7f0d7fe00640 1 -- 172.21.15.6:0/3200936365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8015adb0 con 0x7f0d8006f8c0 2024-03-22T13:11:21.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.812+0000 7f0d7fe00640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8006f8c0 0x7f0d801effb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0d7002c000 tx=0x7f0d700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:21.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.816+0000 7f0d7cc00640 1 -- 172.21.15.6:0/3200936365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70047070 con 0x7f0d8006f8c0 2024-03-22T13:11:21.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.816+0000 7f0d7cc00640 1 -- 172.21.15.6:0/3200936365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d700378d0 con 0x7f0d8006f8c0 2024-03-22T13:11:21.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.816+0000 7f0d7cc00640 1 -- 172.21.15.6:0/3200936365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70037cc0 con 0x7f0d8006f8c0 2024-03-22T13:11:21.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.816+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8015b090 con 0x7f0d8006f8c0 2024-03-22T13:11:21.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.816+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8015b650 con 0x7f0d8006f8c0 2024-03-22T13:11:21.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:21.817+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d800c9340 con 0x7f0d8006f8c0 2024-03-22T13:11:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.135+0000 7f0d7cc00640 1 -- 172.21.15.6:0/3200936365 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0d7004a070 con 0x7f0d8006f8c0 2024-03-22T13:11:22.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.136+0000 7f0d7cc00640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0d5c06bc00 0x7f0d5c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:22.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.137+0000 7f0d77400640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0d5c06bc00 0x7f0d5c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:22.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.138+0000 7f0d77400640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0d5c06bc00 0x7f0d5c06e0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0d800cb770 tx=0x7f0d68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.178+0000 7f0d7cc00640 1 -- 172.21.15.6:0/3200936365 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0d70079570 con 0x7f0d8006f8c0 2024-03-22T13:11:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.179+0000 7f0d7cc00640 1 -- 172.21.15.6:0/3200936365 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0d800c9340 con 0x7f0d8006f8c0 2024-03-22T13:11:22.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.475+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d80159310 con 0x7f0d8006f8c0 2024-03-22T13:11:22.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.498+0000 7f0d7cc00640 1 -- 172.21.15.6:0/3200936365 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f0d80159310 con 0x7f0d8006f8c0 2024-03-22T13:11:22.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:22.499 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:22.499 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0d5c06bc00 msgr2=0x7f0d5c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0d5c06bc00 0x7f0d5c06e0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0d800cb770 tx=0x7f0d68009040 comp rx=0 tx=0).stop 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8006f8c0 msgr2=0x7f0d801effb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8006f8c0 0x7f0d801effb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0d7002c000 tx=0x7f0d700046c0 comp rx=0 tx=0).stop 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 shutdown_connections 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d8015bc80 0x7f0d801597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0d5c06bc00 0x7f0d5c06e0c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:22.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d80152400 0x7f0d80157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 --2- 172.21.15.6:0/3200936365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d8006f8c0 0x7f0d801effb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 >> 172.21.15.6:0/3200936365 conn(0x7f0d800cb930 msgr2=0x7f0d800c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 shutdown_connections 2024-03-22T13:11:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:22.501+0000 7f0d87200640 1 -- 172.21.15.6:0/3200936365 wait complete. 2024-03-22T13:11:23.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:23.128 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:24.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:24.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.322+0000 7f5681800640 1 Processor -- start 2024-03-22T13:11:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.323+0000 7f5681800640 1 -- start start 2024-03-22T13:11:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.324+0000 7f5681800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f567c0c9630 0x7f567c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.324+0000 7f5681800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.324+0000 7f5681800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c15c1e0 con 0x7f567c0c2e50 2024-03-22T13:11:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.324+0000 7f5681800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c158ee0 con 0x7f567c0c9630 2024-03-22T13:11:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.324+0000 7f5681800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c158010 con 0x7f567c15b850 2024-03-22T13:11:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.325+0000 7f567b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.325+0000 7f567a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f567c0c2e50 0x7f567c0c8a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59560/0 (socket says 172.21.15.6:59560) 2024-03-22T13:11:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.325+0000 7f567a000640 1 -- 172.21.15.6:0/1802686811 learned_addr learned my addr 172.21.15.6:0/1802686811 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.325+0000 7f567aa00640 1 --2- 172.21.15.6:0/1802686811 >> v2:172.21.15.92:3300/0 conn(0x7f567c0c9630 0x7f567c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:24.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.329+0000 7f567b400640 1 -- 172.21.15.6:0/1802686811 >> v2:172.21.15.92:3300/0 conn(0x7f567c0c9630 msgr2=0x7f567c0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:24.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.329+0000 7f567b400640 1 --2- 172.21.15.6:0/1802686811 >> v2:172.21.15.92:3300/0 conn(0x7f567c0c9630 0x7f567c0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.329+0000 7f567b400640 1 -- 172.21.15.6:0/1802686811 >> v1:172.21.15.6:6789/0 conn(0x7f567c0c2e50 legacy=0x7f567c0c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:24.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.329+0000 7f567b400640 1 -- 172.21.15.6:0/1802686811 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c1582d0 con 0x7f567c15b850 2024-03-22T13:11:24.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.332+0000 7f567b400640 1 --2- 172.21.15.6:0/1802686811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c15bca0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5674009d90 tx=0x7f567402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc524e798df5a9fb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:24.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.334+0000 7f5679600640 1 -- 172.21.15.6:0/1802686811 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5674039070 con 0x7f567c15b850 2024-03-22T13:11:24.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.336+0000 7f5679600640 1 -- 172.21.15.6:0/1802686811 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567402cce0 con 0x7f567c15b850 2024-03-22T13:11:24.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.336+0000 7f5679600640 1 -- 172.21.15.6:0/1802686811 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5674034660 con 0x7f567c15b850 2024-03-22T13:11:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 -- 172.21.15.6:0/1802686811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 msgr2=0x7f567c15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 --2- 172.21.15.6:0/1802686811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c15bca0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5674009d90 tx=0x7f567402bf30 comp rx=0 tx=0).stop 2024-03-22T13:11:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 -- 172.21.15.6:0/1802686811 shutdown_connections 2024-03-22T13:11:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 --2- 172.21.15.6:0/1802686811 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c15bca0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 --2- 172.21.15.6:0/1802686811 >> v2:172.21.15.92:3300/0 conn(0x7f567c0c9630 0x7f567c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 -- 172.21.15.6:0/1802686811 >> 172.21.15.6:0/1802686811 conn(0x7f567c0714f0 msgr2=0x7f567c071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 -- 172.21.15.6:0/1802686811 shutdown_connections 2024-03-22T13:11:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.337+0000 7f5681800640 1 -- 172.21.15.6:0/1802686811 wait complete. 2024-03-22T13:11:24.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.340+0000 7f5681800640 1 Processor -- start 2024-03-22T13:11:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.340+0000 7f5681800640 1 -- start start 2024-03-22T13:11:24.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.341+0000 7f5681800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f567c0c2e50 0x7f567c0c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:24.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.341+0000 7f5681800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f567c0c9630 0x7f567c1f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:24.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.341+0000 7f5681800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:24.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.341+0000 7f5681800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c158b70 con 0x7f567c0c9630 2024-03-22T13:11:24.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.341+0000 7f5681800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c158890 con 0x7f567c0c2e50 2024-03-22T13:11:24.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.341+0000 7f5681800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0ca330 con 0x7f567c15b850 2024-03-22T13:11:24.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.342+0000 7f567a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f567c0c9630 0x7f567c1f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:24.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.342+0000 7f567aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f567c0c2e50 0x7f567c0c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:24.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.343+0000 7f567b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:24.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.343+0000 7f567a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f567c0c9630 0x7f567c1f0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58018/0 (socket says 172.21.15.6:58018) 2024-03-22T13:11:24.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.343+0000 7f567a000640 1 -- 172.21.15.6:0/137261696 learned_addr learned my addr 172.21.15.6:0/137261696 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:24.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.348+0000 7f567aa00640 1 -- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 msgr2=0x7f567c06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:24.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.348+0000 7f567aa00640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c06d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.348+0000 7f567aa00640 1 -- 172.21.15.6:0/137261696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f567c0c9630 msgr2=0x7f567c1f0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:24.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.348+0000 7f567aa00640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f567c0c9630 0x7f567c1f0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.348+0000 7f567aa00640 1 -- 172.21.15.6:0/137261696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0c7ca0 con 0x7f567c0c2e50 2024-03-22T13:11:24.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.353+0000 7f567aa00640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f567c0c2e50 0x7f567c0c7660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f566c00e8b0 tx=0x7f566c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:24.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.357+0000 7f5672a00640 1 -- 172.21.15.6:0/137261696 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c017070 con 0x7f567c0c2e50 2024-03-22T13:11:24.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.357+0000 7f5672a00640 1 -- 172.21.15.6:0/137261696 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c00c940 con 0x7f567c0c2e50 2024-03-22T13:11:24.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.357+0000 7f5672a00640 1 -- 172.21.15.6:0/137261696 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c00cd30 con 0x7f567c0c2e50 2024-03-22T13:11:24.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.358+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c15a820 con 0x7f567c0c2e50 2024-03-22T13:11:24.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.358+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c15ade0 con 0x7f567c0c2e50 2024-03-22T13:11:24.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.359+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c071310 con 0x7f567c0c2e50 2024-03-22T13:11:24.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.641+0000 7f5672a00640 1 -- 172.21.15.6:0/137261696 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f566c01a070 con 0x7f567c0c2e50 2024-03-22T13:11:24.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.643+0000 7f5672a00640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f565c06bc20 0x7f565c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:24.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.644+0000 7f567a000640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f565c06bc20 0x7f565c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:24.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.644+0000 7f5672a00640 1 -- 172.21.15.6:0/137261696 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f566c057650 con 0x7f567c0c2e50 2024-03-22T13:11:24.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.645+0000 7f567a000640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f565c06bc20 0x7f565c06e0e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f567c071f80 tx=0x7f5668007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:24.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.655+0000 7f5672a00640 1 -- 172.21.15.6:0/137261696 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f567c071310 con 0x7f567c0c2e50 2024-03-22T13:11:24.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.951+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f567c0756c0 con 0x7f567c0c2e50 2024-03-22T13:11:24.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.973+0000 7f5672a00640 1 -- 172.21.15.6:0/137261696 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f566c0595b0 con 0x7f567c0c2e50 2024-03-22T13:11:24.973 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:24.973 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:24.974 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:24.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f565c06bc20 msgr2=0x7f565c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:24.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f565c06bc20 0x7f565c06e0e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f567c071f80 tx=0x7f5668007f10 comp rx=0 tx=0).stop 2024-03-22T13:11:24.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f567c0c2e50 msgr2=0x7f567c0c7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:24.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f567c0c2e50 0x7f567c0c7660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f566c00e8b0 tx=0x7f566c00ed70 comp rx=0 tx=0).stop 2024-03-22T13:11:24.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 shutdown_connections 2024-03-22T13:11:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f567c15b850 0x7f567c06d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f567c0c9630 0x7f567c1f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f565c06bc20 0x7f565c06e0e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 --2- 172.21.15.6:0/137261696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f567c0c2e50 0x7f567c0c7660 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 >> 172.21.15.6:0/137261696 conn(0x7f567c0714f0 msgr2=0x7f567c0ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 shutdown_connections 2024-03-22T13:11:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:24.976+0000 7f5681800640 1 -- 172.21.15.6:0/137261696 wait complete. 2024-03-22T13:11:25.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:25.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:26.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:26.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.800+0000 7fec85400640 1 Processor -- start 2024-03-22T13:11:26.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.801+0000 7fec85400640 1 -- start start 2024-03-22T13:11:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.802+0000 7fec85400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fec800c5210 0x7fec80158a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.802+0000 7fec85400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 0x7fec800c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.802+0000 7fec85400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec800c67d0 0x7fec8006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.802+0000 7fec85400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec800c6f90 con 0x7fec800c5210 2024-03-22T13:11:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.802+0000 7fec85400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8006e0f0 con 0x7fec800c67d0 2024-03-22T13:11:26.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.802+0000 7fec85400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8006d350 con 0x7fec800c55f0 2024-03-22T13:11:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.803+0000 7fec84a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec800c67d0 0x7fec8006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.803+0000 7fec7f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 0x7fec800c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.803+0000 7fec84a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec800c67d0 0x7fec8006ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:44824/0 (socket says 172.21.15.6:44824) 2024-03-22T13:11:26.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.803+0000 7fec84a00640 1 -- 172.21.15.6:0/1067540856 learned_addr learned my addr 172.21.15.6:0/1067540856 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.806+0000 7fec84a00640 1 -- 172.21.15.6:0/1067540856 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 msgr2=0x7fec800c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.806+0000 7fec84a00640 1 --2- 172.21.15.6:0/1067540856 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 0x7fec800c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.806+0000 7fec84a00640 1 -- 172.21.15.6:0/1067540856 >> v2:172.21.15.6:3300/0 conn(0x7fec800c5210 msgr2=0x7fec80158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.806+0000 7fec84a00640 1 --2- 172.21.15.6:0/1067540856 >> v2:172.21.15.6:3300/0 conn(0x7fec800c5210 0x7fec80158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.806+0000 7fec84a00640 1 -- 172.21.15.6:0/1067540856 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec800c6cb0 con 0x7fec800c67d0 2024-03-22T13:11:26.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.810+0000 7fec84a00640 1 --2- 172.21.15.6:0/1067540856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c67d0 0x7fec8006ede0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fec80155b20 tx=0x7fec7802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d1b681f7036a44d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:26.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.881+0000 7fec7ea00640 1 -- 172.21.15.6:0/1067540856 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7802ca80 con 0x7fec800c67d0 2024-03-22T13:11:26.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.881+0000 7fec7ea00640 1 -- 172.21.15.6:0/1067540856 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7802cc20 con 0x7fec800c67d0 2024-03-22T13:11:26.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.881+0000 7fec7ea00640 1 -- 172.21.15.6:0/1067540856 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec780343f0 con 0x7fec800c67d0 2024-03-22T13:11:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.882+0000 7fec85400640 1 -- 172.21.15.6:0/1067540856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c67d0 msgr2=0x7fec8006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.882+0000 7fec85400640 1 --2- 172.21.15.6:0/1067540856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c67d0 0x7fec8006ede0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fec80155b20 tx=0x7fec7802bee0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.883+0000 7fec85400640 1 -- 172.21.15.6:0/1067540856 shutdown_connections 2024-03-22T13:11:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.883+0000 7fec85400640 1 --2- 172.21.15.6:0/1067540856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c67d0 0x7fec8006ede0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.883+0000 7fec85400640 1 --2- 172.21.15.6:0/1067540856 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 0x7fec800c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.883+0000 7fec85400640 1 --2- 172.21.15.6:0/1067540856 >> v2:172.21.15.6:3300/0 conn(0x7fec800c5210 0x7fec80158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.883+0000 7fec85400640 1 -- 172.21.15.6:0/1067540856 >> 172.21.15.6:0/1067540856 conn(0x7fec800cb890 msgr2=0x7fec800cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.883+0000 7fec85400640 1 -- 172.21.15.6:0/1067540856 shutdown_connections 2024-03-22T13:11:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.883+0000 7fec85400640 1 -- 172.21.15.6:0/1067540856 wait complete. 2024-03-22T13:11:26.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.885+0000 7fec85400640 1 Processor -- start 2024-03-22T13:11:26.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.886+0000 7fec85400640 1 -- start start 2024-03-22T13:11:26.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.887+0000 7fec85400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c5210 0x7fec80159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:26.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.887+0000 7fec85400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 0x7fec801f0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:26.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.887+0000 7fec85400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec800c67d0 0x7fec8015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:26.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.887+0000 7fec85400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8006dbd0 con 0x7fec800c67d0 2024-03-22T13:11:26.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.887+0000 7fec85400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8006d8f0 con 0x7fec800c5210 2024-03-22T13:11:26.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.887+0000 7fec85400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8006f320 con 0x7fec800c55f0 2024-03-22T13:11:26.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.888+0000 7fec7fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c5210 0x7fec80159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:26.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.888+0000 7fec7fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c5210 0x7fec80159fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:44826/0 (socket says 172.21.15.6:44826) 2024-03-22T13:11:26.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.888+0000 7fec7fe00640 1 -- 172.21.15.6:0/4172627146 learned_addr learned my addr 172.21.15.6:0/4172627146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.891+0000 7fec84a00640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec800c67d0 0x7fec8015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.893+0000 7fec7fe00640 1 -- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 msgr2=0x7fec801f0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.893+0000 7fec7fe00640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 0x7fec801f0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.893+0000 7fec7fe00640 1 -- 172.21.15.6:0/4172627146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec800c67d0 msgr2=0x7fec8015ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.893+0000 7fec7fe00640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec800c67d0 0x7fec8015ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.893+0000 7fec7fe00640 1 -- 172.21.15.6:0/4172627146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80158de0 con 0x7fec800c5210 2024-03-22T13:11:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.896+0000 7fec7fe00640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c5210 0x7fec80159fc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b4a0 tx=0x7fec7000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.901+0000 7fec77e00640 1 -- 172.21.15.6:0/4172627146 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70014070 con 0x7fec800c5210 2024-03-22T13:11:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.901+0000 7fec77e00640 1 -- 172.21.15.6:0/4172627146 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70010610 con 0x7fec800c5210 2024-03-22T13:11:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.901+0000 7fec77e00640 1 -- 172.21.15.6:0/4172627146 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70010a00 con 0x7fec800c5210 2024-03-22T13:11:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.901+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec80145ff0 con 0x7fec800c5210 2024-03-22T13:11:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.901+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8015a500 con 0x7fec800c5210 2024-03-22T13:11:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.902+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec80075650 con 0x7fec800c5210 2024-03-22T13:11:26.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.929+0000 7fec77e00640 1 -- 172.21.15.6:0/4172627146 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fec70017070 con 0x7fec800c5210 2024-03-22T13:11:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.930+0000 7fec77e00640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6006bc20 0x7fec6006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.930+0000 7fec7f400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6006bc20 0x7fec6006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:26.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:26.931+0000 7fec7f400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6006bc20 0x7fec6006e0e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fec6c006a90 tx=0x7fec6c006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:27.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.168+0000 7fec77e00640 1 -- 172.21.15.6:0/4172627146 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fec70056da0 con 0x7fec800c5210 2024-03-22T13:11:27.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.179+0000 7fec77e00640 1 -- 172.21.15.6:0/4172627146 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fec70058d00 con 0x7fec800c5210 2024-03-22T13:11:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.436+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec80155900 con 0x7fec800c5210 2024-03-22T13:11:27.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.456+0000 7fec77e00640 1 -- 172.21.15.6:0/4172627146 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fec7005cda0 con 0x7fec800c5210 2024-03-22T13:11:27.458 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:27.458 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6006bc20 msgr2=0x7fec6006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:27.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6006bc20 0x7fec6006e0e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fec6c006a90 tx=0x7fec6c006a20 comp rx=0 tx=0).stop 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c5210 msgr2=0x7fec80159fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c5210 0x7fec80159fc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b4a0 tx=0x7fec7000b960 comp rx=0 tx=0).stop 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 shutdown_connections 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fec800c67d0 0x7fec8015ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec800c55f0 0x7fec801f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.459+0000 7fec85400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fec6006bc20 0x7fec6006e0e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.460+0000 7fec85400640 1 --2- 172.21.15.6:0/4172627146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec800c5210 0x7fec80159fc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.460+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 >> 172.21.15.6:0/4172627146 conn(0x7fec800cb890 msgr2=0x7fec800ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.460+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 shutdown_connections 2024-03-22T13:11:27.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:27.460+0000 7fec85400640 1 -- 172.21.15.6:0/4172627146 wait complete. 2024-03-22T13:11:28.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:28.081 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:29.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:29.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.281+0000 7f1b5ca00640 1 Processor -- start 2024-03-22T13:11:29.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.281+0000 7f1b5ca00640 1 -- start start 2024-03-22T13:11:29.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.283+0000 7f1b5ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b5806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:29.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.283+0000 7f1b5ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b5815b0e0 0x7f1b5815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:29.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.283+0000 7f1b5ca00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5806c7a0 con 0x7f1b580ca520 2024-03-22T13:11:29.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.283+0000 7f1b5ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580cd000 con 0x7f1b5815b0e0 2024-03-22T13:11:29.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.283+0000 7f1b5ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5806ca80 con 0x7f1b580ccad0 2024-03-22T13:11:29.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.283+0000 7f1b56800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b5815b0e0 0x7f1b5815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:29.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.283+0000 7f1b4fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b5806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:29.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.284+0000 7f1b55e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1b580ca520 0x7f1b580cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:60170/0 (socket says 172.21.15.6:60170) 2024-03-22T13:11:29.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.284+0000 7f1b55e00640 1 -- 172.21.15.6:0/3201151452 learned_addr learned my addr 172.21.15.6:0/3201151452 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:29.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.286+0000 7f1b56800640 1 -- 172.21.15.6:0/3201151452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 msgr2=0x7f1b5806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:29.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.286+0000 7f1b56800640 1 --2- 172.21.15.6:0/3201151452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b5806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.286+0000 7f1b56800640 1 -- 172.21.15.6:0/3201151452 >> v1:172.21.15.92:6789/0 conn(0x7f1b580ca520 legacy=0x7f1b580cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:29.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.286+0000 7f1b56800640 1 -- 172.21.15.6:0/3201151452 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5806cd60 con 0x7f1b5815b0e0 2024-03-22T13:11:29.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.286+0000 7f1b55400640 1 -- 172.21.15.6:0/3201151452 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204436562 0 0) 0x7f1b5806c7a0 con 0x7f1b580ca520 2024-03-22T13:11:29.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.289+0000 7f1b56800640 1 --2- 172.21.15.6:0/3201151452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b5815b980 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009d90 tx=0x7f1b5002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1992a1386ce2dfb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:29.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.295+0000 7f1b55400640 1 -- 172.21.15.6:0/3201151452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50039070 con 0x7f1b5815b0e0 2024-03-22T13:11:29.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.295+0000 7f1b55400640 1 -- 172.21.15.6:0/3201151452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5002cc60 con 0x7f1b5815b0e0 2024-03-22T13:11:29.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.295+0000 7f1b55400640 1 -- 172.21.15.6:0/3201151452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50034660 con 0x7f1b5815b0e0 2024-03-22T13:11:29.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.296+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3201151452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 msgr2=0x7f1b5815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.296+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3201151452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b5815b980 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009d90 tx=0x7f1b5002beb0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.296+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3201151452 shutdown_connections 2024-03-22T13:11:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.296+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3201151452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b5815b980 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.296+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3201151452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b5806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.296+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3201151452 >> 172.21.15.6:0/3201151452 conn(0x7f1b580b16b0 msgr2=0x7f1b580b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.296+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3201151452 shutdown_connections 2024-03-22T13:11:29.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.297+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3201151452 wait complete. 2024-03-22T13:11:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.299+0000 7f1b5ca00640 1 Processor -- start 2024-03-22T13:11:29.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.299+0000 7f1b5ca00640 1 -- start start 2024-03-22T13:11:29.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b5ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b580ca520 0x7f1b581f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:29.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b5ca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b580ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:29.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b5ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b58159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:29.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b5ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58154c20 con 0x7f1b5815b0e0 2024-03-22T13:11:29.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b5ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5806d320 con 0x7f1b580ca520 2024-03-22T13:11:29.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b5ca00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580ca900 con 0x7f1b580ccad0 2024-03-22T13:11:29.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b56800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b58159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:29.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.300+0000 7f1b55e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b580ca520 0x7f1b581f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:29.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.301+0000 7f1b4fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b580ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:29.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.301+0000 7f1b56800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b58159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58078/0 (socket says 172.21.15.6:58078) 2024-03-22T13:11:29.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.301+0000 7f1b56800640 1 -- 172.21.15.6:0/3806087916 learned_addr learned my addr 172.21.15.6:0/3806087916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:29.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.304+0000 7f1b56800640 1 -- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 msgr2=0x7f1b580ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:29.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.304+0000 7f1b56800640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b580ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.304+0000 7f1b56800640 1 -- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b580ca520 msgr2=0x7f1b581f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:29.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.304+0000 7f1b56800640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b580ca520 0x7f1b581f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.304+0000 7f1b56800640 1 -- 172.21.15.6:0/3806087916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5815a650 con 0x7f1b5815b0e0 2024-03-22T13:11:29.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.307+0000 7f1b56800640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b58159360 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009e80 tx=0x7f1b50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:29.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.312+0000 7f1b4e000640 1 -- 172.21.15.6:0/3806087916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50044020 con 0x7f1b5815b0e0 2024-03-22T13:11:29.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.312+0000 7f1b4e000640 1 -- 172.21.15.6:0/3806087916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5003f470 con 0x7f1b5815b0e0 2024-03-22T13:11:29.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.312+0000 7f1b4e000640 1 -- 172.21.15.6:0/3806087916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50039040 con 0x7f1b5815b0e0 2024-03-22T13:11:29.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.312+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b581f3600 con 0x7f1b5815b0e0 2024-03-22T13:11:29.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.312+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b58157140 con 0x7f1b5815b0e0 2024-03-22T13:11:29.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.314+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b581e4790 con 0x7f1b5815b0e0 2024-03-22T13:11:29.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.340+0000 7f1b4e000640 1 -- 172.21.15.6:0/3806087916 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b50047020 con 0x7f1b5815b0e0 2024-03-22T13:11:29.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.341+0000 7f1b4e000640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b3406bc50 0x7f1b3406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:29.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.342+0000 7f1b55e00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b3406bc50 0x7f1b3406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:29.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.342+0000 7f1b55e00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b3406bc50 0x7f1b3406e110 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1b48007f70 tx=0x7f1b48007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:29.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.349+0000 7f1b4e000640 1 -- 172.21.15.6:0/3806087916 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f1b50079eb0 con 0x7f1b5815b0e0 2024-03-22T13:11:29.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.660+0000 7f1b4e000640 1 -- 172.21.15.6:0/3806087916 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b5004e070 con 0x7f1b5815b0e0 2024-03-22T13:11:29.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.928+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b580cb080 con 0x7f1b5815b0e0 2024-03-22T13:11:29.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.973+0000 7f1b4e000640 1 -- 172.21.15.6:0/3806087916 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f1b5007be10 con 0x7f1b5815b0e0 2024-03-22T13:11:29.974 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:29.974 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:29.974 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b3406bc50 msgr2=0x7f1b3406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b3406bc50 0x7f1b3406e110 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1b48007f70 tx=0x7f1b48007f00 comp rx=0 tx=0).stop 2024-03-22T13:11:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 msgr2=0x7f1b58159360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b58159360 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009e80 tx=0x7f1b50004a40 comp rx=0 tx=0).stop 2024-03-22T13:11:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 shutdown_connections 2024-03-22T13:11:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b5815b0e0 0x7f1b58159360 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b580ccad0 0x7f1b580ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f1b3406bc50 0x7f1b3406e110 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 --2- 172.21.15.6:0/3806087916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b580ca520 0x7f1b581f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 >> 172.21.15.6:0/3806087916 conn(0x7f1b580b16b0 msgr2=0x7f1b5815b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 shutdown_connections 2024-03-22T13:11:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:29.976+0000 7f1b5ca00640 1 -- 172.21.15.6:0/3806087916 wait complete. 2024-03-22T13:11:30.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:30.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:31.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.800+0000 7fdf6a800640 1 Processor -- start 2024-03-22T13:11:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.800+0000 7fdf6a800640 1 -- start start 2024-03-22T13:11:31.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.801+0000 7fdf6a800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf640ccad0 0x7fdf6406d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:31.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.801+0000 7fdf6a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf6415b0e0 0x7fdf6415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:31.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.801+0000 7fdf6a800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6406c7a0 con 0x7fdf640ca520 2024-03-22T13:11:31.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.802+0000 7fdf6a800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf640cd000 con 0x7fdf640ccad0 2024-03-22T13:11:31.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.802+0000 7fdf6a800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6406ca80 con 0x7fdf6415b0e0 2024-03-22T13:11:31.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.802+0000 7fdf5aa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdf640ccad0 0x7fdf6406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:31.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.802+0000 7fdf63e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf6415b0e0 0x7fdf6415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:31.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.803+0000 7fdf63e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf6415b0e0 0x7fdf6415b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53978/0 (socket says 172.21.15.6:53978) 2024-03-22T13:11:31.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.803+0000 7fdf63e00640 1 -- 172.21.15.6:0/3787288253 learned_addr learned my addr 172.21.15.6:0/3787288253 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.806+0000 7fdf63e00640 1 -- 172.21.15.6:0/3787288253 >> v2:172.21.15.92:3300/0 conn(0x7fdf640ccad0 msgr2=0x7fdf6406d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.806+0000 7fdf63e00640 1 --2- 172.21.15.6:0/3787288253 >> v2:172.21.15.92:3300/0 conn(0x7fdf640ccad0 0x7fdf6406d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.806+0000 7fdf63e00640 1 -- 172.21.15.6:0/3787288253 >> v1:172.21.15.6:6789/0 conn(0x7fdf640ca520 legacy=0x7fdf640cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:31.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.806+0000 7fdf63e00640 1 -- 172.21.15.6:0/3787288253 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6406cd60 con 0x7fdf6415b0e0 2024-03-22T13:11:31.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.812+0000 7fdf63e00640 1 --2- 172.21.15.6:0/3787288253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf6415b0e0 0x7fdf6415b980 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009870 tx=0x7fdf5c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b5bfd547b61953 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.817+0000 7fdf62a00640 1 -- 172.21.15.6:0/3787288253 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c034070 con 0x7fdf6415b0e0 2024-03-22T13:11:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.817+0000 7fdf62a00640 1 -- 172.21.15.6:0/3787288253 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c030840 con 0x7fdf6415b0e0 2024-03-22T13:11:31.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.817+0000 7fdf62a00640 1 -- 172.21.15.6:0/3787288253 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c030c10 con 0x7fdf6415b0e0 2024-03-22T13:11:31.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.818+0000 7fdf6a800640 1 -- 172.21.15.6:0/3787288253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf6415b0e0 msgr2=0x7fdf6415b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.818+0000 7fdf6a800640 1 --2- 172.21.15.6:0/3787288253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf6415b0e0 0x7fdf6415b980 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009870 tx=0x7fdf5c02b980 comp rx=0 tx=0).stop 2024-03-22T13:11:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.818+0000 7fdf6a800640 1 -- 172.21.15.6:0/3787288253 shutdown_connections 2024-03-22T13:11:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.818+0000 7fdf6a800640 1 --2- 172.21.15.6:0/3787288253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf6415b0e0 0x7fdf6415b980 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.818+0000 7fdf6a800640 1 --2- 172.21.15.6:0/3787288253 >> v2:172.21.15.92:3300/0 conn(0x7fdf640ccad0 0x7fdf6406d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.818+0000 7fdf6a800640 1 -- 172.21.15.6:0/3787288253 >> 172.21.15.6:0/3787288253 conn(0x7fdf640b16b0 msgr2=0x7fdf640b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.818+0000 7fdf6a800640 1 -- 172.21.15.6:0/3787288253 shutdown_connections 2024-03-22T13:11:31.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.819+0000 7fdf6a800640 1 -- 172.21.15.6:0/3787288253 wait complete. 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.820+0000 7fdf6a800640 1 Processor -- start 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.820+0000 7fdf6a800640 1 -- start start 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.821+0000 7fdf6a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf640ca520 0x7fdf64157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.821+0000 7fdf6a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf640ccad0 0x7fdf641569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.821+0000 7fdf6a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf6415b0e0 0x7fdf64156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.821+0000 7fdf6a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64154c20 con 0x7fdf6415b0e0 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.821+0000 7fdf6a800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6406d320 con 0x7fdf640ca520 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.821+0000 7fdf6a800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf640ca900 con 0x7fdf640ccad0 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.822+0000 7fdf5aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf640ccad0 0x7fdf641569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.822+0000 7fdf63400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf640ca520 0x7fdf64157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.822+0000 7fdf5aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf640ccad0 0x7fdf641569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:53986/0 (socket says 172.21.15.6:53986) 2024-03-22T13:11:31.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.822+0000 7fdf5aa00640 1 -- 172.21.15.6:0/782311687 learned_addr learned my addr 172.21.15.6:0/782311687 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:31.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.824+0000 7fdf63e00640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf6415b0e0 0x7fdf64156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.826+0000 7fdf5aa00640 1 -- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf640ca520 msgr2=0x7fdf64157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.826+0000 7fdf5aa00640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf640ca520 0x7fdf64157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.826+0000 7fdf5aa00640 1 -- 172.21.15.6:0/782311687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf6415b0e0 msgr2=0x7fdf64156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.826+0000 7fdf5aa00640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf6415b0e0 0x7fdf64156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.827+0000 7fdf5aa00640 1 -- 172.21.15.6:0/782311687 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf641559c0 con 0x7fdf640ccad0 2024-03-22T13:11:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.831+0000 7fdf5aa00640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf640ccad0 0x7fdf641569f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00e9b0 tx=0x7fdf4c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.886+0000 7fdf5be00640 1 -- 172.21.15.6:0/782311687 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c017070 con 0x7fdf640ccad0 2024-03-22T13:11:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.887+0000 7fdf5be00640 1 -- 172.21.15.6:0/782311687 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c00c9a0 con 0x7fdf640ccad0 2024-03-22T13:11:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.887+0000 7fdf5be00640 1 -- 172.21.15.6:0/782311687 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c00cd70 con 0x7fdf640ccad0 2024-03-22T13:11:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.887+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf641f1280 con 0x7fdf640ccad0 2024-03-22T13:11:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.887+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf64155f10 con 0x7fdf640ccad0 2024-03-22T13:11:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:31.888+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf640756c0 con 0x7fdf640ccad0 2024-03-22T13:11:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.187+0000 7fdf5be00640 1 -- 172.21.15.6:0/782311687 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf4c01a070 con 0x7fdf640ccad0 2024-03-22T13:11:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.189+0000 7fdf5be00640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf4006bc50 0x7fdf4006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.189+0000 7fdf5be00640 1 -- 172.21.15.6:0/782311687 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdf4c08b5e0 con 0x7fdf640ccad0 2024-03-22T13:11:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.190+0000 7fdf63400640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf4006bc50 0x7fdf4006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.191+0000 7fdf63400640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf4006bc50 0x7fdf4006e110 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdf54001f10 tx=0x7fdf54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:32.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.200+0000 7fdf5be00640 1 -- 172.21.15.6:0/782311687 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf4c059640 con 0x7fdf640ccad0 2024-03-22T13:11:32.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.454+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf640cb080 con 0x7fdf640ccad0 2024-03-22T13:11:32.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.468+0000 7fdf5be00640 1 -- 172.21.15.6:0/782311687 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fdf4c05d6e0 con 0x7fdf640ccad0 2024-03-22T13:11:32.468 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:32.468 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:32.469 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf4006bc50 msgr2=0x7fdf4006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf4006bc50 0x7fdf4006e110 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdf54001f10 tx=0x7fdf54009040 comp rx=0 tx=0).stop 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf640ccad0 msgr2=0x7fdf641569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf640ccad0 0x7fdf641569f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00e9b0 tx=0x7fdf4c00ee70 comp rx=0 tx=0).stop 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 shutdown_connections 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf6415b0e0 0x7fdf64156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf640ccad0 0x7fdf641569f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf4006bc50 0x7fdf4006e110 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 --2- 172.21.15.6:0/782311687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf640ca520 0x7fdf64157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 >> 172.21.15.6:0/782311687 conn(0x7fdf640b16b0 msgr2=0x7fdf6415a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 shutdown_connections 2024-03-22T13:11:32.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:32.471+0000 7fdf6a800640 1 -- 172.21.15.6:0/782311687 wait complete. 2024-03-22T13:11:33.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:33.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:33.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:11:33.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:11:34.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.295+0000 7f541c600640 1 Processor -- start 2024-03-22T13:11:34.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.295+0000 7f541c600640 1 -- start start 2024-03-22T13:11:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.297+0000 7f541c600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f54140cc3b0 0x7f54140cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.297+0000 7f541c600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54140ccea0 0x7f541415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.297+0000 7f541c600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140ce230 con 0x7f54140cf770 2024-03-22T13:11:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.297+0000 7f541c600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54140ce510 con 0x7f54140ccea0 2024-03-22T13:11:34.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.297+0000 7f541c600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54140ce7f0 con 0x7f54140cc3b0 2024-03-22T13:11:34.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.298+0000 7f5419c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f54140cc3b0 0x7f54140cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:34.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.298+0000 7f541a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54140ccea0 0x7f541415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:34.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.299+0000 7f5419200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f54140cf770 0x7f54140cc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38166/0 (socket says 172.21.15.6:38166) 2024-03-22T13:11:34.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.299+0000 7f5419200640 1 -- 172.21.15.6:0/406958101 learned_addr learned my addr 172.21.15.6:0/406958101 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.301+0000 7f5419c00640 1 -- 172.21.15.6:0/406958101 >> v2:172.21.15.6:3300/0 conn(0x7f54140ccea0 msgr2=0x7f541415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.301+0000 7f5419c00640 1 --2- 172.21.15.6:0/406958101 >> v2:172.21.15.6:3300/0 conn(0x7f54140ccea0 0x7f541415ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.301+0000 7f5419c00640 1 -- 172.21.15.6:0/406958101 >> v1:172.21.15.6:6789/0 conn(0x7f54140cf770 legacy=0x7f54140cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:34.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.301+0000 7f5419c00640 1 -- 172.21.15.6:0/406958101 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140cead0 con 0x7f54140cc3b0 2024-03-22T13:11:34.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.303+0000 7f5419c00640 1 --2- 172.21.15.6:0/406958101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f54140cf230 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5414073780 tx=0x7f540802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=479c1c52959fb766 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.308+0000 7f5413e00640 1 -- 172.21.15.6:0/406958101 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408039070 con 0x7f54140cc3b0 2024-03-22T13:11:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.308+0000 7f5413e00640 1 -- 172.21.15.6:0/406958101 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540802cbf0 con 0x7f54140cc3b0 2024-03-22T13:11:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.308+0000 7f5413e00640 1 -- 172.21.15.6:0/406958101 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408034440 con 0x7f54140cc3b0 2024-03-22T13:11:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.308+0000 7f541c600640 1 -- 172.21.15.6:0/406958101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 msgr2=0x7f54140cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.308+0000 7f541c600640 1 --2- 172.21.15.6:0/406958101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f54140cf230 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5414073780 tx=0x7f540802be40 comp rx=0 tx=0).stop 2024-03-22T13:11:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.309+0000 7f541c600640 1 -- 172.21.15.6:0/406958101 shutdown_connections 2024-03-22T13:11:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.309+0000 7f541c600640 1 --2- 172.21.15.6:0/406958101 >> v2:172.21.15.6:3300/0 conn(0x7f54140ccea0 0x7f541415ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.309+0000 7f541c600640 1 --2- 172.21.15.6:0/406958101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f54140cf230 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.309+0000 7f541c600640 1 -- 172.21.15.6:0/406958101 >> 172.21.15.6:0/406958101 conn(0x7f54140b16d0 msgr2=0x7f54140b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.309+0000 7f541c600640 1 -- 172.21.15.6:0/406958101 shutdown_connections 2024-03-22T13:11:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.309+0000 7f541c600640 1 -- 172.21.15.6:0/406958101 wait complete. 2024-03-22T13:11:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.312+0000 7f541c600640 1 Processor -- start 2024-03-22T13:11:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.312+0000 7f541c600640 1 -- start start 2024-03-22T13:11:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.313+0000 7f541c600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f5414153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.313+0000 7f541c600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54140ccea0 0x7f54140b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.313+0000 7f541c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54140cf770 0x7f54140b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.313+0000 7f541c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541415a5d0 con 0x7f54140cf770 2024-03-22T13:11:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.313+0000 7f541c600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541406dc10 con 0x7f54140cc3b0 2024-03-22T13:11:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.313+0000 7f541c600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5414154bb0 con 0x7f54140ccea0 2024-03-22T13:11:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.314+0000 7f5419200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54140ccea0 0x7f54140b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.314+0000 7f5419c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f5414153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.314+0000 7f541a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54140cf770 0x7f54140b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.314+0000 7f5419200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54140ccea0 0x7f54140b2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34898/0 (socket says 172.21.15.6:34898) 2024-03-22T13:11:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.314+0000 7f5419200640 1 -- 172.21.15.6:0/1105467106 learned_addr learned my addr 172.21.15.6:0/1105467106 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.319+0000 7f5419200640 1 -- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 msgr2=0x7f5414153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.319+0000 7f5419200640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f5414153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.319+0000 7f5419200640 1 -- 172.21.15.6:0/1105467106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54140cf770 msgr2=0x7f54140b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.319+0000 7f5419200640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54140cf770 0x7f54140b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.319+0000 7f5419200640 1 -- 172.21.15.6:0/1105467106 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140c5c90 con 0x7f54140ccea0 2024-03-22T13:11:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.319+0000 7f5419c00640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f5414153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:11:34.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.323+0000 7f5419200640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54140ccea0 0x7f54140b2dc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f540000c470 tx=0x7f540000c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.325+0000 7f5411600640 1 -- 172.21.15.6:0/1105467106 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400017070 con 0x7f54140ccea0 2024-03-22T13:11:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.325+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54140c5f30 con 0x7f54140ccea0 2024-03-22T13:11:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.326+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541414e8a0 con 0x7f54140ccea0 2024-03-22T13:11:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.327+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541415a210 con 0x7f54140ccea0 2024-03-22T13:11:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.327+0000 7f5411600640 1 -- 172.21.15.6:0/1105467106 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5400005ce0 con 0x7f54140ccea0 2024-03-22T13:11:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.327+0000 7f5411600640 1 -- 172.21.15.6:0/1105467106 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400021060 con 0x7f54140ccea0 2024-03-22T13:11:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.608+0000 7f5411600640 1 -- 172.21.15.6:0/1105467106 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f540001a070 con 0x7f54140ccea0 2024-03-22T13:11:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.610+0000 7f5411600640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f406bc00 0x7f53f406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.610+0000 7f5419c00640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f406bc00 0x7f53f406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.610+0000 7f5411600640 1 -- 172.21.15.6:0/1105467106 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5400056aa0 con 0x7f54140ccea0 2024-03-22T13:11:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.611+0000 7f5419c00640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f406bc00 0x7f53f406e0c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5408036000 tx=0x7f5408030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:34.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.629+0000 7f5411600640 1 -- 172.21.15.6:0/1105467106 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f541415a210 con 0x7f54140ccea0 2024-03-22T13:11:34.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.867+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54140756c0 con 0x7f54140ccea0 2024-03-22T13:11:34.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.890+0000 7f5411600640 1 -- 172.21.15.6:0/1105467106 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f5400058a00 con 0x7f54140ccea0 2024-03-22T13:11:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:34.890 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:34.891 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f406bc00 msgr2=0x7f53f406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f406bc00 0x7f53f406e0c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5408036000 tx=0x7f5408030000 comp rx=0 tx=0).stop 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54140ccea0 msgr2=0x7f54140b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54140ccea0 0x7f54140b2dc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f540000c470 tx=0x7f540000c930 comp rx=0 tx=0).stop 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 shutdown_connections 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54140cf770 0x7f54140b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54140ccea0 0x7f54140b2dc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f53f406bc00 0x7f53f406e0c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 --2- 172.21.15.6:0/1105467106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54140cc3b0 0x7f5414153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 >> 172.21.15.6:0/1105467106 conn(0x7f54140b16d0 msgr2=0x7f541406cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:34.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 shutdown_connections 2024-03-22T13:11:34.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:34.893+0000 7f541c600640 1 -- 172.21.15.6:0/1105467106 wait complete. 2024-03-22T13:11:35.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:35.516 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:36.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:36.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.711+0000 7fb401c00640 1 Processor -- start 2024-03-22T13:11:36.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.711+0000 7fb401c00640 1 -- start start 2024-03-22T13:11:36.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.712+0000 7fb401c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3fc1589e0 0x7fb3fc15f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:36.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.712+0000 7fb401c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc072c10 0x7fb3fc1623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:36.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.712+0000 7fb401c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0715b0 con 0x7fb3fc1628f0 2024-03-22T13:11:36.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.712+0000 7fb401c00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc0c6dd0 con 0x7fb3fc1589e0 2024-03-22T13:11:36.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.712+0000 7fb401c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc161f20 con 0x7fb3fc072c10 2024-03-22T13:11:36.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.713+0000 7fb3fb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc072c10 0x7fb3fc1623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:36.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.713+0000 7fb3fbe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3fc1589e0 0x7fb3fc15f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:36.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.713+0000 7fb3fb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc072c10 0x7fb3fc1623b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34904/0 (socket says 172.21.15.6:34904) 2024-03-22T13:11:36.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.713+0000 7fb3fb400640 1 -- 172.21.15.6:0/2940354114 learned_addr learned my addr 172.21.15.6:0/2940354114 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:36.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.716+0000 7fb3fb400640 1 -- 172.21.15.6:0/2940354114 >> v2:172.21.15.92:3300/0 conn(0x7fb3fc1589e0 msgr2=0x7fb3fc15f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:36.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.716+0000 7fb3fb400640 1 --2- 172.21.15.6:0/2940354114 >> v2:172.21.15.92:3300/0 conn(0x7fb3fc1589e0 0x7fb3fc15f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:36.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.716+0000 7fb3fb400640 1 -- 172.21.15.6:0/2940354114 >> v1:172.21.15.6:6789/0 conn(0x7fb3fc1628f0 legacy=0x7fb3fc06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:36.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.716+0000 7fb3fb400640 1 -- 172.21.15.6:0/2940354114 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc071890 con 0x7fb3fc072c10 2024-03-22T13:11:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.719+0000 7fb3fb400640 1 --2- 172.21.15.6:0/2940354114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc072c10 0x7fb3fc1623b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40098a0 tx=0x7fb3e402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d922bd2d8adc66de server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:36.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.724+0000 7fb3faa00640 1 -- 172.21.15.6:0/2940354114 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4035020 con 0x7fb3fc072c10 2024-03-22T13:11:36.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.724+0000 7fb3faa00640 1 -- 172.21.15.6:0/2940354114 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4031640 con 0x7fb3fc072c10 2024-03-22T13:11:36.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.724+0000 7fb3faa00640 1 -- 172.21.15.6:0/2940354114 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e402d040 con 0x7fb3fc072c10 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 -- 172.21.15.6:0/2940354114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc072c10 msgr2=0x7fb3fc1623b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 --2- 172.21.15.6:0/2940354114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc072c10 0x7fb3fc1623b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40098a0 tx=0x7fb3e402b8f0 comp rx=0 tx=0).stop 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 -- 172.21.15.6:0/2940354114 shutdown_connections 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 --2- 172.21.15.6:0/2940354114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc072c10 0x7fb3fc1623b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 --2- 172.21.15.6:0/2940354114 >> v2:172.21.15.92:3300/0 conn(0x7fb3fc1589e0 0x7fb3fc15f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 -- 172.21.15.6:0/2940354114 >> 172.21.15.6:0/2940354114 conn(0x7fb3fc0cb890 msgr2=0x7fb3fc0cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 -- 172.21.15.6:0/2940354114 shutdown_connections 2024-03-22T13:11:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.725+0000 7fb401c00640 1 -- 172.21.15.6:0/2940354114 wait complete. 2024-03-22T13:11:36.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.727+0000 7fb401c00640 1 Processor -- start 2024-03-22T13:11:36.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.727+0000 7fb401c00640 1 -- start start 2024-03-22T13:11:36.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.728+0000 7fb401c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3fc072c10 0x7fb3fc160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:36.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.728+0000 7fb401c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3fc1589e0 0x7fb3fc149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:36.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.729+0000 7fb401c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc1628f0 0x7fb3fc148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.729+0000 7fb401c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc072450 con 0x7fb3fc072c10 2024-03-22T13:11:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.729+0000 7fb401c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc15f670 con 0x7fb3fc1589e0 2024-03-22T13:11:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.729+0000 7fb401c00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc158dc0 con 0x7fb3fc1628f0 2024-03-22T13:11:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.729+0000 7fb401200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc1628f0 0x7fb3fc148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.729+0000 7fb3fb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3fc1589e0 0x7fb3fc149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.730+0000 7fb401200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc1628f0 0x7fb3fc148790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34918/0 (socket says 172.21.15.6:34918) 2024-03-22T13:11:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.730+0000 7fb401200640 1 -- 172.21.15.6:0/2406058379 learned_addr learned my addr 172.21.15.6:0/2406058379 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:36.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.733+0000 7fb3fbe00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3fc072c10 0x7fb3fc160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:36.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.733+0000 7fb401200640 1 -- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3fc1589e0 msgr2=0x7fb3fc149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:36.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.733+0000 7fb401200640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3fc1589e0 0x7fb3fc149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:36.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.733+0000 7fb401200640 1 -- 172.21.15.6:0/2406058379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3fc072c10 msgr2=0x7fb3fc160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:36.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.733+0000 7fb401200640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3fc072c10 0x7fb3fc160d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:36.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.733+0000 7fb401200640 1 -- 172.21.15.6:0/2406058379 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc1f48f0 con 0x7fb3fc1628f0 2024-03-22T13:11:36.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.736+0000 7fb401200640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc1628f0 0x7fb3fc148790 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c780 tx=0x7fb3f400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.739+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2406058379 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4017070 con 0x7fb3fc1628f0 2024-03-22T13:11:36.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.739+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc1f4bb0 con 0x7fb3fc1628f0 2024-03-22T13:11:36.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.739+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc1f5170 con 0x7fb3fc1628f0 2024-03-22T13:11:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.740+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc075650 con 0x7fb3fc1628f0 2024-03-22T13:11:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.741+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2406058379 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f400ce30 con 0x7fb3fc1628f0 2024-03-22T13:11:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:36.741+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2406058379 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4021060 con 0x7fb3fc1628f0 2024-03-22T13:11:37.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.003+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2406058379 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb3f401a070 con 0x7fb3fc1628f0 2024-03-22T13:11:37.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.005+0000 7fb3f3e00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d806bc00 0x7fb3d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:37.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.005+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2406058379 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb3f4005610 con 0x7fb3fc1628f0 2024-03-22T13:11:37.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.006+0000 7fb3fbe00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d806bc00 0x7fb3d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:37.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.007+0000 7fb3fbe00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d806bc00 0x7fb3d806e0c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009ad0 tx=0x7fb3ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:37.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.016+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2406058379 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb3f4058950 con 0x7fb3fc1628f0 2024-03-22T13:11:37.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.283+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3fc1580c0 con 0x7fb3fc1628f0 2024-03-22T13:11:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.372+0000 7fb3f3e00640 1 -- 172.21.15.6:0/2406058379 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fb3f405c9f0 con 0x7fb3fc1628f0 2024-03-22T13:11:37.372 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:37.372 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:37.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d806bc00 msgr2=0x7fb3d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:37.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d806bc00 0x7fb3d806e0c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009ad0 tx=0x7fb3ec008040 comp rx=0 tx=0).stop 2024-03-22T13:11:37.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc1628f0 msgr2=0x7fb3fc148790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc1628f0 0x7fb3fc148790 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c780 tx=0x7fb3f400cc40 comp rx=0 tx=0).stop 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 shutdown_connections 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3fc1628f0 0x7fb3fc148790 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb3d806bc00 0x7fb3d806e0c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3fc1589e0 0x7fb3fc149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 --2- 172.21.15.6:0/2406058379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3fc072c10 0x7fb3fc160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.374+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 >> 172.21.15.6:0/2406058379 conn(0x7fb3fc0cb890 msgr2=0x7fb3fc0c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.375+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 shutdown_connections 2024-03-22T13:11:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:37.375+0000 7fb401c00640 1 -- 172.21.15.6:0/2406058379 wait complete. 2024-03-22T13:11:37.996 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:37.997 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:38.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:39.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.194+0000 7f8d6e400640 1 Processor -- start 2024-03-22T13:11:39.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.194+0000 7f8d6e400640 1 -- start start 2024-03-22T13:11:39.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.195+0000 7f8d6e400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8d6806c230 0x7f8d6815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:39.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.195+0000 7f8d6e400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6806c640 con 0x7f8d680ce8a0 2024-03-22T13:11:39.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.195+0000 7f8d6e400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d680cee00 con 0x7f8d680cc380 2024-03-22T13:11:39.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.196+0000 7f8d6e400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d680cf0c0 con 0x7f8d6806c230 2024-03-22T13:11:39.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.196+0000 7f8d6da00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8d6806c230 0x7f8d6815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:39.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.196+0000 7f8d67e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8d680ce8a0 0x7f8d68162b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38180/0 (socket says 172.21.15.6:38180) 2024-03-22T13:11:39.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.196+0000 7f8d67e00640 1 -- 172.21.15.6:0/2248878677 learned_addr learned my addr 172.21.15.6:0/2248878677 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.200+0000 7f8d67400640 1 -- 172.21.15.6:0/2248878677 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353832973 0 0) 0x7f8d54001110 con 0x7f8d680ce8a0 2024-03-22T13:11:39.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.200+0000 7f8d67400640 1 -- 172.21.15.6:0/2248878677 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d50003520 con 0x7f8d680ce8a0 2024-03-22T13:11:39.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.201+0000 7f8d6da00640 1 -- 172.21.15.6:0/2248878677 >> v1:172.21.15.92:6789/0 conn(0x7f8d680cc380 legacy=0x7f8d680cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:39.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.201+0000 7f8d6da00640 1 -- 172.21.15.6:0/2248878677 >> v1:172.21.15.6:6789/0 conn(0x7f8d680ce8a0 legacy=0x7f8d68162b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:39.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.201+0000 7f8d6da00640 1 -- 172.21.15.6:0/2248878677 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d680cf3a0 con 0x7f8d6806c230 2024-03-22T13:11:39.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.207+0000 7f8d6da00640 1 --2- 172.21.15.6:0/2248878677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d6806c230 0x7f8d6815ae30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8d680cbef0 tx=0x7f8d6002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a99583a287bda89 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.211+0000 7f8d67400640 1 -- 172.21.15.6:0/2248878677 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60039070 con 0x7f8d6806c230 2024-03-22T13:11:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.211+0000 7f8d67400640 1 -- 172.21.15.6:0/2248878677 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6002cce0 con 0x7f8d6806c230 2024-03-22T13:11:39.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.211+0000 7f8d67400640 1 -- 172.21.15.6:0/2248878677 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60034660 con 0x7f8d6806c230 2024-03-22T13:11:39.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.212+0000 7f8d6e400640 1 -- 172.21.15.6:0/2248878677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d6806c230 msgr2=0x7f8d6815ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:39.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.212+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2248878677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d6806c230 0x7f8d6815ae30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8d680cbef0 tx=0x7f8d6002bf30 comp rx=0 tx=0).stop 2024-03-22T13:11:39.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.212+0000 7f8d6e400640 1 -- 172.21.15.6:0/2248878677 shutdown_connections 2024-03-22T13:11:39.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.212+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2248878677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d6806c230 0x7f8d6815ae30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:39.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.212+0000 7f8d6e400640 1 -- 172.21.15.6:0/2248878677 >> 172.21.15.6:0/2248878677 conn(0x7f8d680b1480 msgr2=0x7f8d680b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:39.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.212+0000 7f8d6e400640 1 -- 172.21.15.6:0/2248878677 shutdown_connections 2024-03-22T13:11:39.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.212+0000 7f8d6e400640 1 -- 172.21.15.6:0/2248878677 wait complete. 2024-03-22T13:11:39.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.215+0000 7f8d6e400640 1 Processor -- start 2024-03-22T13:11:39.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.215+0000 7f8d6e400640 1 -- start start 2024-03-22T13:11:39.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.216+0000 7f8d6e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d6806c230 0x7f8d6814eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:39.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.216+0000 7f8d6e400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d680cc380 0x7f8d6815c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:39.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.216+0000 7f8d6e400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d680ce8a0 0x7f8d6815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:39.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.216+0000 7f8d6e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6815a910 con 0x7f8d6806c230 2024-03-22T13:11:39.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.216+0000 7f8d6e400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6815a630 con 0x7f8d680ce8a0 2024-03-22T13:11:39.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.216+0000 7f8d6e400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d681549d0 con 0x7f8d680cc380 2024-03-22T13:11:39.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.217+0000 7f8d6da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d680ce8a0 0x7f8d6815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:39.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.217+0000 7f8d67e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d680cc380 0x7f8d6815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:39.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.217+0000 7f8d6d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d6806c230 0x7f8d6814eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:39.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.218+0000 7f8d67e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d680cc380 0x7f8d6815c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34924/0 (socket says 172.21.15.6:34924) 2024-03-22T13:11:39.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.218+0000 7f8d6da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d680ce8a0 0x7f8d6815c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:43030/0 (socket says 172.21.15.6:43030) 2024-03-22T13:11:39.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.218+0000 7f8d67e00640 1 -- 172.21.15.6:0/2772515989 learned_addr learned my addr 172.21.15.6:0/2772515989 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:39.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.222+0000 7f8d6da00640 1 -- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d680cc380 msgr2=0x7f8d6815c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:39.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.222+0000 7f8d6da00640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d680cc380 0x7f8d6815c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:39.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.222+0000 7f8d6da00640 1 -- 172.21.15.6:0/2772515989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d6806c230 msgr2=0x7f8d6814eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:39.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.222+0000 7f8d6da00640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d6806c230 0x7f8d6814eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:39.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.222+0000 7f8d6da00640 1 -- 172.21.15.6:0/2772515989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6815de30 con 0x7f8d680ce8a0 2024-03-22T13:11:39.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.222+0000 7f8d6d000640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d6806c230 0x7f8d6814eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:11:39.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.224+0000 7f8d6da00640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d680ce8a0 0x7f8d6815c5d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8d60033000 tx=0x7f8d60004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:39.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.226+0000 7f8d64c00640 1 -- 172.21.15.6:0/2772515989 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60044020 con 0x7f8d680ce8a0 2024-03-22T13:11:39.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.227+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6815e0d0 con 0x7f8d680ce8a0 2024-03-22T13:11:39.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.227+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6815adc0 con 0x7f8d680ce8a0 2024-03-22T13:11:39.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.228+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d68075650 con 0x7f8d680ce8a0 2024-03-22T13:11:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.433+0000 7f8d64c00640 1 -- 172.21.15.6:0/2772515989 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6002cce0 con 0x7f8d680ce8a0 2024-03-22T13:11:39.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.531+0000 7f8d64c00640 1 -- 172.21.15.6:0/2772515989 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60039040 con 0x7f8d680ce8a0 2024-03-22T13:11:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.534+0000 7f8d64c00640 1 -- 172.21.15.6:0/2772515989 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8d60047020 con 0x7f8d680ce8a0 2024-03-22T13:11:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.536+0000 7f8d64c00640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8d4806bc00 0x7f8d4806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.537+0000 7f8d6d000640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8d4806bc00 0x7f8d4806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:39.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.537+0000 7f8d6d000640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8d4806bc00 0x7f8d4806e0c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c001b70 tx=0x7f8d5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.538+0000 7f8d64c00640 1 -- 172.21.15.6:0/2772515989 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f8d60079d20 con 0x7f8d680ce8a0 2024-03-22T13:11:39.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.557+0000 7f8d64c00640 1 -- 172.21.15.6:0/2772515989 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8d6004e070 con 0x7f8d680ce8a0 2024-03-22T13:11:39.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.809+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d68145e10 con 0x7f8d680ce8a0 2024-03-22T13:11:39.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.833+0000 7f8d64c00640 1 -- 172.21.15.6:0/2772515989 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f8d6007bc80 con 0x7f8d680ce8a0 2024-03-22T13:11:39.833 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:39.834 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:39.834 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.836+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8d4806bc00 msgr2=0x7f8d4806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8d4806bc00 0x7f8d4806e0c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c001b70 tx=0x7f8d5c009040 comp rx=0 tx=0).stop 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d680ce8a0 msgr2=0x7f8d6815c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d680ce8a0 0x7f8d6815c5d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8d60033000 tx=0x7f8d60004a40 comp rx=0 tx=0).stop 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 shutdown_connections 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f8d4806bc00 0x7f8d4806e0c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d680ce8a0 0x7f8d6815c5d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d680cc380 0x7f8d6815c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 --2- 172.21.15.6:0/2772515989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d6806c230 0x7f8d6814eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 >> 172.21.15.6:0/2772515989 conn(0x7f8d680b1480 msgr2=0x7f8d681498e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:39.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 shutdown_connections 2024-03-22T13:11:39.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:39.837+0000 7f8d6e400640 1 -- 172.21.15.6:0/2772515989 wait complete. 2024-03-22T13:11:40.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:40.460 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:41.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:41.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.665+0000 7fdf74e00640 1 Processor -- start 2024-03-22T13:11:41.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.665+0000 7fdf74e00640 1 -- start start 2024-03-22T13:11:41.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.667+0000 7fdf74e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf7006f8c0 0x7fdf700716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:41.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.667+0000 7fdf74e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdf70071c10 0x7fdf700c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:41.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.667+0000 7fdf74e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf701590b0 con 0x7fdf70152400 2024-03-22T13:11:41.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.667+0000 7fdf74e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf700724d0 con 0x7fdf70071c10 2024-03-22T13:11:41.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.667+0000 7fdf74e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf700721f0 con 0x7fdf7006f8c0 2024-03-22T13:11:41.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.668+0000 7fdf6d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf7006f8c0 0x7fdf700716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:41.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.668+0000 7fdf6d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf7006f8c0 0x7fdf700716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34934/0 (socket says 172.21.15.6:34934) 2024-03-22T13:11:41.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.668+0000 7fdf6d400640 1 -- 172.21.15.6:0/3386671115 learned_addr learned my addr 172.21.15.6:0/3386671115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:41.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.668+0000 7fdf6e800640 1 --2- 172.21.15.6:0/3386671115 >> v2:172.21.15.6:3300/0 conn(0x7fdf70071c10 0x7fdf700c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:41.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.670+0000 7fdf6d400640 1 -- 172.21.15.6:0/3386671115 >> v2:172.21.15.6:3300/0 conn(0x7fdf70071c10 msgr2=0x7fdf700c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:41.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.670+0000 7fdf6d400640 1 --2- 172.21.15.6:0/3386671115 >> v2:172.21.15.6:3300/0 conn(0x7fdf70071c10 0x7fdf700c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:41.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.670+0000 7fdf6d400640 1 -- 172.21.15.6:0/3386671115 >> v1:172.21.15.6:6789/0 conn(0x7fdf70152400 legacy=0x7fdf7015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:41.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.670+0000 7fdf6d400640 1 -- 172.21.15.6:0/3386671115 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf701582f0 con 0x7fdf7006f8c0 2024-03-22T13:11:41.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.672+0000 7fdf6d400640 1 --2- 172.21.15.6:0/3386671115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf7006f8c0 0x7fdf700716d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009870 tx=0x7fdf5c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d30880b1da53b269 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:41.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.675+0000 7fdf6ca00640 1 -- 172.21.15.6:0/3386671115 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c02f830 con 0x7fdf7006f8c0 2024-03-22T13:11:41.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.677+0000 7fdf6ca00640 1 -- 172.21.15.6:0/3386671115 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c02f9d0 con 0x7fdf7006f8c0 2024-03-22T13:11:41.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.677+0000 7fdf6ca00640 1 -- 172.21.15.6:0/3386671115 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c02fda0 con 0x7fdf7006f8c0 2024-03-22T13:11:41.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 -- 172.21.15.6:0/3386671115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf7006f8c0 msgr2=0x7fdf700716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:41.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3386671115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf7006f8c0 0x7fdf700716d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009870 tx=0x7fdf5c02ba10 comp rx=0 tx=0).stop 2024-03-22T13:11:41.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 -- 172.21.15.6:0/3386671115 shutdown_connections 2024-03-22T13:11:41.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3386671115 >> v2:172.21.15.6:3300/0 conn(0x7fdf70071c10 0x7fdf700c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:41.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3386671115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf7006f8c0 0x7fdf700716d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:41.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 -- 172.21.15.6:0/3386671115 >> 172.21.15.6:0/3386671115 conn(0x7fdf700cb930 msgr2=0x7fdf700cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:41.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 -- 172.21.15.6:0/3386671115 shutdown_connections 2024-03-22T13:11:41.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.678+0000 7fdf74e00640 1 -- 172.21.15.6:0/3386671115 wait complete. 2024-03-22T13:11:41.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.680+0000 7fdf74e00640 1 Processor -- start 2024-03-22T13:11:41.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.681+0000 7fdf74e00640 1 -- start start 2024-03-22T13:11:41.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.682+0000 7fdf74e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7006f8c0 0x7fdf701475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.682+0000 7fdf74e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf70071c10 0x7fdf701462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.682+0000 7fdf74e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf70152400 0x7fdf70146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.682+0000 7fdf74e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf70158b90 con 0x7fdf7006f8c0 2024-03-22T13:11:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.682+0000 7fdf74e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf701588b0 con 0x7fdf70152400 2024-03-22T13:11:41.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.682+0000 7fdf74e00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf701527e0 con 0x7fdf70071c10 2024-03-22T13:11:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.682+0000 7fdf6d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf70071c10 0x7fdf701462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.683+0000 7fdf6d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf70071c10 0x7fdf701462d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:34946/0 (socket says 172.21.15.6:34946) 2024-03-22T13:11:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.683+0000 7fdf6d400640 1 -- 172.21.15.6:0/3236208005 learned_addr learned my addr 172.21.15.6:0/3236208005 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.683+0000 7fdf6e800640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf70152400 0x7fdf70146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.683+0000 7fdf6de00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7006f8c0 0x7fdf701475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.688+0000 7fdf6d400640 1 -- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf70152400 msgr2=0x7fdf70146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.688+0000 7fdf6d400640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf70152400 0x7fdf70146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.688+0000 7fdf6d400640 1 -- 172.21.15.6:0/3236208005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7006f8c0 msgr2=0x7fdf701475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.688+0000 7fdf6d400640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7006f8c0 0x7fdf701475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:41.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.688+0000 7fdf6d400640 1 -- 172.21.15.6:0/3236208005 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70149ce0 con 0x7fdf70071c10 2024-03-22T13:11:41.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.691+0000 7fdf6d400640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf70071c10 0x7fdf701462d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c02ba50 tx=0x7fdf5c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:41.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.694+0000 7fdf66000640 1 -- 172.21.15.6:0/3236208005 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c047070 con 0x7fdf70071c10 2024-03-22T13:11:41.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.694+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf701f0260 con 0x7fdf70071c10 2024-03-22T13:11:41.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.694+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf701f0820 con 0x7fdf70071c10 2024-03-22T13:11:41.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.695+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf700756c0 con 0x7fdf70071c10 2024-03-22T13:11:41.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.696+0000 7fdf66000640 1 -- 172.21.15.6:0/3236208005 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c02f930 con 0x7fdf70071c10 2024-03-22T13:11:41.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.696+0000 7fdf66000640 1 -- 172.21.15.6:0/3236208005 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c051060 con 0x7fdf70071c10 2024-03-22T13:11:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.721+0000 7fdf66000640 1 -- 172.21.15.6:0/3236208005 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf5c04a070 con 0x7fdf70071c10 2024-03-22T13:11:41.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.723+0000 7fdf66000640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf5006bbd0 0x7fdf5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:41.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.724+0000 7fdf6de00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf5006bbd0 0x7fdf5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.724+0000 7fdf6de00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf5006bbd0 0x7fdf5006e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdf600016e0 tx=0x7fdf60009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.725+0000 7fdf66000640 1 -- 172.21.15.6:0/3236208005 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fdf5c0795e0 con 0x7fdf70071c10 2024-03-22T13:11:41.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:41.988+0000 7fdf66000640 1 -- 172.21.15.6:0/3236208005 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf5c033030 con 0x7fdf70071c10 2024-03-22T13:11:42.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.267+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf701515d0 con 0x7fdf70071c10 2024-03-22T13:11:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.290+0000 7fdf66000640 1 -- 172.21.15.6:0/3236208005 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fdf5c07b540 con 0x7fdf70071c10 2024-03-22T13:11:42.291 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:42.291 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:42.291 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf5006bbd0 msgr2=0x7fdf5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf5006bbd0 0x7fdf5006e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdf600016e0 tx=0x7fdf60009040 comp rx=0 tx=0).stop 2024-03-22T13:11:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf70071c10 msgr2=0x7fdf701462d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf70071c10 0x7fdf701462d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c02ba50 tx=0x7fdf5c004730 comp rx=0 tx=0).stop 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 shutdown_connections 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fdf5006bbd0 0x7fdf5006e090 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf70152400 0x7fdf70146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf70071c10 0x7fdf701462d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 --2- 172.21.15.6:0/3236208005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7006f8c0 0x7fdf701475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.293+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 >> 172.21.15.6:0/3236208005 conn(0x7fdf700cb930 msgr2=0x7fdf700c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.294+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 shutdown_connections 2024-03-22T13:11:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:42.294+0000 7fdf74e00640 1 -- 172.21.15.6:0/3236208005 wait complete. 2024-03-22T13:11:42.919 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:42.919 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:43.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.119+0000 7f43ea200640 1 Processor -- start 2024-03-22T13:11:44.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.120+0000 7f43ea200640 1 -- start start 2024-03-22T13:11:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.121+0000 7f43ea200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43e4152400 0x7f43e415b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.121+0000 7f43ea200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43e415bc80 0x7f43e40716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.121+0000 7f43ea200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e41590b0 con 0x7f43e406f8c0 2024-03-22T13:11:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.121+0000 7f43ea200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e40724d0 con 0x7f43e4152400 2024-03-22T13:11:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.121+0000 7f43ea200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e40721f0 con 0x7f43e415bc80 2024-03-22T13:11:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.122+0000 7f43e2a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43e415bc80 0x7f43e40716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.122+0000 7f43e2a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43e415bc80 0x7f43e40716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47940/0 (socket says 172.21.15.6:47940) 2024-03-22T13:11:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.122+0000 7f43e3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f43e406f8c0 0x7f43e40c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35980/0 (socket says 172.21.15.6:35980) 2024-03-22T13:11:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.122+0000 7f43e3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43e4152400 0x7f43e415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.122+0000 7f43e3e00640 1 -- 172.21.15.6:0/211063862 learned_addr learned my addr 172.21.15.6:0/211063862 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:44.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.126+0000 7f43e2a00640 1 -- 172.21.15.6:0/211063862 >> v2:172.21.15.6:3300/0 conn(0x7f43e4152400 msgr2=0x7f43e415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:44.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.126+0000 7f43e2a00640 1 --2- 172.21.15.6:0/211063862 >> v2:172.21.15.6:3300/0 conn(0x7f43e4152400 0x7f43e415b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.126+0000 7f43e2a00640 1 -- 172.21.15.6:0/211063862 >> v1:172.21.15.6:6789/0 conn(0x7f43e406f8c0 legacy=0x7f43e40c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:44.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.126+0000 7f43e2a00640 1 -- 172.21.15.6:0/211063862 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e41582f0 con 0x7f43e415bc80 2024-03-22T13:11:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.129+0000 7f43e2a00640 1 --2- 172.21.15.6:0/211063862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e40716d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f43e415c060 tx=0x7f43d002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df9d05bbfd61177 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:44.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.134+0000 7f43e2000640 1 -- 172.21.15.6:0/211063862 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0034070 con 0x7f43e415bc80 2024-03-22T13:11:44.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.134+0000 7f43e2000640 1 -- 172.21.15.6:0/211063862 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d0030610 con 0x7f43e415bc80 2024-03-22T13:11:44.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.134+0000 7f43e2000640 1 -- 172.21.15.6:0/211063862 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00309e0 con 0x7f43e415bc80 2024-03-22T13:11:44.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 -- 172.21.15.6:0/211063862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 msgr2=0x7f43e40716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:44.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 --2- 172.21.15.6:0/211063862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e40716d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f43e415c060 tx=0x7f43d002b960 comp rx=0 tx=0).stop 2024-03-22T13:11:44.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 -- 172.21.15.6:0/211063862 shutdown_connections 2024-03-22T13:11:44.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 --2- 172.21.15.6:0/211063862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e40716d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 --2- 172.21.15.6:0/211063862 >> v2:172.21.15.6:3300/0 conn(0x7f43e4152400 0x7f43e415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 -- 172.21.15.6:0/211063862 >> 172.21.15.6:0/211063862 conn(0x7f43e40cb930 msgr2=0x7f43e40cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:44.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 -- 172.21.15.6:0/211063862 shutdown_connections 2024-03-22T13:11:44.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.135+0000 7f43ea200640 1 -- 172.21.15.6:0/211063862 wait complete. 2024-03-22T13:11:44.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.137+0000 7f43ea200640 1 Processor -- start 2024-03-22T13:11:44.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.138+0000 7f43ea200640 1 -- start start 2024-03-22T13:11:44.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.139+0000 7f43ea200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43e406f8c0 0x7f43e40caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:44.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.139+0000 7f43ea200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43e4152400 0x7f43e4145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:44.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.139+0000 7f43ea200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e4146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:44.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.139+0000 7f43ea200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e4158b90 con 0x7f43e406f8c0 2024-03-22T13:11:44.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.139+0000 7f43ea200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41588b0 con 0x7f43e415bc80 2024-03-22T13:11:44.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.139+0000 7f43ea200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41527e0 con 0x7f43e4152400 2024-03-22T13:11:44.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.140+0000 7f43e3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e4146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:44.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.140+0000 7f43e3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43e406f8c0 0x7f43e40caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:44.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.140+0000 7f43e3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e4146440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47952/0 (socket says 172.21.15.6:47952) 2024-03-22T13:11:44.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.140+0000 7f43e3e00640 1 -- 172.21.15.6:0/4164029618 learned_addr learned my addr 172.21.15.6:0/4164029618 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:44.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.142+0000 7f43e2a00640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43e4152400 0x7f43e4145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:44.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.143+0000 7f43e3e00640 1 -- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43e4152400 msgr2=0x7f43e4145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:44.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.143+0000 7f43e3e00640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43e4152400 0x7f43e4145f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.143+0000 7f43e3e00640 1 -- 172.21.15.6:0/4164029618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43e406f8c0 msgr2=0x7f43e40caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:44.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.143+0000 7f43e3e00640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43e406f8c0 0x7f43e40caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.144+0000 7f43e3e00640 1 -- 172.21.15.6:0/4164029618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e41eed40 con 0x7f43e415bc80 2024-03-22T13:11:44.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.146+0000 7f43e3e00640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e4146440 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00cc20 tx=0x7f43dc00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:44.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.150+0000 7f43db400640 1 -- 172.21.15.6:0/4164029618 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc017070 con 0x7f43e415bc80 2024-03-22T13:11:44.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.150+0000 7f43db400640 1 -- 172.21.15.6:0/4164029618 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc00ad40 con 0x7f43e415bc80 2024-03-22T13:11:44.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.150+0000 7f43db400640 1 -- 172.21.15.6:0/4164029618 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc012600 con 0x7f43e415bc80 2024-03-22T13:11:44.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.150+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e414e2a0 con 0x7f43e415bc80 2024-03-22T13:11:44.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.150+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e41eea60 con 0x7f43e415bc80 2024-03-22T13:11:44.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.151+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e40756c0 con 0x7f43e415bc80 2024-03-22T13:11:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.177+0000 7f43db400640 1 -- 172.21.15.6:0/4164029618 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f43dc01a070 con 0x7f43e415bc80 2024-03-22T13:11:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.177+0000 7f43db400640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f43c406bc20 0x7f43c406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.178+0000 7f43e3400640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f43c406bc20 0x7f43c406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.179+0000 7f43e3400640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f43c406bc20 0x7f43c406e0e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009b40 tx=0x7f43d4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.179+0000 7f43db400640 1 -- 172.21.15.6:0/4164029618 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f43dc0571c0 con 0x7f43e415bc80 2024-03-22T13:11:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.452+0000 7f43db400640 1 -- 172.21.15.6:0/4164029618 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f43dc059120 con 0x7f43e415bc80 2024-03-22T13:11:44.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.698+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43e40c6400 con 0x7f43e415bc80 2024-03-22T13:11:44.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.722+0000 7f43db400640 1 -- 172.21.15.6:0/4164029618 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f43dc05d1c0 con 0x7f43e415bc80 2024-03-22T13:11:44.723 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:44.723 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:44.723 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:44.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f43c406bc20 msgr2=0x7f43c406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f43c406bc20 0x7f43c406e0e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009b40 tx=0x7f43d4009190 comp rx=0 tx=0).stop 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 msgr2=0x7f43e4146440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e4146440 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00cc20 tx=0x7f43dc00a430 comp rx=0 tx=0).stop 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 shutdown_connections 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f43c406bc20 0x7f43c406e0e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43e415bc80 0x7f43e4146440 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43e4152400 0x7f43e4145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 --2- 172.21.15.6:0/4164029618 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43e406f8c0 0x7f43e40caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.726+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 >> 172.21.15.6:0/4164029618 conn(0x7f43e40cb930 msgr2=0x7f43e4074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.727+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 shutdown_connections 2024-03-22T13:11:44.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:44.727+0000 7f43ea200640 1 -- 172.21.15.6:0/4164029618 wait complete. 2024-03-22T13:11:45.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:45.348 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:46.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.549+0000 7fe920a00640 1 Processor -- start 2024-03-22T13:11:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.550+0000 7fe920a00640 1 -- start start 2024-03-22T13:11:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.551+0000 7fe920a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cc640 0x7fe91c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.551+0000 7fe920a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe91c15b7a0 0x7fe91c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:46.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.551+0000 7fe920a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c06dd00 con 0x7fe91c0cca20 2024-03-22T13:11:46.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.551+0000 7fe920a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91c06d330 con 0x7fe91c15b7a0 2024-03-22T13:11:46.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.551+0000 7fe920a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c06c440 con 0x7fe91c0cc640 2024-03-22T13:11:46.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.552+0000 7fe91a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cc640 0x7fe91c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:46.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.552+0000 7fe91b200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe91c15b7a0 0x7fe91c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:46.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.552+0000 7fe91a800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cc640 0x7fe91c06d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33532/0 (socket says 172.21.15.6:33532) 2024-03-22T13:11:46.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.552+0000 7fe91a800640 1 -- 172.21.15.6:0/2967618155 learned_addr learned my addr 172.21.15.6:0/2967618155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:46.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.555+0000 7fe91a800640 1 -- 172.21.15.6:0/2967618155 >> v2:172.21.15.92:3300/0 conn(0x7fe91c15b7a0 msgr2=0x7fe91c15e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:46.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.555+0000 7fe91a800640 1 --2- 172.21.15.6:0/2967618155 >> v2:172.21.15.92:3300/0 conn(0x7fe91c15b7a0 0x7fe91c15e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:46.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.555+0000 7fe91a800640 1 -- 172.21.15.6:0/2967618155 >> v1:172.21.15.92:6789/0 conn(0x7fe91c0cca20 legacy=0x7fe91c0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:46.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.555+0000 7fe91a800640 1 -- 172.21.15.6:0/2967618155 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c06c720 con 0x7fe91c0cc640 2024-03-22T13:11:46.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.558+0000 7fe91a800640 1 --2- 172.21.15.6:0/2967618155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cc640 0x7fe91c06d7c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009870 tx=0x7fe90c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a87308058e830b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:46.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.563+0000 7fe919400640 1 -- 172.21.15.6:0/2967618155 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c02f840 con 0x7fe91c0cc640 2024-03-22T13:11:46.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.563+0000 7fe919400640 1 -- 172.21.15.6:0/2967618155 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c02f9e0 con 0x7fe91c0cc640 2024-03-22T13:11:46.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.563+0000 7fe919400640 1 -- 172.21.15.6:0/2967618155 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c02fdb0 con 0x7fe91c0cc640 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 -- 172.21.15.6:0/2967618155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cc640 msgr2=0x7fe91c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 --2- 172.21.15.6:0/2967618155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cc640 0x7fe91c06d7c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009870 tx=0x7fe90c02ba20 comp rx=0 tx=0).stop 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 -- 172.21.15.6:0/2967618155 shutdown_connections 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 --2- 172.21.15.6:0/2967618155 >> v2:172.21.15.92:3300/0 conn(0x7fe91c15b7a0 0x7fe91c15e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 --2- 172.21.15.6:0/2967618155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cc640 0x7fe91c06d7c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 -- 172.21.15.6:0/2967618155 >> 172.21.15.6:0/2967618155 conn(0x7fe91c0cf9c0 msgr2=0x7fe91c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 -- 172.21.15.6:0/2967618155 shutdown_connections 2024-03-22T13:11:46.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.564+0000 7fe920a00640 1 -- 172.21.15.6:0/2967618155 wait complete. 2024-03-22T13:11:46.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.566+0000 7fe920a00640 1 Processor -- start 2024-03-22T13:11:46.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.566+0000 7fe920a00640 1 -- start start 2024-03-22T13:11:46.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.567+0000 7fe920a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe91c0cc640 0x7fe91c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.567+0000 7fe920a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cca20 0x7fe91c1f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.567+0000 7fe920a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe91c15b7a0 0x7fe91c1f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.567+0000 7fe920a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c06cfc0 con 0x7fe91c15b7a0 2024-03-22T13:11:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.567+0000 7fe920a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c06cce0 con 0x7fe91c0cc640 2024-03-22T13:11:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.567+0000 7fe920a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c0caa50 con 0x7fe91c0cca20 2024-03-22T13:11:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.568+0000 7fe919e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cca20 0x7fe91c1f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:46.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.568+0000 7fe919e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cca20 0x7fe91c1f14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33534/0 (socket says 172.21.15.6:33534) 2024-03-22T13:11:46.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.568+0000 7fe919e00640 1 -- 172.21.15.6:0/1919397459 learned_addr learned my addr 172.21.15.6:0/1919397459 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:46.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.568+0000 7fe91a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe91c0cc640 0x7fe91c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:46.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.572+0000 7fe91a800640 1 -- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cca20 msgr2=0x7fe91c1f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:46.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.572+0000 7fe91a800640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cca20 0x7fe91c1f14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:46.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.572+0000 7fe91a800640 1 -- 172.21.15.6:0/1919397459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe91c15b7a0 msgr2=0x7fe91c1f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:46.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.572+0000 7fe91a800640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe91c15b7a0 0x7fe91c1f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:46.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.572+0000 7fe91a800640 1 -- 172.21.15.6:0/1919397459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c157a80 con 0x7fe91c0cc640 2024-03-22T13:11:46.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.577+0000 7fe91a800640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe91c0cc640 0x7fe91c0c78d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe90c033000 tx=0x7fe90c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:46.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.583+0000 7fe912a00640 1 -- 172.21.15.6:0/1919397459 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c047070 con 0x7fe91c0cc640 2024-03-22T13:11:46.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.583+0000 7fe912a00640 1 -- 172.21.15.6:0/1919397459 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c037bc0 con 0x7fe91c0cc640 2024-03-22T13:11:46.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.583+0000 7fe912a00640 1 -- 172.21.15.6:0/1919397459 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c042460 con 0x7fe91c0cc640 2024-03-22T13:11:46.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.583+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c157d60 con 0x7fe91c0cc640 2024-03-22T13:11:46.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.583+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c155c70 con 0x7fe91c0cc640 2024-03-22T13:11:46.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.585+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91c075650 con 0x7fe91c0cc640 2024-03-22T13:11:46.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.896+0000 7fe912a00640 1 -- 172.21.15.6:0/1919397459 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe90c04a070 con 0x7fe91c0cc640 2024-03-22T13:11:46.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.899+0000 7fe912a00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe8fc06bc20 0x7fe8fc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:46.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.899+0000 7fe919e00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe8fc06bc20 0x7fe8fc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:46.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.900+0000 7fe912a00640 1 -- 172.21.15.6:0/1919397459 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe90c0794d0 con 0x7fe91c0cc640 2024-03-22T13:11:46.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.900+0000 7fe919e00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe8fc06bc20 0x7fe8fc06e0e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0c8c40 tx=0x7fe908009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:46.911+0000 7fe912a00640 1 -- 172.21.15.6:0/1919397459 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe90c052080 con 0x7fe91c0cc640 2024-03-22T13:11:47.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.191+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe91c0af100 con 0x7fe91c0cc640 2024-03-22T13:11:47.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.215+0000 7fe912a00640 1 -- 172.21.15.6:0/1919397459 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fe90c033030 con 0x7fe91c0cc640 2024-03-22T13:11:47.215 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:47.215 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:47.215 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe8fc06bc20 msgr2=0x7fe8fc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe8fc06bc20 0x7fe8fc06e0e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0c8c40 tx=0x7fe908009040 comp rx=0 tx=0).stop 2024-03-22T13:11:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe91c0cc640 msgr2=0x7fe91c0c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe91c0cc640 0x7fe91c0c78d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe90c033000 tx=0x7fe90c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:11:47.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 shutdown_connections 2024-03-22T13:11:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe91c15b7a0 0x7fe91c1f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe91c0cca20 0x7fe91c1f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe8fc06bc20 0x7fe8fc06e0e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 --2- 172.21.15.6:0/1919397459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe91c0cc640 0x7fe91c0c78d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 >> 172.21.15.6:0/1919397459 conn(0x7fe91c0cf9c0 msgr2=0x7fe91c0cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 shutdown_connections 2024-03-22T13:11:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:47.218+0000 7fe920a00640 1 -- 172.21.15.6:0/1919397459 wait complete. 2024-03-22T13:11:47.839 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:47.840 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:48.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:49.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.038+0000 7fedff600640 1 Processor -- start 2024-03-22T13:11:49.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.038+0000 7fedff600640 1 -- start start 2024-03-22T13:11:49.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.040+0000 7fedff600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf80cd300 0x7fedf815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:49.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.040+0000 7fedff600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fedf815aef0 0x7fedf80ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:49.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.040+0000 7fedff600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80cc380 con 0x7fedf8154980 2024-03-22T13:11:49.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.040+0000 7fedff600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf815a520 con 0x7fedf815aef0 2024-03-22T13:11:49.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.040+0000 7fedff600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf815b3a0 con 0x7fedf80cd300 2024-03-22T13:11:49.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.041+0000 7fedfd600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fedf815aef0 0x7fedf80ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:49.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.041+0000 7fedf7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf80cd300 0x7fedf815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:49.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.041+0000 7fedfd600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fedf815aef0 0x7fedf80ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47970/0 (socket says 172.21.15.6:47970) 2024-03-22T13:11:49.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.041+0000 7fedfd600640 1 -- 172.21.15.6:0/2990242642 learned_addr learned my addr 172.21.15.6:0/2990242642 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.044+0000 7fedf7e00640 1 -- 172.21.15.6:0/2990242642 >> v2:172.21.15.92:3300/0 conn(0x7fedf815aef0 msgr2=0x7fedf80ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.044+0000 7fedf7e00640 1 --2- 172.21.15.6:0/2990242642 >> v2:172.21.15.92:3300/0 conn(0x7fedf815aef0 0x7fedf80ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.044+0000 7fedf7e00640 1 -- 172.21.15.6:0/2990242642 >> v1:172.21.15.92:6789/0 conn(0x7fedf8154980 legacy=0x7fedf80ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:49.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.044+0000 7fedf7e00640 1 -- 172.21.15.6:0/2990242642 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80cc660 con 0x7fedf80cd300 2024-03-22T13:11:49.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.047+0000 7fedf7e00640 1 --2- 172.21.15.6:0/2990242642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf80cd300 0x7fedf815a9b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fede00097a0 tx=0x7fede002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3de02dc81a3b983a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.052+0000 7fedf7400640 1 -- 172.21.15.6:0/2990242642 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0035020 con 0x7fedf80cd300 2024-03-22T13:11:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.052+0000 7fedf7400640 1 -- 172.21.15.6:0/2990242642 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede0031830 con 0x7fedf80cd300 2024-03-22T13:11:49.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.052+0000 7fedf7400640 1 -- 172.21.15.6:0/2990242642 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0031c00 con 0x7fedf80cd300 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 -- 172.21.15.6:0/2990242642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf80cd300 msgr2=0x7fedf815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 --2- 172.21.15.6:0/2990242642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf80cd300 0x7fedf815a9b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fede00097a0 tx=0x7fede002b8d0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 -- 172.21.15.6:0/2990242642 shutdown_connections 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 --2- 172.21.15.6:0/2990242642 >> v2:172.21.15.92:3300/0 conn(0x7fedf815aef0 0x7fedf80ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 --2- 172.21.15.6:0/2990242642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf80cd300 0x7fedf815a9b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 -- 172.21.15.6:0/2990242642 >> 172.21.15.6:0/2990242642 conn(0x7fedf80b16c0 msgr2=0x7fedf80b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 -- 172.21.15.6:0/2990242642 shutdown_connections 2024-03-22T13:11:49.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.053+0000 7fedff600640 1 -- 172.21.15.6:0/2990242642 wait complete. 2024-03-22T13:11:49.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.056+0000 7fedff600640 1 Processor -- start 2024-03-22T13:11:49.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.056+0000 7fedff600640 1 -- start start 2024-03-22T13:11:49.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.058+0000 7fedff600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf80cd300 0x7fedf80c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:49.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.058+0000 7fedff600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedf8154980 0x7fedf815d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:49.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.058+0000 7fedff600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf815aef0 0x7fedf81f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:49.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.058+0000 7fedff600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80ccf00 con 0x7fedf80cd300 2024-03-22T13:11:49.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.058+0000 7fedff600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80ccc20 con 0x7fedf8154980 2024-03-22T13:11:49.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.058+0000 7fedff600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80cea10 con 0x7fedf815aef0 2024-03-22T13:11:49.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.059+0000 7fedf7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedf8154980 0x7fedf815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:49.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.059+0000 7fedfcc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf80cd300 0x7fedf80c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:49.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.059+0000 7fedfd600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf815aef0 0x7fedf81f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:49.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.059+0000 7fedf7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedf8154980 0x7fedf815d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47980/0 (socket says 172.21.15.6:47980) 2024-03-22T13:11:49.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.059+0000 7fedf7e00640 1 -- 172.21.15.6:0/2992913986 learned_addr learned my addr 172.21.15.6:0/2992913986 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:49.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.063+0000 7fedfd600640 1 -- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedf8154980 msgr2=0x7fedf815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:49.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.063+0000 7fedfd600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedf8154980 0x7fedf815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.063+0000 7fedfd600640 1 -- 172.21.15.6:0/2992913986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf80cd300 msgr2=0x7fedf80c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:49.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.063+0000 7fedfd600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf80cd300 0x7fedf80c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.063+0000 7fedfd600640 1 -- 172.21.15.6:0/2992913986 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80c9d60 con 0x7fedf815aef0 2024-03-22T13:11:49.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.065+0000 7fedfd600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf815aef0 0x7fedf81f3c40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fedf000e670 tx=0x7fedf000eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:49.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.070+0000 7fedf4c00640 1 -- 172.21.15.6:0/2992913986 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0017070 con 0x7fedf815aef0 2024-03-22T13:11:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.070+0000 7fedf4c00640 1 -- 172.21.15.6:0/2992913986 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf000c730 con 0x7fedf815aef0 2024-03-22T13:11:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.070+0000 7fedf4c00640 1 -- 172.21.15.6:0/2992913986 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf000cb00 con 0x7fedf815aef0 2024-03-22T13:11:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.070+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf815b580 con 0x7fedf815aef0 2024-03-22T13:11:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.070+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf815bb00 con 0x7fedf815aef0 2024-03-22T13:11:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.071+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf80756c0 con 0x7fedf815aef0 2024-03-22T13:11:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.095+0000 7fedf4c00640 1 -- 172.21.15.6:0/2992913986 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fedf001a070 con 0x7fedf815aef0 2024-03-22T13:11:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.096+0000 7fedf4c00640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fedd406bbd0 0x7fedd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.096+0000 7fedfcc00640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fedd406bbd0 0x7fedd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.097+0000 7fedfcc00640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fedd406bbd0 0x7fedd406e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fede8001f10 tx=0x7fede8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:49.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.099+0000 7fedf4c00640 1 -- 172.21.15.6:0/2992913986 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fedf00575a0 con 0x7fedf815aef0 2024-03-22T13:11:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.391+0000 7fedf4c00640 1 -- 172.21.15.6:0/2992913986 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fedf0059500 con 0x7fedf815aef0 2024-03-22T13:11:49.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.637+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf80b5b30 con 0x7fedf815aef0 2024-03-22T13:11:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.701+0000 7fedf4c00640 1 -- 172.21.15.6:0/2992913986 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7fedf005d5a0 con 0x7fedf815aef0 2024-03-22T13:11:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:49.701 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:49.701 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fedd406bbd0 msgr2=0x7fedd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fedd406bbd0 0x7fedd406e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fede8001f10 tx=0x7fede8009040 comp rx=0 tx=0).stop 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf815aef0 msgr2=0x7fedf81f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf815aef0 0x7fedf81f3c40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fedf000e670 tx=0x7fedf000eb30 comp rx=0 tx=0).stop 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 shutdown_connections 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fedf815aef0 0x7fedf81f3c40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fedd406bbd0 0x7fedd406e090 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedf8154980 0x7fedf815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 --2- 172.21.15.6:0/2992913986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedf80cd300 0x7fedf80c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:49.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 >> 172.21.15.6:0/2992913986 conn(0x7fedf80b16c0 msgr2=0x7fedf8150c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:49.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 shutdown_connections 2024-03-22T13:11:49.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:49.704+0000 7fedff600640 1 -- 172.21.15.6:0/2992913986 wait complete. 2024-03-22T13:11:50.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:50.324 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:51.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:51.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.530+0000 7f6867200640 1 Processor -- start 2024-03-22T13:11:51.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.530+0000 7f6867200640 1 -- start start 2024-03-22T13:11:51.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.531+0000 7f6867200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68680cc380 0x7f686806c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:51.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.531+0000 7f6867200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686815b260 con 0x7f686815ae30 2024-03-22T13:11:51.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.531+0000 7f6867200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68680cee00 con 0x7f68680ce8a0 2024-03-22T13:11:51.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.531+0000 7f6867200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68680cf0c0 con 0x7f68680cc380 2024-03-22T13:11:51.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.532+0000 7f6865e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68680cc380 0x7f686806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:51.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.532+0000 7f6865400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f68680ce8a0 0x7f68680cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:52150/0 (socket says 172.21.15.6:52150) 2024-03-22T13:11:51.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.533+0000 7f6865400640 1 -- 172.21.15.6:0/3032190781 learned_addr learned my addr 172.21.15.6:0/3032190781 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:51.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.536+0000 7f6864a00640 1 -- 172.21.15.6:0/3032190781 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340620813 0 0) 0x7f68680cee00 con 0x7f68680ce8a0 2024-03-22T13:11:51.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.536+0000 7f6864a00640 1 -- 172.21.15.6:0/3032190781 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684c003520 con 0x7f68680ce8a0 2024-03-22T13:11:51.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.536+0000 7f6864a00640 1 -- 172.21.15.6:0/3032190781 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274621812 0 0) 0x7f6860001490 con 0x7f686815ae30 2024-03-22T13:11:51.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.536+0000 7f6864a00640 1 -- 172.21.15.6:0/3032190781 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68680cee00 con 0x7f686815ae30 2024-03-22T13:11:51.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.536+0000 7f6865e00640 1 -- 172.21.15.6:0/3032190781 >> v1:172.21.15.92:6789/0 conn(0x7f68680ce8a0 legacy=0x7f68680cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:51.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.536+0000 7f6865e00640 1 -- 172.21.15.6:0/3032190781 >> v1:172.21.15.6:6789/0 conn(0x7f686815ae30 legacy=0x7f6868164b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:51.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.536+0000 7f6865e00640 1 -- 172.21.15.6:0/3032190781 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68680cf3a0 con 0x7f68680cc380 2024-03-22T13:11:51.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.539+0000 7f6865e00640 1 --2- 172.21.15.6:0/3032190781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68680cc380 0x7f686806c230 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f68580097d0 tx=0x7f685802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27ed92821ca81c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.544+0000 7f6864a00640 1 -- 172.21.15.6:0/3032190781 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6858035020 con 0x7f68680cc380 2024-03-22T13:11:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.544+0000 7f6864a00640 1 -- 172.21.15.6:0/3032190781 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6858031890 con 0x7f68680cc380 2024-03-22T13:11:51.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.544+0000 7f6864a00640 1 -- 172.21.15.6:0/3032190781 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6858031c60 con 0x7f68680cc380 2024-03-22T13:11:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.545+0000 7f6867200640 1 -- 172.21.15.6:0/3032190781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68680cc380 msgr2=0x7f686806c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.545+0000 7f6867200640 1 --2- 172.21.15.6:0/3032190781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68680cc380 0x7f686806c230 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f68580097d0 tx=0x7f685802b930 comp rx=0 tx=0).stop 2024-03-22T13:11:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.545+0000 7f6867200640 1 -- 172.21.15.6:0/3032190781 shutdown_connections 2024-03-22T13:11:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.545+0000 7f6867200640 1 --2- 172.21.15.6:0/3032190781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68680cc380 0x7f686806c230 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.545+0000 7f6867200640 1 -- 172.21.15.6:0/3032190781 >> 172.21.15.6:0/3032190781 conn(0x7f68680b1480 msgr2=0x7f68680b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.545+0000 7f6867200640 1 -- 172.21.15.6:0/3032190781 shutdown_connections 2024-03-22T13:11:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.545+0000 7f6867200640 1 -- 172.21.15.6:0/3032190781 wait complete. 2024-03-22T13:11:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.547+0000 7f6867200640 1 Processor -- start 2024-03-22T13:11:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.547+0000 7f6867200640 1 -- start start 2024-03-22T13:11:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6867200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68680ce8a0 0x7f68681f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6867200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f686815ae30 0x7f6868149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6867200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68681e59a0 0x7f6868155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6867200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686815a910 con 0x7f68680ce8a0 2024-03-22T13:11:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6867200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686815a630 con 0x7f68681e59a0 2024-03-22T13:11:51.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6867200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68681549d0 con 0x7f686815ae30 2024-03-22T13:11:51.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6865400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f686815ae30 0x7f6868149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:51.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.548+0000 7f6865e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68680ce8a0 0x7f68681f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:51.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.549+0000 7f6866800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68681e59a0 0x7f6868155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:51.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.549+0000 7f6865400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f686815ae30 0x7f6868149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:33574/0 (socket says 172.21.15.6:33574) 2024-03-22T13:11:51.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.549+0000 7f6865400640 1 -- 172.21.15.6:0/4224208941 learned_addr learned my addr 172.21.15.6:0/4224208941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:51.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.553+0000 7f6866800640 1 -- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f686815ae30 msgr2=0x7f6868149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:51.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.553+0000 7f6866800640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f686815ae30 0x7f6868149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:51.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.553+0000 7f6866800640 1 -- 172.21.15.6:0/4224208941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68680ce8a0 msgr2=0x7f68681f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:51.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.553+0000 7f6866800640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68680ce8a0 0x7f68681f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:51.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.553+0000 7f6866800640 1 -- 172.21.15.6:0/4224208941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68681f5140 con 0x7f68681e59a0 2024-03-22T13:11:51.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.555+0000 7f6866800640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68681e59a0 0x7f6868155000 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f686000c700 tx=0x7f686000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:51.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.557+0000 7f685e000640 1 -- 172.21.15.6:0/4224208941 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860017070 con 0x7f68681e59a0 2024-03-22T13:11:51.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.557+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6868156c70 con 0x7f68681e59a0 2024-03-22T13:11:51.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.558+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6868156f10 con 0x7f68681e59a0 2024-03-22T13:11:51.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.559+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68680c9fc0 con 0x7f68681e59a0 2024-03-22T13:11:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.560+0000 7f685e000640 1 -- 172.21.15.6:0/4224208941 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686000ce10 con 0x7f68681e59a0 2024-03-22T13:11:51.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.560+0000 7f685e000640 1 -- 172.21.15.6:0/4224208941 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860003d70 con 0x7f68681e59a0 2024-03-22T13:11:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.841+0000 7f685e000640 1 -- 172.21.15.6:0/4224208941 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f686001a070 con 0x7f68681e59a0 2024-03-22T13:11:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.842+0000 7f685e000640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f684806bc00 0x7f684806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.843+0000 7f6865e00640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f684806bc00 0x7f684806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.843+0000 7f685e000640 1 -- 172.21.15.6:0/4224208941 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f68600568f0 con 0x7f68681e59a0 2024-03-22T13:11:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.844+0000 7f6865e00640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f684806bc00 0x7f684806e0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6858036000 tx=0x7f6858008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:51.855+0000 7f685e000640 1 -- 172.21.15.6:0/4224208941 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f68680c9fc0 con 0x7f68681e59a0 2024-03-22T13:11:52.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.137+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6868075650 con 0x7f68681e59a0 2024-03-22T13:11:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.161+0000 7f685e000640 1 -- 172.21.15.6:0/4224208941 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f6860021020 con 0x7f68681e59a0 2024-03-22T13:11:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:52.161 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:52.161 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f684806bc00 msgr2=0x7f684806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f684806bc00 0x7f684806e0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6858036000 tx=0x7f6858008030 comp rx=0 tx=0).stop 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68681e59a0 msgr2=0x7f6868155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68681e59a0 0x7f6868155000 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f686000c700 tx=0x7f686000cbc0 comp rx=0 tx=0).stop 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 shutdown_connections 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f684806bc00 0x7f684806e0c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68681e59a0 0x7f6868155000 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f686815ae30 0x7f6868149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:52.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 --2- 172.21.15.6:0/4224208941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68680ce8a0 0x7f68681f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:52.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 >> 172.21.15.6:0/4224208941 conn(0x7f68680b1480 msgr2=0x7f68681597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:52.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 shutdown_connections 2024-03-22T13:11:52.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:52.164+0000 7f6867200640 1 -- 172.21.15.6:0/4224208941 wait complete. 2024-03-22T13:11:52.786 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:52.786 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:53.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.988+0000 7f2de8400640 1 Processor -- start 2024-03-22T13:11:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.988+0000 7f2de8400640 1 -- start start 2024-03-22T13:11:53.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.990+0000 7f2de8400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2de006f8c0 0x7f2de00716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.990+0000 7f2de8400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de01590b0 con 0x7f2de0152400 2024-03-22T13:11:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.990+0000 7f2de8400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de00721f0 con 0x7f2de0071c10 2024-03-22T13:11:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.990+0000 7f2de8400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de00724d0 con 0x7f2de006f8c0 2024-03-22T13:11:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.991+0000 7f2de6400640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2de0071c10 0x7f2de00c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:53766/0 (socket says 172.21.15.6:53766) 2024-03-22T13:11:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.991+0000 7f2de6400640 1 -- 172.21.15.6:0/1261726381 learned_addr learned my addr 172.21.15.6:0/1261726381 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.991+0000 7f2de5000640 1 --2- 172.21.15.6:0/1261726381 >> v2:172.21.15.6:3300/0 conn(0x7f2de006f8c0 0x7f2de00716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.994+0000 7f2ddfe00640 1 -- 172.21.15.6:0/1261726381 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413108961 0 0) 0x7f2de00721f0 con 0x7f2de0071c10 2024-03-22T13:11:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.994+0000 7f2ddfe00640 1 -- 172.21.15.6:0/1261726381 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc8003520 con 0x7f2de0071c10 2024-03-22T13:11:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.996+0000 7f2de5000640 1 -- 172.21.15.6:0/1261726381 >> v1:172.21.15.92:6789/0 conn(0x7f2de0071c10 legacy=0x7f2de00c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.996+0000 7f2de5000640 1 -- 172.21.15.6:0/1261726381 >> v1:172.21.15.6:6789/0 conn(0x7f2de0152400 legacy=0x7f2de015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:53.996+0000 7f2de5000640 1 -- 172.21.15.6:0/1261726381 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de01582f0 con 0x7f2de006f8c0 2024-03-22T13:11:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.002+0000 7f2de5000640 1 --2- 172.21.15.6:0/1261726381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de006f8c0 0x7f2de00716d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0097a0 tx=0x7f2dcc02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd0130743e52482 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.008+0000 7f2ddfe00640 1 -- 172.21.15.6:0/1261726381 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc02f810 con 0x7f2de006f8c0 2024-03-22T13:11:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.008+0000 7f2ddfe00640 1 -- 172.21.15.6:0/1261726381 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc02f9b0 con 0x7f2de006f8c0 2024-03-22T13:11:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.008+0000 7f2ddfe00640 1 -- 172.21.15.6:0/1261726381 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc02fd80 con 0x7f2de006f8c0 2024-03-22T13:11:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.009+0000 7f2de8400640 1 -- 172.21.15.6:0/1261726381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de006f8c0 msgr2=0x7f2de00716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.009+0000 7f2de8400640 1 --2- 172.21.15.6:0/1261726381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de006f8c0 0x7f2de00716d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0097a0 tx=0x7f2dcc02b9f0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.009+0000 7f2de8400640 1 -- 172.21.15.6:0/1261726381 shutdown_connections 2024-03-22T13:11:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.009+0000 7f2de8400640 1 --2- 172.21.15.6:0/1261726381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de006f8c0 0x7f2de00716d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.009+0000 7f2de8400640 1 -- 172.21.15.6:0/1261726381 >> 172.21.15.6:0/1261726381 conn(0x7f2de00cb930 msgr2=0x7f2de00cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:54.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.010+0000 7f2de8400640 1 -- 172.21.15.6:0/1261726381 shutdown_connections 2024-03-22T13:11:54.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.010+0000 7f2de8400640 1 -- 172.21.15.6:0/1261726381 wait complete. 2024-03-22T13:11:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.012+0000 7f2de8400640 1 Processor -- start 2024-03-22T13:11:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.012+0000 7f2de8400640 1 -- start start 2024-03-22T13:11:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.014+0000 7f2de8400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2de006f8c0 0x7f2de0156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.014+0000 7f2de8400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2de0071c10 0x7f2de00c3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.014+0000 7f2de8400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de0152400 0x7f2de00c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.014+0000 7f2de8400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de0158b90 con 0x7f2de0152400 2024-03-22T13:11:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.014+0000 7f2de8400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01588b0 con 0x7f2de006f8c0 2024-03-22T13:11:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.014+0000 7f2de8400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01527e0 con 0x7f2de0071c10 2024-03-22T13:11:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.015+0000 7f2de5000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2de0071c10 0x7f2de00c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.015+0000 7f2de5a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2de006f8c0 0x7f2de0156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.015+0000 7f2de5000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2de0071c10 0x7f2de00c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:48278/0 (socket says 172.21.15.6:48278) 2024-03-22T13:11:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.015+0000 7f2de5000640 1 -- 172.21.15.6:0/699692499 learned_addr learned my addr 172.21.15.6:0/699692499 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.018+0000 7f2de6400640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de0152400 0x7f2de00c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.019+0000 7f2de5a00640 1 -- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2de0071c10 msgr2=0x7f2de00c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.019+0000 7f2de5a00640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2de0071c10 0x7f2de00c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.019+0000 7f2de5a00640 1 -- 172.21.15.6:0/699692499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de0152400 msgr2=0x7f2de00c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.019+0000 7f2de5a00640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de0152400 0x7f2de00c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.019+0000 7f2de5a00640 1 -- 172.21.15.6:0/699692499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de0155830 con 0x7f2de006f8c0 2024-03-22T13:11:54.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.025+0000 7f2de5a00640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2de006f8c0 0x7f2de0156270 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400c3c0 tx=0x7f2dd400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:54.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.030+0000 7f2ddd600640 1 -- 172.21.15.6:0/699692499 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40052d0 con 0x7f2de006f8c0 2024-03-22T13:11:54.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.030+0000 7f2ddd600640 1 -- 172.21.15.6:0/699692499 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd4005490 con 0x7f2de006f8c0 2024-03-22T13:11:54.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.030+0000 7f2ddd600640 1 -- 172.21.15.6:0/699692499 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4007260 con 0x7f2de006f8c0 2024-03-22T13:11:54.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.030+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de0145fe0 con 0x7f2de006f8c0 2024-03-22T13:11:54.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.030+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de00c3040 con 0x7f2de006f8c0 2024-03-22T13:11:54.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.032+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de0146500 con 0x7f2de006f8c0 2024-03-22T13:11:54.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.056+0000 7f2ddd600640 1 -- 172.21.15.6:0/699692499 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2dd4007400 con 0x7f2de006f8c0 2024-03-22T13:11:54.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.057+0000 7f2ddd600640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2dc006bc20 0x7f2dc006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:54.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.058+0000 7f2ddd600640 1 -- 172.21.15.6:0/699692499 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f2dd4020cb0 con 0x7f2de006f8c0 2024-03-22T13:11:54.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.058+0000 7f2de5000640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2dc006bc20 0x7f2dc006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:54.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.059+0000 7f2de5000640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2dc006bc20 0x7f2dc006e0e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2de0150fd0 tx=0x7f2dcc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:54.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.367+0000 7f2ddd600640 1 -- 172.21.15.6:0/699692499 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2dd4058ae0 con 0x7f2de006f8c0 2024-03-22T13:11:54.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.621+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2de00c2e30 con 0x7f2de006f8c0 2024-03-22T13:11:54.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.641+0000 7f2ddd600640 1 -- 172.21.15.6:0/699692499 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f2dd405cb80 con 0x7f2de006f8c0 2024-03-22T13:11:54.641 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:54.641 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:54.642 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2dc006bc20 msgr2=0x7f2dc006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2dc006bc20 0x7f2dc006e0e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2de0150fd0 tx=0x7f2dcc008030 comp rx=0 tx=0).stop 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2de006f8c0 msgr2=0x7f2de0156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2de006f8c0 0x7f2de0156270 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400c3c0 tx=0x7f2dd400c880 comp rx=0 tx=0).stop 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 shutdown_connections 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2de0152400 0x7f2de00c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2de0071c10 0x7f2de00c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2dc006bc20 0x7f2dc006e0e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 --2- 172.21.15.6:0/699692499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2de006f8c0 0x7f2de0156270 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 >> 172.21.15.6:0/699692499 conn(0x7f2de00cb930 msgr2=0x7f2de0151670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 shutdown_connections 2024-03-22T13:11:54.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:54.644+0000 7f2de8400640 1 -- 172.21.15.6:0/699692499 wait complete. 2024-03-22T13:11:55.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:55.268 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:56.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:56.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.477+0000 7ff200600640 1 Processor -- start 2024-03-22T13:11:56.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.477+0000 7ff200600640 1 -- start start 2024-03-22T13:11:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.479+0000 7ff200600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff1f806fea0 0x7ff1f806b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.479+0000 7ff200600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff1f80c2ea0 0x7ff1f80c3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.479+0000 7ff200600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f80c3d30 con 0x7ff1f806b080 2024-03-22T13:11:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.479+0000 7ff200600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f80c3a50 con 0x7ff1f806fea0 2024-03-22T13:11:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.479+0000 7ff200600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f815a450 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.480+0000 7ff1fdc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff1f806b080 0x7ff1f815d8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38284/0 (socket says 172.21.15.6:38284) 2024-03-22T13:11:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.480+0000 7ff1fdc00640 1 -- 172.21.15.6:0/4152617538 learned_addr learned my addr 172.21.15.6:0/4152617538 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.480+0000 7ff1fd200640 1 --2- 172.21.15.6:0/4152617538 >> v2:172.21.15.6:3300/0 conn(0x7ff1f806fea0 0x7ff1f806b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.486+0000 7ff1fd200640 1 -- 172.21.15.6:0/4152617538 >> v2:172.21.15.92:3300/0 conn(0x7ff1f80c2ea0 msgr2=0x7ff1f80c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.486+0000 7ff1fd200640 1 --2- 172.21.15.6:0/4152617538 >> v2:172.21.15.92:3300/0 conn(0x7ff1f80c2ea0 0x7ff1f80c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.486+0000 7ff1fd200640 1 -- 172.21.15.6:0/4152617538 >> v1:172.21.15.6:6789/0 conn(0x7ff1f806b080 legacy=0x7ff1f815d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.486+0000 7ff1fd200640 1 -- 172.21.15.6:0/4152617538 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f815a730 con 0x7ff1f806fea0 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.491+0000 7ff1fd200640 1 --2- 172.21.15.6:0/4152617538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806fea0 0x7ff1f806b460 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff1f806fa90 tx=0x7ff1e402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4172503a6a4e78d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.497+0000 7ff1f7e00640 1 -- 172.21.15.6:0/4152617538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4034070 con 0x7ff1f806fea0 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.497+0000 7ff1f7e00640 1 -- 172.21.15.6:0/4152617538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e4030610 con 0x7ff1f806fea0 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.497+0000 7ff1f7e00640 1 -- 172.21.15.6:0/4152617538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40309e0 con 0x7ff1f806fea0 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 -- 172.21.15.6:0/4152617538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806fea0 msgr2=0x7ff1f806b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 --2- 172.21.15.6:0/4152617538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806fea0 0x7ff1f806b460 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff1f806fa90 tx=0x7ff1e402b960 comp rx=0 tx=0).stop 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 -- 172.21.15.6:0/4152617538 shutdown_connections 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 --2- 172.21.15.6:0/4152617538 >> v2:172.21.15.92:3300/0 conn(0x7ff1f80c2ea0 0x7ff1f80c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 --2- 172.21.15.6:0/4152617538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806fea0 0x7ff1f806b460 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 -- 172.21.15.6:0/4152617538 >> 172.21.15.6:0/4152617538 conn(0x7ff1f80ccb50 msgr2=0x7ff1f80ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 -- 172.21.15.6:0/4152617538 shutdown_connections 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.498+0000 7ff200600640 1 -- 172.21.15.6:0/4152617538 wait complete. 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.500+0000 7ff200600640 1 Processor -- start 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.500+0000 7ff200600640 1 -- start start 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.501+0000 7ff200600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806b080 0x7ff1f81effd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.501+0000 7ff200600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1f806fea0 0x7ff1f80c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.501+0000 7ff200600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1f80c2ea0 0x7ff1f80cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.501+0000 7ff200600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f815e000 con 0x7ff1f806b080 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.501+0000 7ff200600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f815acf0 con 0x7ff1f806fea0 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.501+0000 7ff200600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f815b210 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.502+0000 7ff1fe600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1f80c2ea0 0x7ff1f80cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.502+0000 7ff1fe600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1f80c2ea0 0x7ff1f80cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:48282/0 (socket says 172.21.15.6:48282) 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.502+0000 7ff1fe600640 1 -- 172.21.15.6:0/900130246 learned_addr learned my addr 172.21.15.6:0/900130246 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.504+0000 7ff1fdc00640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806b080 0x7ff1f81effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.507+0000 7ff1fe600640 1 -- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1f806fea0 msgr2=0x7ff1f80c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:56.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.507+0000 7ff1fe600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1f806fea0 0x7ff1f80c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:56.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.507+0000 7ff1fe600640 1 -- 172.21.15.6:0/900130246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806b080 msgr2=0x7ff1f81effd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:56.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.507+0000 7ff1fe600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806b080 0x7ff1f81effd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:56.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.507+0000 7ff1fe600640 1 -- 172.21.15.6:0/900130246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f81569e0 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.510+0000 7ff1fe600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1f80c2ea0 0x7ff1f80cbdf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000b970 tx=0x7ff1f000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:56.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.515+0000 7ff1f5600640 1 -- 172.21.15.6:0/900130246 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0019070 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.515+0000 7ff1f5600640 1 -- 172.21.15.6:0/900130246 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f000cb60 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.515+0000 7ff1f5600640 1 -- 172.21.15.6:0/900130246 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0014440 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.515+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f8156ca0 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.516+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f8157260 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.517+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f80756c0 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.546+0000 7ff1f5600640 1 -- 172.21.15.6:0/900130246 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff1f001c070 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.547+0000 7ff1f5600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1d806bc20 0x7ff1d806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:56.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.547+0000 7ff1fdc00640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1d806bc20 0x7ff1d806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:56.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.548+0000 7ff1fdc00640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1d806bc20 0x7ff1d806e0e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec007fd0 tx=0x7ff1ec007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:56.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.794+0000 7ff1f5600640 1 -- 172.21.15.6:0/900130246 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff1f0057130 con 0x7ff1f80c2ea0 2024-03-22T13:11:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:56.813+0000 7ff1f5600640 1 -- 172.21.15.6:0/900130246 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1f0023020 con 0x7ff1f80c2ea0 2024-03-22T13:11:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.083+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1f806be50 con 0x7ff1f80c2ea0 2024-03-22T13:11:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.098+0000 7ff1f5600640 1 -- 172.21.15.6:0/900130246 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7ff1f0059090 con 0x7ff1f80c2ea0 2024-03-22T13:11:57.098 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:57.098 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1d806bc20 msgr2=0x7ff1d806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1d806bc20 0x7ff1d806e0e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec007fd0 tx=0x7ff1ec007e80 comp rx=0 tx=0).stop 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1f80c2ea0 msgr2=0x7ff1f80cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1f80c2ea0 0x7ff1f80cbdf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000b970 tx=0x7ff1f000be30 comp rx=0 tx=0).stop 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 shutdown_connections 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1f80c2ea0 0x7ff1f80cbdf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff1d806bc20 0x7ff1d806e0e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1f806fea0 0x7ff1f80c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 --2- 172.21.15.6:0/900130246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1f806b080 0x7ff1f81effd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.101+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 >> 172.21.15.6:0/900130246 conn(0x7ff1f80ccb50 msgr2=0x7ff1f81532d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.102+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 shutdown_connections 2024-03-22T13:11:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:57.102+0000 7ff200600640 1 -- 172.21.15.6:0/900130246 wait complete. 2024-03-22T13:11:57.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:11:57.723 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:11:58.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:11:58.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.926+0000 7f3266800640 1 Processor -- start 2024-03-22T13:11:58.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.926+0000 7f3266800640 1 -- start start 2024-03-22T13:11:58.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.928+0000 7f3266800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3260152400 0x7f326015b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:58.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.928+0000 7f3266800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f326015bc80 0x7f32600716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:58.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.928+0000 7f3266800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f326006f8c0 0x7f3260071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:58.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.928+0000 7f3266800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32600724d0 con 0x7f326015bc80 2024-03-22T13:11:58.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.928+0000 7f3266800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32601590b0 con 0x7f326006f8c0 2024-03-22T13:11:58.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.928+0000 7f3266800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32600721f0 con 0x7f3260152400 2024-03-22T13:11:58.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.929+0000 7f325ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f326015bc80 0x7f32600716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:58.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.929+0000 7f325f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3260152400 0x7f326015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:58.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.929+0000 7f325fe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f326006f8c0 0x7f3260071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:58.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.929+0000 7f325ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f326015bc80 0x7f32600716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60346/0 (socket says 172.21.15.6:60346) 2024-03-22T13:11:58.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.929+0000 7f325ea00640 1 -- 172.21.15.6:0/2332760506 learned_addr learned my addr 172.21.15.6:0/2332760506 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:58.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.932+0000 7f325fe00640 1 -- 172.21.15.6:0/2332760506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3260152400 msgr2=0x7f326015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:58.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.932+0000 7f325fe00640 1 --2- 172.21.15.6:0/2332760506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3260152400 0x7f326015b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.932+0000 7f325fe00640 1 -- 172.21.15.6:0/2332760506 >> v2:172.21.15.6:3300/0 conn(0x7f326015bc80 msgr2=0x7f32600716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:11:58.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.932+0000 7f325fe00640 1 --2- 172.21.15.6:0/2332760506 >> v2:172.21.15.6:3300/0 conn(0x7f326015bc80 0x7f32600716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.932+0000 7f325fe00640 1 -- 172.21.15.6:0/2332760506 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32601582f0 con 0x7f326006f8c0 2024-03-22T13:11:58.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.935+0000 7f325fe00640 1 --2- 172.21.15.6:0/2332760506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326006f8c0 0x7f3260071c10 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f326006e7f0 tx=0x7f325802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa81341467024783 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:58.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f325e000640 1 -- 172.21.15.6:0/2332760506 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258035020 con 0x7f326006f8c0 2024-03-22T13:11:58.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f325e000640 1 -- 172.21.15.6:0/2332760506 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3258031830 con 0x7f326006f8c0 2024-03-22T13:11:58.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f325e000640 1 -- 172.21.15.6:0/2332760506 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258031c00 con 0x7f326006f8c0 2024-03-22T13:11:58.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f3266800640 1 -- 172.21.15.6:0/2332760506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326006f8c0 msgr2=0x7f3260071c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:58.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f3266800640 1 --2- 172.21.15.6:0/2332760506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326006f8c0 0x7f3260071c10 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f326006e7f0 tx=0x7f325802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f3266800640 1 -- 172.21.15.6:0/2332760506 shutdown_connections 2024-03-22T13:11:58.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f3266800640 1 --2- 172.21.15.6:0/2332760506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326006f8c0 0x7f3260071c10 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f3266800640 1 --2- 172.21.15.6:0/2332760506 >> v2:172.21.15.6:3300/0 conn(0x7f326015bc80 0x7f32600716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f3266800640 1 --2- 172.21.15.6:0/2332760506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3260152400 0x7f326015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.941+0000 7f3266800640 1 -- 172.21.15.6:0/2332760506 >> 172.21.15.6:0/2332760506 conn(0x7f32600cb930 msgr2=0x7f32600cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.942+0000 7f3266800640 1 -- 172.21.15.6:0/2332760506 shutdown_connections 2024-03-22T13:11:58.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.942+0000 7f3266800640 1 -- 172.21.15.6:0/2332760506 wait complete. 2024-03-22T13:11:58.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.943+0000 7f3266800640 1 Processor -- start 2024-03-22T13:11:58.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.943+0000 7f3266800640 1 -- start start 2024-03-22T13:11:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.944+0000 7f3266800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f326006f8c0 0x7f326015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.944+0000 7f3266800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3260152400 0x7f32600c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.944+0000 7f3266800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326015bc80 0x7f32600c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.944+0000 7f3266800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3260158b90 con 0x7f3260152400 2024-03-22T13:11:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.944+0000 7f3266800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32601588b0 con 0x7f326015bc80 2024-03-22T13:11:58.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.944+0000 7f3266800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32601527e0 con 0x7f326006f8c0 2024-03-22T13:11:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.945+0000 7f325fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326015bc80 0x7f32600c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.945+0000 7f325f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f326006f8c0 0x7f326015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:58.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.945+0000 7f325ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3260152400 0x7f32600c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.946+0000 7f325f400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f326006f8c0 0x7f326015a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:48294/0 (socket says 172.21.15.6:48294) 2024-03-22T13:11:58.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.946+0000 7f325f400640 1 -- 172.21.15.6:0/394977059 learned_addr learned my addr 172.21.15.6:0/394977059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:11:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.950+0000 7f325ea00640 1 -- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f326006f8c0 msgr2=0x7f326015a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.950+0000 7f325ea00640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f326006f8c0 0x7f326015a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.950+0000 7f325ea00640 1 -- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326015bc80 msgr2=0x7f32600c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.950+0000 7f325ea00640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326015bc80 0x7f32600c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.950+0000 7f325ea00640 1 -- 172.21.15.6:0/394977059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600c36d0 con 0x7f3260152400 2024-03-22T13:11:58.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.952+0000 7f325ea00640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3260152400 0x7f32600c44d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f324c00e840 tx=0x7f324c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:58.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.957+0000 7f3257400640 1 -- 172.21.15.6:0/394977059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c017070 con 0x7f3260152400 2024-03-22T13:11:58.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.957+0000 7f3257400640 1 -- 172.21.15.6:0/394977059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c00c730 con 0x7f3260152400 2024-03-22T13:11:58.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.957+0000 7f3257400640 1 -- 172.21.15.6:0/394977059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c00cb00 con 0x7f3260152400 2024-03-22T13:11:58.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.957+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32600c39b0 con 0x7f3260152400 2024-03-22T13:11:58.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.958+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3260157440 con 0x7f3260152400 2024-03-22T13:11:58.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.959+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32600756c0 con 0x7f3260152400 2024-03-22T13:11:58.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.985+0000 7f3257400640 1 -- 172.21.15.6:0/394977059 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f324c01a070 con 0x7f3260152400 2024-03-22T13:11:58.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.986+0000 7f3257400640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f324006bc50 0x7f324006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:11:58.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.986+0000 7f325f400640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f324006bc50 0x7f324006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:11:58.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:58.987+0000 7f325f400640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f324006bc50 0x7f324006e110 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3250009a40 tx=0x7f325000d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:11:59.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.282+0000 7f3257400640 1 -- 172.21.15.6:0/394977059 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f324c057460 con 0x7f3260152400 2024-03-22T13:11:59.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.283+0000 7f3257400640 1 -- 172.21.15.6:0/394977059 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f324c0593c0 con 0x7f3260152400 2024-03-22T13:11:59.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.540+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3260148f00 con 0x7f3260152400 2024-03-22T13:11:59.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.561+0000 7f3257400640 1 -- 172.21.15.6:0/394977059 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f324c05d460 con 0x7f3260152400 2024-03-22T13:11:59.561 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:11:59.561 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:11:59.561 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:11:59.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f324006bc50 msgr2=0x7f324006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:59.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f324006bc50 0x7f324006e110 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3250009a40 tx=0x7f325000d040 comp rx=0 tx=0).stop 2024-03-22T13:11:59.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3260152400 msgr2=0x7f32600c44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:11:59.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3260152400 0x7f32600c44d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f324c00e840 tx=0x7f324c00ed00 comp rx=0 tx=0).stop 2024-03-22T13:11:59.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 shutdown_connections 2024-03-22T13:11:59.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f324006bc50 0x7f324006e110 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:59.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f326015bc80 0x7f32600c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:59.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3260152400 0x7f32600c44d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:59.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 --2- 172.21.15.6:0/394977059 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f326006f8c0 0x7f326015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:11:59.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 >> 172.21.15.6:0/394977059 conn(0x7f32600cb930 msgr2=0x7f32600c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:11:59.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 shutdown_connections 2024-03-22T13:11:59.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:11:59.564+0000 7f3266800640 1 -- 172.21.15.6:0/394977059 wait complete. 2024-03-22T13:12:00.185 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:12:00.185 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:12:01.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T13:12:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.382+0000 7f76efe00640 1 Processor -- start 2024-03-22T13:12:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.382+0000 7f76efe00640 1 -- start start 2024-03-22T13:12:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.383+0000 7f76efe00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f76e8152400 0x7f76e815b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.383+0000 7f76efe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76e806f8c0 0x7f76e80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.383+0000 7f76efe00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e81590b0 con 0x7f76e815bc80 2024-03-22T13:12:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.383+0000 7f76efe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e80724d0 con 0x7f76e806f8c0 2024-03-22T13:12:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.383+0000 7f76efe00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e80721f0 con 0x7f76e8152400 2024-03-22T13:12:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.384+0000 7f76ed400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f76e8152400 0x7f76e815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:01.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.385+0000 7f76ed400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f76e8152400 0x7f76e815b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47512/0 (socket says 172.21.15.6:47512) 2024-03-22T13:12:01.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.385+0000 7f76ed400640 1 -- 172.21.15.6:0/571151120 learned_addr learned my addr 172.21.15.6:0/571151120 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:01.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.388+0000 7f76ed400640 1 -- 172.21.15.6:0/571151120 >> v2:172.21.15.6:3300/0 conn(0x7f76e806f8c0 msgr2=0x7f76e80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:01.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.388+0000 7f76ed400640 1 --2- 172.21.15.6:0/571151120 >> v2:172.21.15.6:3300/0 conn(0x7f76e806f8c0 0x7f76e80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:01.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.388+0000 7f76ed400640 1 -- 172.21.15.6:0/571151120 >> v1:172.21.15.92:6789/0 conn(0x7f76e815bc80 legacy=0x7f76e80716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:01.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.388+0000 7f76ed400640 1 -- 172.21.15.6:0/571151120 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e81582f0 con 0x7f76e8152400 2024-03-22T13:12:01.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.392+0000 7f76ed400640 1 --2- 172.21.15.6:0/571151120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e8152400 0x7f76e815b740 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f76e8071390 tx=0x7f76dc02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7c6d2101cfa5672 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:01.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.396+0000 7f76e7e00640 1 -- 172.21.15.6:0/571151120 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc02f850 con 0x7f76e8152400 2024-03-22T13:12:01.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.396+0000 7f76e7e00640 1 -- 172.21.15.6:0/571151120 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76dc02f9f0 con 0x7f76e8152400 2024-03-22T13:12:01.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.396+0000 7f76e7e00640 1 -- 172.21.15.6:0/571151120 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc02fdc0 con 0x7f76e8152400 2024-03-22T13:12:01.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 -- 172.21.15.6:0/571151120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e8152400 msgr2=0x7f76e815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:01.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 --2- 172.21.15.6:0/571151120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e8152400 0x7f76e815b740 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f76e8071390 tx=0x7f76dc02ba30 comp rx=0 tx=0).stop 2024-03-22T13:12:01.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 -- 172.21.15.6:0/571151120 shutdown_connections 2024-03-22T13:12:01.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 --2- 172.21.15.6:0/571151120 >> v2:172.21.15.6:3300/0 conn(0x7f76e806f8c0 0x7f76e80c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:01.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 --2- 172.21.15.6:0/571151120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e8152400 0x7f76e815b740 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:01.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 -- 172.21.15.6:0/571151120 >> 172.21.15.6:0/571151120 conn(0x7f76e80cb930 msgr2=0x7f76e80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:01.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 -- 172.21.15.6:0/571151120 shutdown_connections 2024-03-22T13:12:01.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.397+0000 7f76efe00640 1 -- 172.21.15.6:0/571151120 wait complete. 2024-03-22T13:12:01.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.399+0000 7f76efe00640 1 Processor -- start 2024-03-22T13:12:01.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.399+0000 7f76efe00640 1 -- start start 2024-03-22T13:12:01.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.400+0000 7f76efe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e806f8c0 0x7f76e8147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:01.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.400+0000 7f76efe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e8152400 0x7f76e8147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:01.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.400+0000 7f76efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e815bc80 0x7f76e814cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:01.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.400+0000 7f76efe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e8158b90 con 0x7f76e815bc80 2024-03-22T13:12:01.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.400+0000 7f76efe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e81588b0 con 0x7f76e806f8c0 2024-03-22T13:12:01.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.400+0000 7f76efe00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e81527e0 con 0x7f76e8152400 2024-03-22T13:12:01.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.401+0000 7f76ed400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e806f8c0 0x7f76e8147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:01.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.401+0000 7f76ede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e815bc80 0x7f76e814cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:01.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.401+0000 7f76eca00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e8152400 0x7f76e8147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:01.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.401+0000 7f76ed400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e806f8c0 0x7f76e8147a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:47516/0 (socket says 172.21.15.6:47516) 2024-03-22T13:12:01.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.401+0000 7f76ed400640 1 -- 172.21.15.6:0/42077760 learned_addr learned my addr 172.21.15.6:0/42077760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:01.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.405+0000 7f76ed400640 1 -- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e8152400 msgr2=0x7f76e8147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:01.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.405+0000 7f76ed400640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e8152400 0x7f76e8147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:01.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.405+0000 7f76ed400640 1 -- 172.21.15.6:0/42077760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e815bc80 msgr2=0x7f76e814cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:01.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.405+0000 7f76ed400640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e815bc80 0x7f76e814cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:01.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.405+0000 7f76ed400640 1 -- 172.21.15.6:0/42077760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e814e910 con 0x7f76e806f8c0 2024-03-22T13:12:01.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.408+0000 7f76ed400640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e806f8c0 0x7f76e8147a80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f76dc000c00 tx=0x7f76dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:01.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.413+0000 7f76e5600640 1 -- 172.21.15.6:0/42077760 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc037dd0 con 0x7f76e806f8c0 2024-03-22T13:12:01.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.413+0000 7f76e5600640 1 -- 172.21.15.6:0/42077760 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76dc047070 con 0x7f76e806f8c0 2024-03-22T13:12:01.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.413+0000 7f76e5600640 1 -- 172.21.15.6:0/42077760 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76dc0426c0 con 0x7f76e806f8c0 2024-03-22T13:12:01.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.413+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e814ebf0 con 0x7f76e806f8c0 2024-03-22T13:12:01.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.413+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e81538d0 con 0x7f76e806f8c0 2024-03-22T13:12:01.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.414+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e80756c0 con 0x7f76e806f8c0 2024-03-22T13:12:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.448+0000 7f76e5600640 1 -- 172.21.15.6:0/42077760 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f76dc040050 con 0x7f76e806f8c0 2024-03-22T13:12:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.449+0000 7f76e5600640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f76c806bc50 0x7f76c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:01.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.450+0000 7f76eca00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f76c806bc50 0x7f76c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.451+0000 7f76eca00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f76c806bc50 0x7f76c806e110 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f76e81504e0 tx=0x7f76d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:01.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.451+0000 7f76e5600640 1 -- 172.21.15.6:0/42077760 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f76dc02f980 con 0x7f76e806f8c0 2024-03-22T13:12:01.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:01.758+0000 7f76e5600640 1 -- 172.21.15.6:0/42077760 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f76dc033030 con 0x7f76e806f8c0 2024-03-22T13:12:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.016+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76e814c9c0 con 0x7f76e806f8c0 2024-03-22T13:12:02.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.036+0000 7f76e5600640 1 -- 172.21.15.6:0/42077760 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6029 (secure 0 0 0) 0x7f76dc07c690 con 0x7f76e806f8c0 2024-03-22T13:12:02.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:12:02.038 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T12:56:29.698344+0000","modified":"2024-03-22T13:09:29.617728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4268,"mds_1":4265,"mds_2":4261,"mds_3":4194,"mds_4":4252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4194":{"gid":4194,"name":"c","rank":3,"incarnation":19,"state":"up:resolve","state_seq":31,"addr":"172.21.15.6:6837/329268154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":329268154},{"type":"v1","addr":"172.21.15.6:6837","nonce":329268154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4252":{"gid":4252,"name":"f","rank":4,"incarnation":17,"state":"up:active","state_seq":16,"addr":"172.21.15.92:6835/3494871082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3494871082},{"type":"v1","addr":"172.21.15.92:6835","nonce":3494871082}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:18.421622+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4261":{"gid":4261,"name":"b","rank":2,"incarnation":13,"state":"up:active","state_seq":15,"addr":"172.21.15.92:6839/3559989974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3559989974},{"type":"v1","addr":"172.21.15.92:6839","nonce":3559989974}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:17.935911+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4265":{"gid":4265,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":14,"addr":"172.21.15.6:6833/2018699611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2018699611},{"type":"v1","addr":"172.21.15.6:6833","nonce":2018699611}]},"join_fscid":-1,"laggy_since":"2024-03-22T12:58:20.444154+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4268":{"gid":4268,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":40,"addr":"172.21.15.6:6835/1733016967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1733016967},{"type":"v1","addr":"172.21.15.6:6835","nonce":1733016967}]},"join_fscid":-1,"laggy_since":"2024-03-22T13:09:29.617726+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4268,"qdb_cluster":[4252,4265,4261,4268]},"id":1}]} 2024-03-22T13:12:02.038 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-03-22T13:12:02.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f76c806bc50 msgr2=0x7f76c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:02.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f76c806bc50 0x7f76c806e110 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f76e81504e0 tx=0x7f76d4009040 comp rx=0 tx=0).stop 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e806f8c0 msgr2=0x7f76e8147a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e806f8c0 0x7f76e8147a80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f76dc000c00 tx=0x7f76dc0046c0 comp rx=0 tx=0).stop 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 shutdown_connections 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76e815bc80 0x7f76e814cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76e8152400 0x7f76e8147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f76c806bc50 0x7f76c806e110 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 --2- 172.21.15.6:0/42077760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76e806f8c0 0x7f76e8147a80 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:02.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 >> 172.21.15.6:0/42077760 conn(0x7f76e80cb930 msgr2=0x7f76e80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:02.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.039+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 shutdown_connections 2024-03-22T13:12:02.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:02.040+0000 7f76efe00640 1 -- 172.21.15.6:0/42077760 wait complete. 2024-03-22T13:12:02.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T12:56:29.698344+0000', 'modified': '2024-03-22T13:09:29.617728+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 31, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4268, 'mds_1': 4265, 'mds_2': 4261, 'mds_3': 4194, 'mds_4': 4252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4194': {'gid': 4194, 'name': 'c', 'rank': 3, 'incarnation': 19, 'state': 'up:resolve', 'state_seq': 31, 'addr': '172.21.15.6:6837/329268154', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 329268154}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 329268154}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4252': {'gid': 4252, 'name': 'f', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.92:6835/3494871082', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3494871082}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3494871082}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:18.421622+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4261': {'gid': 4261, 'name': 'b', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.92:6839/3559989974', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 3559989974}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 3559989974}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:17.935911+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4265': {'gid': 4265, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.6:6833/2018699611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2018699611}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 2018699611}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T12:58:20.444154+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4268': {'gid': 4268, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.6:6835/1733016967', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1733016967}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1733016967}]}, 'join_fscid': -1, 'laggy_since': '2024-03-22T13:09:29.617726+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4268, 'qdb_cluster': [4252, 4265, 4261, 4268]} 2024-03-22T13:12:02.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4194:up:resolve 2024-03-22T13:12:03.672 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-22T12:56:29.698344+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4194": { "addr": "172.21.15.6:6837/329268154", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 329268154, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 329268154, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4194, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:resolve", "state_seq": 31 }, "gid_4252": { "addr": "172.21.15.92:6835/3494871082", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6834", "nonce": 3494871082, "type": "v2" }, { "addr": "172.21.15.92:6835", "nonce": 3494871082, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4252, "incarnation": 17, "join_fscid": -1, "laggy_since": "2024-03-22T12:58:18.421622+0000", "name": "f", "rank": 4, "state": "up:active", "state_seq": 16 }, "gid_4261": { "addr": "172.21.15.92:6839/3559989974", "addrs": { "addrvec": [ { "addr": "172.21.15.92:6838", "nonce": 3559989974, "type": "v2" }, { "addr": "172.21.15.92:6839", "nonce": 3559989974, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4261, "incarnation": 13, "join_fscid": -1, "laggy_since": "2024-03-22T12:58:17.935911+0000", "name": "b", "rank": 2, "state": "up:active", "state_seq": 15 }, "gid_4265": { "addr": "172.21.15.6:6833/2018699611", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 2018699611, "type": "v2" }, { "addr": "172.21.15.6:6833", "nonce": 2018699611, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4265, "incarnation": 10, "join_fscid": -1, "laggy_since": "2024-03-22T12:58:20.444154+0000", "name": "e", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_4268": { "addr": "172.21.15.6:6835/1733016967", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6834", "nonce": 1733016967, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 1733016967, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4268, "incarnation": 4, "join_fscid": -1, "laggy_since": "2024-03-22T13:09:29.617726+0000", "name": "a", "rank": 0, "state": "up:active", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-22T13:09:29.617728+0000", "qdb_cluster": [ 4252, 4265, 4261, 4268 ], "qdb_leader": 4268, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4268, "mds_1": 4265, "mds_2": 4261, "mds_3": 4194, "mds_4": 4252 } } } ], "standbys": [] } 2024-03-22T13:12:03.672 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-22T13:12:03.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T13:12:03.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:12:03.767 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:12:03.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.876+0000 7ff7a1a00640 1 Processor -- start 2024-03-22T13:12:03.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.876+0000 7ff7a1a00640 1 -- start start 2024-03-22T13:12:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.878+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ca520 0x7ff79c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:03.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.878+0000 7ff7a1a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff79c0ccad0 0x7ff79c15b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:03.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.878+0000 7ff7a1a00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79c06c7a0 con 0x7ff79c15b0e0 2024-03-22T13:12:03.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.878+0000 7ff7a1a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c0cd000 con 0x7ff79c0ccad0 2024-03-22T13:12:03.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.878+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c06ca80 con 0x7ff79c0ca520 2024-03-22T13:12:03.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.878+0000 7ff79aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ca520 0x7ff79c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:03.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.879+0000 7ff79b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff79c0ccad0 0x7ff79c15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:03.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.879+0000 7ff79aa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ca520 0x7ff79c06d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54210/0 (socket says 172.21.15.6:54210) 2024-03-22T13:12:03.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.879+0000 7ff79aa00640 1 -- 172.21.15.6:0/3590579876 learned_addr learned my addr 172.21.15.6:0/3590579876 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.882+0000 7ff79b400640 1 -- 172.21.15.6:0/3590579876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ca520 msgr2=0x7ff79c06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.882+0000 7ff79b400640 1 --2- 172.21.15.6:0/3590579876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ca520 0x7ff79c06d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.882+0000 7ff79b400640 1 -- 172.21.15.6:0/3590579876 >> v1:172.21.15.92:6789/0 conn(0x7ff79c15b0e0 legacy=0x7ff79c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:03.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.882+0000 7ff79b400640 1 -- 172.21.15.6:0/3590579876 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c06cd60 con 0x7ff79c0ccad0 2024-03-22T13:12:03.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.887+0000 7ff79b400640 1 --2- 172.21.15.6:0/3590579876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ccad0 0x7ff79c15b980 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff794009d90 tx=0x7ff79402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cfd535e8310b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.893+0000 7ff799600640 1 -- 172.21.15.6:0/3590579876 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794039070 con 0x7ff79c0ccad0 2024-03-22T13:12:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.893+0000 7ff799600640 1 -- 172.21.15.6:0/3590579876 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79402cbc0 con 0x7ff79c0ccad0 2024-03-22T13:12:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.893+0000 7ff799600640 1 -- 172.21.15.6:0/3590579876 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794034440 con 0x7ff79c0ccad0 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.893+0000 7ff7a1a00640 1 -- 172.21.15.6:0/3590579876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ccad0 msgr2=0x7ff79c15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.894+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/3590579876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ccad0 0x7ff79c15b980 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff794009d90 tx=0x7ff79402be10 comp rx=0 tx=0).stop 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.894+0000 7ff7a1a00640 1 -- 172.21.15.6:0/3590579876 shutdown_connections 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.894+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/3590579876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ccad0 0x7ff79c15b980 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.894+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/3590579876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ca520 0x7ff79c06d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.894+0000 7ff7a1a00640 1 -- 172.21.15.6:0/3590579876 >> 172.21.15.6:0/3590579876 conn(0x7ff79c0b16b0 msgr2=0x7ff79c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.894+0000 7ff7a1a00640 1 -- 172.21.15.6:0/3590579876 shutdown_connections 2024-03-22T13:12:03.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.894+0000 7ff7a1a00640 1 -- 172.21.15.6:0/3590579876 wait complete. 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.896+0000 7ff7a1a00640 1 Processor -- start 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.897+0000 7ff7a1a00640 1 -- start start 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.898+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ca520 0x7ff79c0b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.898+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ccad0 0x7ff79c0c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.898+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c15b0e0 0x7ff79c0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.898+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c154c20 con 0x7ff79c0ca520 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.898+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c06d320 con 0x7ff79c15b0e0 2024-03-22T13:12:03.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.898+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0ca900 con 0x7ff79c0ccad0 2024-03-22T13:12:03.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.899+0000 7ff79a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ccad0 0x7ff79c0c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:03.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.899+0000 7ff79aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ca520 0x7ff79c0b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:03.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.899+0000 7ff79a000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ccad0 0x7ff79c0c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54222/0 (socket says 172.21.15.6:54222) 2024-03-22T13:12:03.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.899+0000 7ff79a000640 1 -- 172.21.15.6:0/1985487954 learned_addr learned my addr 172.21.15.6:0/1985487954 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:03.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.899+0000 7ff79b400640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c15b0e0 0x7ff79c0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:03.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.904+0000 7ff79a000640 1 -- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c15b0e0 msgr2=0x7ff79c0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:03.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.904+0000 7ff79a000640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c15b0e0 0x7ff79c0c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:03.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.904+0000 7ff79a000640 1 -- 172.21.15.6:0/1985487954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ca520 msgr2=0x7ff79c0b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:03.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.904+0000 7ff79a000640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ca520 0x7ff79c0b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:03.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.904+0000 7ff79a000640 1 -- 172.21.15.6:0/1985487954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c1f3d00 con 0x7ff79c0ccad0 2024-03-22T13:12:03.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.908+0000 7ff79a000640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ccad0 0x7ff79c0c8670 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff78800b3f0 tx=0x7ff78800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.912+0000 7ff792a00640 1 -- 172.21.15.6:0/1985487954 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788015020 con 0x7ff79c0ccad0 2024-03-22T13:12:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.912+0000 7ff792a00640 1 -- 172.21.15.6:0/1985487954 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff788011830 con 0x7ff79c0ccad0 2024-03-22T13:12:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.912+0000 7ff792a00640 1 -- 172.21.15.6:0/1985487954 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788011c20 con 0x7ff79c0ccad0 2024-03-22T13:12:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.912+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c1f3fe0 con 0x7ff79c0ccad0 2024-03-22T13:12:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.912+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c1f45a0 con 0x7ff79c0ccad0 2024-03-22T13:12:03.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:03.913+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c0756c0 con 0x7ff79c0ccad0 2024-03-22T13:12:04.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.178+0000 7ff792a00640 1 -- 172.21.15.6:0/1985487954 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff788018020 con 0x7ff79c0ccad0 2024-03-22T13:12:04.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.180+0000 7ff792a00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff77c06bc50 0x7ff77c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:04.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.181+0000 7ff792a00640 1 -- 172.21.15.6:0/1985487954 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7ff78808ae50 con 0x7ff79c0ccad0 2024-03-22T13:12:04.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.181+0000 7ff79aa00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff77c06bc50 0x7ff77c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:04.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.182+0000 7ff79aa00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff77c06bc50 0x7ff77c06e110 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009bd0 tx=0x7ff78c009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:04.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.190+0000 7ff792a00640 1 -- 172.21.15.6:0/1985487954 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff78801f030 con 0x7ff79c0ccad0 2024-03-22T13:12:04.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.412+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff79c0cb080 con 0x7ff77c06bc50 2024-03-22T13:12:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.429+0000 7ff792a00640 1 -- 172.21.15.6:0/1985487954 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332703 (secure 0 0 0) 0x7ff79c0cb080 con 0x7ff77c06bc50 2024-03-22T13:12:04.434 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:12:04.436 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T13:12:04.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.436+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff77c06bc50 msgr2=0x7ff77c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:04.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.436+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff77c06bc50 0x7ff77c06e110 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009bd0 tx=0x7ff78c009240 comp rx=0 tx=0).stop 2024-03-22T13:12:04.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ccad0 msgr2=0x7ff79c0c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ccad0 0x7ff79c0c8670 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff78800b3f0 tx=0x7ff78800b8b0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 shutdown_connections 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7ff77c06bc50 0x7ff77c06e110 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c15b0e0 0x7ff79c0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff79c0ccad0 0x7ff79c0c8670 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 --2- 172.21.15.6:0/1985487954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff79c0ca520 0x7ff79c0b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 >> 172.21.15.6:0/1985487954 conn(0x7ff79c0b16b0 msgr2=0x7ff79c0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 shutdown_connections 2024-03-22T13:12:04.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.437+0000 7ff7a1a00640 1 -- 172.21.15.6:0/1985487954 wait complete. 2024-03-22T13:12:04.453 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-03-22T13:12:03.912288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2976,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":3047424,"data_stored":2312816,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003023"},"pg_stats":[{"pgid":"2.1f","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431578+0000","last_change":"2024-03-22T12:56:28.778294+0000","last_active":"2024-03-22T13:08:59.431578+0000","last_peered":"2024-03-22T13:08:59.431578+0000","last_clean":"2024-03-22T13:08:59.431578+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T13:08:59.431578+0000","last_undegraded":"2024-03-22T13:08:59.431578+0000","last_fullsized":"2024-03-22T13:08:59.431578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.355851+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T13:08:29.355851+0000","last_peered":"2024-03-22T13:08:29.355851+0000","last_clean":"2024-03-22T13:08:29.355851+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T13:08:29.355851+0000","last_undegraded":"2024-03-22T13:08:29.355851+0000","last_fullsized":"2024-03-22T13:08:29.355851+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.358314+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T13:08:44.358314+0000","last_peered":"2024-03-22T13:08:44.358314+0000","last_clean":"2024-03-22T13:08:44.358314+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T13:08:44.358314+0000","last_undegraded":"2024-03-22T13:08:44.358314+0000","last_fullsized":"2024-03-22T13:08:44.358314+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:35:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"30'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.355891+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T13:08:29.355891+0000","last_peered":"2024-03-22T13:08:29.355891+0000","last_clean":"2024-03-22T13:08:29.355891+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T13:08:29.355891+0000","last_undegraded":"2024-03-22T13:08:29.355891+0000","last_fullsized":"2024-03-22T13:08:29.355891+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431556+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T13:08:59.431556+0000","last_peered":"2024-03-22T13:08:59.431556+0000","last_clean":"2024-03-22T13:08:59.431556+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T13:08:59.431556+0000","last_undegraded":"2024-03-22T13:08:59.431556+0000","last_fullsized":"2024-03-22T13:08:59.431556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.473319+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T13:08:44.473319+0000","last_peered":"2024-03-22T13:08:44.473319+0000","last_clean":"2024-03-22T13:08:44.473319+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T13:08:44.473319+0000","last_undegraded":"2024-03-22T13:08:44.473319+0000","last_fullsized":"2024-03-22T13:08:44.473319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:40.477732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'12","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093797+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T13:08:58.093797+0000","last_peered":"2024-03-22T13:08:58.093797+0000","last_clean":"2024-03-22T13:08:58.093797+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T13:08:58.093797+0000","last_undegraded":"2024-03-22T13:08:58.093797+0000","last_fullsized":"2024-03-22T13:08:58.093797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452477+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T13:08:59.452477+0000","last_peered":"2024-03-22T13:08:59.452477+0000","last_clean":"2024-03-22T13:08:59.452477+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T13:08:59.452477+0000","last_undegraded":"2024-03-22T13:08:59.452477+0000","last_fullsized":"2024-03-22T13:08:59.452477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475708+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T13:09:09.475708+0000","last_peered":"2024-03-22T13:09:09.475708+0000","last_clean":"2024-03-22T13:09:09.475708+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T13:09:09.475708+0000","last_undegraded":"2024-03-22T13:09:09.475708+0000","last_fullsized":"2024-03-22T13:09:09.475708+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:29.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093761+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T13:08:58.093761+0000","last_peered":"2024-03-22T13:08:58.093761+0000","last_clean":"2024-03-22T13:08:58.093761+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T13:08:58.093761+0000","last_undegraded":"2024-03-22T13:08:58.093761+0000","last_fullsized":"2024-03-22T13:08:58.093761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452502+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T13:08:59.452502+0000","last_peered":"2024-03-22T13:08:59.452502+0000","last_clean":"2024-03-22T13:08:59.452502+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T13:08:59.452502+0000","last_undegraded":"2024-03-22T13:08:59.452502+0000","last_fullsized":"2024-03-22T13:08:59.452502+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475689+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T13:09:09.475689+0000","last_peered":"2024-03-22T13:09:09.475689+0000","last_clean":"2024-03-22T13:09:09.475689+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T13:09:09.475689+0000","last_undegraded":"2024-03-22T13:09:09.475689+0000","last_fullsized":"2024-03-22T13:09:09.475689+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049612+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T13:09:33.049612+0000","last_peered":"2024-03-22T13:09:33.049612+0000","last_clean":"2024-03-22T13:09:33.049612+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T13:09:33.049612+0000","last_undegraded":"2024-03-22T13:09:33.049612+0000","last_fullsized":"2024-03-22T13:09:33.049612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.086340+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T13:07:43.086340+0000","last_peered":"2024-03-22T13:07:43.086340+0000","last_clean":"2024-03-22T13:07:43.086340+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T13:07:43.086340+0000","last_undegraded":"2024-03-22T13:07:43.086340+0000","last_fullsized":"2024-03-22T13:07:43.086340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468904+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T13:07:54.468904+0000","last_peered":"2024-03-22T13:07:54.468904+0000","last_clean":"2024-03-22T13:07:54.468904+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T13:07:54.468904+0000","last_undegraded":"2024-03-22T13:07:54.468904+0000","last_fullsized":"2024-03-22T13:07:54.468904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:29.130909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.350707+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T13:07:44.350707+0000","last_peered":"2024-03-22T13:07:44.350707+0000","last_clean":"2024-03-22T13:07:44.350707+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T13:07:44.350707+0000","last_undegraded":"2024-03-22T13:07:44.350707+0000","last_fullsized":"2024-03-22T13:07:44.350707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"31'4","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542796+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T13:09:29.542796+0000","last_peered":"2024-03-22T13:09:29.542796+0000","last_clean":"2024-03-22T13:09:29.542796+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T13:09:29.542796+0000","last_undegraded":"2024-03-22T13:09:29.542796+0000","last_fullsized":"2024-03-22T13:09:29.542796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468869+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T13:07:54.468869+0000","last_peered":"2024-03-22T13:07:54.468869+0000","last_clean":"2024-03-22T13:07:54.468869+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T13:07:54.468869+0000","last_undegraded":"2024-03-22T13:07:54.468869+0000","last_fullsized":"2024-03-22T13:07:54.468869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:40.776689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363615+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T13:09:29.363615+0000","last_peered":"2024-03-22T13:09:29.363615+0000","last_clean":"2024-03-22T13:09:29.363615+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T13:09:29.363615+0000","last_undegraded":"2024-03-22T13:09:29.363615+0000","last_fullsized":"2024-03-22T13:09:29.363615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"30'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542761+0000","last_change":"2024-03-22T12:56:28.757253+0000","last_active":"2024-03-22T13:09:29.542761+0000","last_peered":"2024-03-22T13:09:29.542761+0000","last_clean":"2024-03-22T13:09:29.542761+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T13:09:29.542761+0000","last_undegraded":"2024-03-22T13:09:29.542761+0000","last_fullsized":"2024-03-22T13:09:29.542761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:48.506816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.529674+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T13:07:54.529674+0000","last_peered":"2024-03-22T13:07:54.529674+0000","last_clean":"2024-03-22T13:07:54.529674+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T13:07:54.529674+0000","last_undegraded":"2024-03-22T13:07:54.529674+0000","last_fullsized":"2024-03-22T13:07:54.529674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.477525+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T13:09:29.477525+0000","last_peered":"2024-03-22T13:09:29.477525+0000","last_clean":"2024-03-22T13:09:29.477525+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T13:09:29.477525+0000","last_undegraded":"2024-03-22T13:09:29.477525+0000","last_fullsized":"2024-03-22T13:09:29.477525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:34.508555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.096649+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T13:09:28.096649+0000","last_peered":"2024-03-22T13:09:28.096649+0000","last_clean":"2024-03-22T13:09:28.096649+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T13:09:28.096649+0000","last_undegraded":"2024-03-22T13:09:28.096649+0000","last_fullsized":"2024-03-22T13:09:28.096649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468920+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T13:07:54.468920+0000","last_peered":"2024-03-22T13:07:54.468920+0000","last_clean":"2024-03-22T13:07:54.468920+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T13:07:54.468920+0000","last_undegraded":"2024-03-22T13:07:54.468920+0000","last_fullsized":"2024-03-22T13:07:54.468920+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:09.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.435947+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T13:09:29.435947+0000","last_peered":"2024-03-22T13:09:29.435947+0000","last_clean":"2024-03-22T13:09:29.435947+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T13:09:29.435947+0000","last_undegraded":"2024-03-22T13:09:29.435947+0000","last_fullsized":"2024-03-22T13:09:29.435947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'7","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.049231+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T13:09:28.049231+0000","last_peered":"2024-03-22T13:09:28.049231+0000","last_clean":"2024-03-22T13:09:28.049231+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T13:09:28.049231+0000","last_undegraded":"2024-03-22T13:09:28.049231+0000","last_fullsized":"2024-03-22T13:09:28.049231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475657+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T13:09:09.475657+0000","last_peered":"2024-03-22T13:09:09.475657+0000","last_clean":"2024-03-22T13:09:09.475657+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T13:09:09.475657+0000","last_undegraded":"2024-03-22T13:09:09.475657+0000","last_fullsized":"2024-03-22T13:09:09.475657+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:13.307688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363715+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T13:09:29.363715+0000","last_peered":"2024-03-22T13:09:29.363715+0000","last_clean":"2024-03-22T13:09:29.363715+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T13:09:29.363715+0000","last_undegraded":"2024-03-22T13:09:29.363715+0000","last_fullsized":"2024-03-22T13:09:29.363715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:06.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"30'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542839+0000","last_change":"2024-03-22T12:56:28.767925+0000","last_active":"2024-03-22T13:09:29.542839+0000","last_peered":"2024-03-22T13:09:29.542839+0000","last_clean":"2024-03-22T13:09:29.542839+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T13:09:29.542839+0000","last_undegraded":"2024-03-22T13:09:29.542839+0000","last_fullsized":"2024-03-22T13:09:29.542839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:39.357442+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T13:08:39.357442+0000","last_peered":"2024-03-22T13:08:39.357442+0000","last_clean":"2024-03-22T13:08:39.357442+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T13:08:39.357442+0000","last_undegraded":"2024-03-22T13:08:39.357442+0000","last_fullsized":"2024-03-22T13:08:39.357442+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:40:09.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539070+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T13:08:59.539070+0000","last_peered":"2024-03-22T13:08:59.539070+0000","last_clean":"2024-03-22T13:08:59.539070+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T13:08:59.539070+0000","last_undegraded":"2024-03-22T13:08:59.539070+0000","last_fullsized":"2024-03-22T13:08:59.539070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093813+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T13:08:58.093813+0000","last_peered":"2024-03-22T13:08:58.093813+0000","last_clean":"2024-03-22T13:08:58.093813+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T13:08:58.093813+0000","last_undegraded":"2024-03-22T13:08:58.093813+0000","last_fullsized":"2024-03-22T13:08:58.093813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.433290+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T13:09:09.433290+0000","last_peered":"2024-03-22T13:09:09.433290+0000","last_clean":"2024-03-22T13:09:09.433290+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T13:09:09.433290+0000","last_undegraded":"2024-03-22T13:09:09.433290+0000","last_fullsized":"2024-03-22T13:09:09.433290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452512+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T13:08:59.452512+0000","last_peered":"2024-03-22T13:08:59.452512+0000","last_clean":"2024-03-22T13:08:59.452512+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T13:08:59.452512+0000","last_undegraded":"2024-03-22T13:08:59.452512+0000","last_fullsized":"2024-03-22T13:08:59.452512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539108+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T13:08:59.539108+0000","last_peered":"2024-03-22T13:08:59.539108+0000","last_clean":"2024-03-22T13:08:59.539108+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T13:08:59.539108+0000","last_undegraded":"2024-03-22T13:08:59.539108+0000","last_fullsized":"2024-03-22T13:08:59.539108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:38.043405+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T13:09:38.043405+0000","last_peered":"2024-03-22T13:09:38.043405+0000","last_clean":"2024-03-22T13:09:38.043405+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T13:09:38.043405+0000","last_undegraded":"2024-03-22T13:09:38.043405+0000","last_fullsized":"2024-03-22T13:09:38.043405+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:45.197847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.474702+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T13:08:59.474702+0000","last_peered":"2024-03-22T13:08:59.474702+0000","last_clean":"2024-03-22T13:08:59.474702+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T13:08:59.474702+0000","last_undegraded":"2024-03-22T13:08:59.474702+0000","last_fullsized":"2024-03-22T13:08:59.474702+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:23.839356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452447+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T13:08:59.452447+0000","last_peered":"2024-03-22T13:08:59.452447+0000","last_clean":"2024-03-22T13:08:59.452447+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T13:08:59.452447+0000","last_undegraded":"2024-03-22T13:08:59.452447+0000","last_fullsized":"2024-03-22T13:08:59.452447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.540600+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T13:09:09.540600+0000","last_peered":"2024-03-22T13:09:09.540600+0000","last_clean":"2024-03-22T13:09:09.540600+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T13:09:09.540600+0000","last_undegraded":"2024-03-22T13:09:09.540600+0000","last_fullsized":"2024-03-22T13:09:09.540600+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.534260+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T13:08:29.534260+0000","last_peered":"2024-03-22T13:08:29.534260+0000","last_clean":"2024-03-22T13:08:29.534260+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T13:08:29.534260+0000","last_undegraded":"2024-03-22T13:08:29.534260+0000","last_fullsized":"2024-03-22T13:08:29.534260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.037760+0000","last_change":"2024-03-22T12:56:28.763384+0000","last_active":"2024-03-22T13:08:58.037760+0000","last_peered":"2024-03-22T13:08:58.037760+0000","last_clean":"2024-03-22T13:08:58.037760+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T13:08:58.037760+0000","last_undegraded":"2024-03-22T13:08:58.037760+0000","last_fullsized":"2024-03-22T13:08:58.037760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:15:45.463056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:58.029027+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T13:07:58.029027+0000","last_peered":"2024-03-22T13:07:58.029027+0000","last_clean":"2024-03-22T13:07:58.029027+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T13:07:58.029027+0000","last_undegraded":"2024-03-22T13:07:58.029027+0000","last_fullsized":"2024-03-22T13:07:58.029027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:48.651995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431440+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T13:08:59.431440+0000","last_peered":"2024-03-22T13:08:59.431440+0000","last_clean":"2024-03-22T13:08:59.431440+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T13:08:59.431440+0000","last_undegraded":"2024-03-22T13:08:59.431440+0000","last_fullsized":"2024-03-22T13:08:59.431440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:36:50.048241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:28.090677+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T13:08:28.090677+0000","last_peered":"2024-03-22T13:08:28.090677+0000","last_clean":"2024-03-22T13:08:28.090677+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T13:08:28.090677+0000","last_undegraded":"2024-03-22T13:08:28.090677+0000","last_fullsized":"2024-03-22T13:08:28.090677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.437421+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T13:09:39.437421+0000","last_peered":"2024-03-22T13:09:39.437421+0000","last_clean":"2024-03-22T13:09:39.437421+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T13:09:39.437421+0000","last_undegraded":"2024-03-22T13:09:39.437421+0000","last_fullsized":"2024-03-22T13:09:39.437421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:29:31.400078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360426+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T13:08:59.360426+0000","last_peered":"2024-03-22T13:08:59.360426+0000","last_clean":"2024-03-22T13:08:59.360426+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T13:08:59.360426+0000","last_undegraded":"2024-03-22T13:08:59.360426+0000","last_fullsized":"2024-03-22T13:08:59.360426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452461+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T13:08:59.452461+0000","last_peered":"2024-03-22T13:08:59.452461+0000","last_clean":"2024-03-22T13:08:59.452461+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T13:08:59.452461+0000","last_undegraded":"2024-03-22T13:08:59.452461+0000","last_fullsized":"2024-03-22T13:08:59.452461+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":194,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:18.089652+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T13:08:18.089652+0000","last_peered":"2024-03-22T13:08:18.089652+0000","last_clean":"2024-03-22T13:08:18.089652+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T13:08:18.089652+0000","last_undegraded":"2024-03-22T13:08:18.089652+0000","last_fullsized":"2024-03-22T13:08:18.089652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047239+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T13:09:08.047239+0000","last_peered":"2024-03-22T13:09:08.047239+0000","last_clean":"2024-03-22T13:09:08.047239+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T13:09:08.047239+0000","last_undegraded":"2024-03-22T13:09:08.047239+0000","last_fullsized":"2024-03-22T13:09:08.047239+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086769+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T13:07:48.086769+0000","last_peered":"2024-03-22T13:07:48.086769+0000","last_clean":"2024-03-22T13:07:48.086769+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T13:07:48.086769+0000","last_undegraded":"2024-03-22T13:07:48.086769+0000","last_fullsized":"2024-03-22T13:07:48.086769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"31'6","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.026555+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T13:07:43.026555+0000","last_peered":"2024-03-22T13:07:43.026555+0000","last_clean":"2024-03-22T13:07:43.026555+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T13:07:43.026555+0000","last_undegraded":"2024-03-22T13:07:43.026555+0000","last_fullsized":"2024-03-22T13:07:43.026555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:38.050005+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T13:09:38.050005+0000","last_peered":"2024-03-22T13:09:38.050005+0000","last_clean":"2024-03-22T13:09:38.050005+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T13:09:38.050005+0000","last_undegraded":"2024-03-22T13:09:38.050005+0000","last_fullsized":"2024-03-22T13:09:38.050005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.445504+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T13:07:44.445504+0000","last_peered":"2024-03-22T13:07:44.445504+0000","last_clean":"2024-03-22T13:07:44.445504+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T13:07:44.445504+0000","last_undegraded":"2024-03-22T13:07:44.445504+0000","last_fullsized":"2024-03-22T13:07:44.445504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"31'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.528186+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T13:07:44.528186+0000","last_peered":"2024-03-22T13:07:44.528186+0000","last_clean":"2024-03-22T13:07:44.528186+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T13:07:44.528186+0000","last_undegraded":"2024-03-22T13:07:44.528186+0000","last_fullsized":"2024-03-22T13:07:44.528186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:22.329365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.421240+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T13:07:54.421240+0000","last_peered":"2024-03-22T13:07:54.421240+0000","last_clean":"2024-03-22T13:07:54.421240+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T13:07:54.421240+0000","last_undegraded":"2024-03-22T13:07:54.421240+0000","last_fullsized":"2024-03-22T13:07:54.421240+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086812+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T13:07:48.086812+0000","last_peered":"2024-03-22T13:07:48.086812+0000","last_clean":"2024-03-22T13:07:48.086812+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T13:07:48.086812+0000","last_undegraded":"2024-03-22T13:07:48.086812+0000","last_fullsized":"2024-03-22T13:07:48.086812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"31'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.026598+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T13:07:43.026598+0000","last_peered":"2024-03-22T13:07:43.026598+0000","last_clean":"2024-03-22T13:07:43.026598+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T13:07:43.026598+0000","last_undegraded":"2024-03-22T13:07:43.026598+0000","last_fullsized":"2024-03-22T13:07:43.026598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.456333+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T13:09:39.456333+0000","last_peered":"2024-03-22T13:09:39.456333+0000","last_clean":"2024-03-22T13:09:39.456333+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T13:09:39.456333+0000","last_undegraded":"2024-03-22T13:09:39.456333+0000","last_fullsized":"2024-03-22T13:09:39.456333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086795+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T13:07:48.086795+0000","last_peered":"2024-03-22T13:07:48.086795+0000","last_clean":"2024-03-22T13:07:48.086795+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T13:07:48.086795+0000","last_undegraded":"2024-03-22T13:07:48.086795+0000","last_fullsized":"2024-03-22T13:07:48.086795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"31'2","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:15.533469+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T13:08:15.533469+0000","last_peered":"2024-03-22T13:08:15.533469+0000","last_clean":"2024-03-22T13:08:15.533469+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T13:08:15.533469+0000","last_undegraded":"2024-03-22T13:08:15.533469+0000","last_fullsized":"2024-03-22T13:08:15.533469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:24:46.021828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.478501+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T13:09:39.478501+0000","last_peered":"2024-03-22T13:09:39.478501+0000","last_clean":"2024-03-22T13:09:39.478501+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T13:09:39.478501+0000","last_undegraded":"2024-03-22T13:09:39.478501+0000","last_fullsized":"2024-03-22T13:09:39.478501+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:38.316319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089125+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T13:08:13.089125+0000","last_peered":"2024-03-22T13:08:13.089125+0000","last_clean":"2024-03-22T13:08:13.089125+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T13:08:13.089125+0000","last_undegraded":"2024-03-22T13:08:13.089125+0000","last_fullsized":"2024-03-22T13:08:13.089125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"31'18","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.030746+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T13:08:08.030746+0000","last_peered":"2024-03-22T13:08:08.030746+0000","last_clean":"2024-03-22T13:08:08.030746+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T13:08:08.030746+0000","last_undegraded":"2024-03-22T13:08:08.030746+0000","last_fullsized":"2024-03-22T13:08:08.030746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:20:57.722062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.429460+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T13:08:44.429460+0000","last_peered":"2024-03-22T13:08:44.429460+0000","last_clean":"2024-03-22T13:08:44.429460+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T13:08:44.429460+0000","last_undegraded":"2024-03-22T13:08:44.429460+0000","last_fullsized":"2024-03-22T13:08:44.429460+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.042504+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T13:08:13.042504+0000","last_peered":"2024-03-22T13:08:13.042504+0000","last_clean":"2024-03-22T13:08:13.042504+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T13:08:13.042504+0000","last_undegraded":"2024-03-22T13:08:13.042504+0000","last_fullsized":"2024-03-22T13:08:13.042504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.041967+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T13:08:08.041967+0000","last_peered":"2024-03-22T13:08:08.041967+0000","last_clean":"2024-03-22T13:08:08.041967+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T13:08:08.041967+0000","last_undegraded":"2024-03-22T13:08:08.041967+0000","last_fullsized":"2024-03-22T13:08:08.041967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.358426+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T13:08:44.358426+0000","last_peered":"2024-03-22T13:08:44.358426+0000","last_clean":"2024-03-22T13:08:44.358426+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T13:08:44.358426+0000","last_undegraded":"2024-03-22T13:08:44.358426+0000","last_fullsized":"2024-03-22T13:08:44.358426+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:51.788457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089153+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T13:08:13.089153+0000","last_peered":"2024-03-22T13:08:13.089153+0000","last_clean":"2024-03-22T13:08:13.089153+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T13:08:13.089153+0000","last_undegraded":"2024-03-22T13:08:13.089153+0000","last_fullsized":"2024-03-22T13:08:13.089153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447799+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T13:08:09.447799+0000","last_peered":"2024-03-22T13:08:09.447799+0000","last_clean":"2024-03-22T13:08:09.447799+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T13:08:09.447799+0000","last_undegraded":"2024-03-22T13:08:09.447799+0000","last_fullsized":"2024-03-22T13:08:09.447799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354809+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T13:08:19.354809+0000","last_peered":"2024-03-22T13:08:19.354809+0000","last_clean":"2024-03-22T13:08:19.354809+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T13:08:19.354809+0000","last_undegraded":"2024-03-22T13:08:19.354809+0000","last_fullsized":"2024-03-22T13:08:19.354809+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:08:12.860933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423855+0000","last_change":"2024-03-22T12:56:30.299969+0000","last_active":"2024-03-22T13:08:09.423855+0000","last_peered":"2024-03-22T13:08:09.423855+0000","last_clean":"2024-03-22T13:08:09.423855+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T13:08:09.423855+0000","last_undegraded":"2024-03-22T13:08:09.423855+0000","last_fullsized":"2024-03-22T13:08:09.423855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'9","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423914+0000","last_change":"2024-03-22T12:56:28.777142+0000","last_active":"2024-03-22T13:08:09.423914+0000","last_peered":"2024-03-22T13:08:09.423914+0000","last_clean":"2024-03-22T13:08:09.423914+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T13:08:09.423914+0000","last_undegraded":"2024-03-22T13:08:09.423914+0000","last_fullsized":"2024-03-22T13:08:09.423914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:49.411808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.451139+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T13:08:44.451139+0000","last_peered":"2024-03-22T13:08:44.451139+0000","last_clean":"2024-03-22T13:08:44.451139+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T13:08:44.451139+0000","last_undegraded":"2024-03-22T13:08:44.451139+0000","last_fullsized":"2024-03-22T13:08:44.451139+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091724+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T13:08:38.091724+0000","last_peered":"2024-03-22T13:08:38.091724+0000","last_clean":"2024-03-22T13:08:38.091724+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T13:08:38.091724+0000","last_undegraded":"2024-03-22T13:08:38.091724+0000","last_fullsized":"2024-03-22T13:08:38.091724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'14","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.042011+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T13:08:08.042011+0000","last_peered":"2024-03-22T13:08:08.042011+0000","last_clean":"2024-03-22T13:08:08.042011+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T13:08:08.042011+0000","last_undegraded":"2024-03-22T13:08:08.042011+0000","last_fullsized":"2024-03-22T13:08:08.042011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533275+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T13:08:19.533275+0000","last_peered":"2024-03-22T13:08:19.533275+0000","last_clean":"2024-03-22T13:08:19.533275+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T13:08:19.533275+0000","last_undegraded":"2024-03-22T13:08:19.533275+0000","last_fullsized":"2024-03-22T13:08:19.533275+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.470116+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T13:08:09.470116+0000","last_peered":"2024-03-22T13:08:09.470116+0000","last_clean":"2024-03-22T13:08:09.470116+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T13:08:09.470116+0000","last_undegraded":"2024-03-22T13:08:09.470116+0000","last_fullsized":"2024-03-22T13:08:09.470116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:49.776407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.535102+0000","last_change":"2024-03-22T12:56:28.769041+0000","last_active":"2024-03-22T13:08:34.535102+0000","last_peered":"2024-03-22T13:08:34.535102+0000","last_clean":"2024-03-22T13:08:34.535102+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T13:08:34.535102+0000","last_undegraded":"2024-03-22T13:08:34.535102+0000","last_fullsized":"2024-03-22T13:08:34.535102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.471072+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T13:08:19.471072+0000","last_peered":"2024-03-22T13:08:19.471072+0000","last_clean":"2024-03-22T13:08:19.471072+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T13:08:19.471072+0000","last_undegraded":"2024-03-22T13:08:19.471072+0000","last_fullsized":"2024-03-22T13:08:19.471072+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:56.815720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.428126+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T13:08:34.428126+0000","last_peered":"2024-03-22T13:08:34.428126+0000","last_clean":"2024-03-22T13:08:34.428126+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T13:08:34.428126+0000","last_undegraded":"2024-03-22T13:08:34.428126+0000","last_fullsized":"2024-03-22T13:08:34.428126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450282+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T13:08:34.450282+0000","last_peered":"2024-03-22T13:08:34.450282+0000","last_clean":"2024-03-22T13:08:34.450282+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T13:08:34.450282+0000","last_undegraded":"2024-03-22T13:08:34.450282+0000","last_fullsized":"2024-03-22T13:08:34.450282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354747+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T13:08:19.354747+0000","last_peered":"2024-03-22T13:08:19.354747+0000","last_clean":"2024-03-22T13:08:19.354747+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T13:08:19.354747+0000","last_undegraded":"2024-03-22T13:08:19.354747+0000","last_fullsized":"2024-03-22T13:08:19.354747+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:06:47.716384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450270+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T13:08:34.450270+0000","last_peered":"2024-03-22T13:08:34.450270+0000","last_clean":"2024-03-22T13:08:34.450270+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T13:08:34.450270+0000","last_undegraded":"2024-03-22T13:08:34.450270+0000","last_fullsized":"2024-03-22T13:08:34.450270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:33.044141+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T13:08:33.044141+0000","last_peered":"2024-03-22T13:08:33.044141+0000","last_clean":"2024-03-22T13:08:33.044141+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T13:08:33.044141+0000","last_undegraded":"2024-03-22T13:08:33.044141+0000","last_fullsized":"2024-03-22T13:08:33.044141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354781+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T13:08:19.354781+0000","last_peered":"2024-03-22T13:08:19.354781+0000","last_clean":"2024-03-22T13:08:19.354781+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T13:08:19.354781+0000","last_undegraded":"2024-03-22T13:08:19.354781+0000","last_fullsized":"2024-03-22T13:08:19.354781+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:39.428904+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T13:08:39.428904+0000","last_peered":"2024-03-22T13:08:39.428904+0000","last_clean":"2024-03-22T13:08:39.428904+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T13:08:39.428904+0000","last_undegraded":"2024-03-22T13:08:39.428904+0000","last_fullsized":"2024-03-22T13:08:39.428904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.534984+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T13:08:34.534984+0000","last_peered":"2024-03-22T13:08:34.534984+0000","last_clean":"2024-03-22T13:08:34.534984+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T13:08:34.534984+0000","last_undegraded":"2024-03-22T13:08:34.534984+0000","last_fullsized":"2024-03-22T13:08:34.534984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450228+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T13:08:34.450228+0000","last_peered":"2024-03-22T13:08:34.450228+0000","last_clean":"2024-03-22T13:08:34.450228+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T13:08:34.450228+0000","last_undegraded":"2024-03-22T13:08:34.450228+0000","last_fullsized":"2024-03-22T13:08:34.450228+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047276+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T13:09:08.047276+0000","last_peered":"2024-03-22T13:09:08.047276+0000","last_clean":"2024-03-22T13:09:08.047276+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T13:09:08.047276+0000","last_undegraded":"2024-03-22T13:09:08.047276+0000","last_fullsized":"2024-03-22T13:09:08.047276+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356605+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T13:08:34.356605+0000","last_peered":"2024-03-22T13:08:34.356605+0000","last_clean":"2024-03-22T13:08:34.356605+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T13:08:34.356605+0000","last_undegraded":"2024-03-22T13:08:34.356605+0000","last_fullsized":"2024-03-22T13:08:34.356605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356655+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T13:08:34.356655+0000","last_peered":"2024-03-22T13:08:34.356655+0000","last_clean":"2024-03-22T13:08:34.356655+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T13:08:34.356655+0000","last_undegraded":"2024-03-22T13:08:34.356655+0000","last_fullsized":"2024-03-22T13:08:34.356655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:31:41.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.046243+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T13:08:58.046243+0000","last_peered":"2024-03-22T13:08:58.046243+0000","last_clean":"2024-03-22T13:08:58.046243+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T13:08:58.046243+0000","last_undegraded":"2024-03-22T13:08:58.046243+0000","last_fullsized":"2024-03-22T13:08:58.046243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:08.021699+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T13:07:08.021699+0000","last_peered":"2024-03-22T13:07:08.021699+0000","last_clean":"2024-03-22T13:07:08.021699+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T13:07:08.021699+0000","last_undegraded":"2024-03-22T13:07:08.021699+0000","last_fullsized":"2024-03-22T13:07:08.021699+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.039286+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T13:09:08.039286+0000","last_peered":"2024-03-22T13:09:08.039286+0000","last_clean":"2024-03-22T13:09:08.039286+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T13:09:08.039286+0000","last_undegraded":"2024-03-22T13:09:08.039286+0000","last_fullsized":"2024-03-22T13:09:08.039286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:43.256977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360446+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T13:08:59.360446+0000","last_peered":"2024-03-22T13:08:59.360446+0000","last_clean":"2024-03-22T13:08:59.360446+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T13:08:59.360446+0000","last_undegraded":"2024-03-22T13:08:59.360446+0000","last_fullsized":"2024-03-22T13:08:59.360446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.474675+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T13:08:59.474675+0000","last_peered":"2024-03-22T13:08:59.474675+0000","last_clean":"2024-03-22T13:08:59.474675+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T13:08:59.474675+0000","last_undegraded":"2024-03-22T13:08:59.474675+0000","last_fullsized":"2024-03-22T13:08:59.474675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:48.882251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.361490+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T13:09:09.361490+0000","last_peered":"2024-03-22T13:09:09.361490+0000","last_clean":"2024-03-22T13:09:09.361490+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T13:09:09.361490+0000","last_undegraded":"2024-03-22T13:09:09.361490+0000","last_fullsized":"2024-03-22T13:09:09.361490+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.046714+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T13:09:03.046714+0000","last_peered":"2024-03-22T13:09:03.046714+0000","last_clean":"2024-03-22T13:09:03.046714+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T13:09:03.046714+0000","last_undegraded":"2024-03-22T13:09:03.046714+0000","last_fullsized":"2024-03-22T13:09:03.046714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.038633+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T13:09:03.038633+0000","last_peered":"2024-03-22T13:09:03.038633+0000","last_clean":"2024-03-22T13:09:03.038633+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T13:09:03.038633+0000","last_undegraded":"2024-03-22T13:09:03.038633+0000","last_fullsized":"2024-03-22T13:09:03.038633+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452433+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T13:08:59.452433+0000","last_peered":"2024-03-22T13:08:59.452433+0000","last_clean":"2024-03-22T13:08:59.452433+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T13:08:59.452433+0000","last_undegraded":"2024-03-22T13:08:59.452433+0000","last_fullsized":"2024-03-22T13:08:59.452433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431501+0000","last_change":"2024-03-22T12:56:30.300244+0000","last_active":"2024-03-22T13:08:59.431501+0000","last_peered":"2024-03-22T13:08:59.431501+0000","last_clean":"2024-03-22T13:08:59.431501+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T13:08:59.431501+0000","last_undegraded":"2024-03-22T13:08:59.431501+0000","last_fullsized":"2024-03-22T13:08:59.431501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363660+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T13:09:29.363660+0000","last_peered":"2024-03-22T13:09:29.363660+0000","last_clean":"2024-03-22T13:09:29.363660+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T13:09:29.363660+0000","last_undegraded":"2024-03-22T13:09:29.363660+0000","last_fullsized":"2024-03-22T13:09:29.363660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542819+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T13:09:29.542819+0000","last_peered":"2024-03-22T13:09:29.542819+0000","last_clean":"2024-03-22T13:09:29.542819+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T13:09:29.542819+0000","last_undegraded":"2024-03-22T13:09:29.542819+0000","last_fullsized":"2024-03-22T13:09:29.542819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049650+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T13:09:33.049650+0000","last_peered":"2024-03-22T13:09:33.049650+0000","last_clean":"2024-03-22T13:09:33.049650+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T13:09:33.049650+0000","last_undegraded":"2024-03-22T13:09:33.049650+0000","last_fullsized":"2024-03-22T13:09:33.049650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363696+0000","last_change":"2024-03-22T12:56:30.308903+0000","last_active":"2024-03-22T13:09:29.363696+0000","last_peered":"2024-03-22T13:09:29.363696+0000","last_clean":"2024-03-22T13:09:29.363696+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T13:09:29.363696+0000","last_undegraded":"2024-03-22T13:09:29.363696+0000","last_fullsized":"2024-03-22T13:09:29.363696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:58.028924+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T13:07:58.028924+0000","last_peered":"2024-03-22T13:07:58.028924+0000","last_clean":"2024-03-22T13:07:58.028924+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T13:07:58.028924+0000","last_undegraded":"2024-03-22T13:07:58.028924+0000","last_fullsized":"2024-03-22T13:07:58.028924+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:13:57.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533242+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T13:08:19.533242+0000","last_peered":"2024-03-22T13:08:19.533242+0000","last_clean":"2024-03-22T13:08:19.533242+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T13:08:19.533242+0000","last_undegraded":"2024-03-22T13:08:19.533242+0000","last_fullsized":"2024-03-22T13:08:19.533242+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:19.797978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.445455+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T13:07:44.445455+0000","last_peered":"2024-03-22T13:07:44.445455+0000","last_clean":"2024-03-22T13:07:44.445455+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T13:07:44.445455+0000","last_undegraded":"2024-03-22T13:07:44.445455+0000","last_fullsized":"2024-03-22T13:07:44.445455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.419780+0000","last_change":"2024-03-22T12:56:30.299214+0000","last_active":"2024-03-22T13:07:44.419780+0000","last_peered":"2024-03-22T13:07:44.419780+0000","last_clean":"2024-03-22T13:07:44.419780+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T13:07:44.419780+0000","last_undegraded":"2024-03-22T13:07:44.419780+0000","last_fullsized":"2024-03-22T13:07:44.419780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:35.356723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.040052+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T13:07:48.040052+0000","last_peered":"2024-03-22T13:07:48.040052+0000","last_clean":"2024-03-22T13:07:48.040052+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T13:07:48.040052+0000","last_undegraded":"2024-03-22T13:07:48.040052+0000","last_fullsized":"2024-03-22T13:07:48.040052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533301+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T13:08:19.533301+0000","last_peered":"2024-03-22T13:08:19.533301+0000","last_clean":"2024-03-22T13:08:19.533301+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T13:08:19.533301+0000","last_undegraded":"2024-03-22T13:08:19.533301+0000","last_fullsized":"2024-03-22T13:08:19.533301+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:57:33.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.031478+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T13:08:13.031478+0000","last_peered":"2024-03-22T13:08:13.031478+0000","last_clean":"2024-03-22T13:08:13.031478+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T13:08:13.031478+0000","last_undegraded":"2024-03-22T13:08:13.031478+0000","last_fullsized":"2024-03-22T13:08:13.031478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:38.987151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447831+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T13:08:09.447831+0000","last_peered":"2024-03-22T13:08:09.447831+0000","last_clean":"2024-03-22T13:08:09.447831+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T13:08:09.447831+0000","last_undegraded":"2024-03-22T13:08:09.447831+0000","last_fullsized":"2024-03-22T13:08:09.447831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447764+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T13:08:09.447764+0000","last_peered":"2024-03-22T13:08:09.447764+0000","last_clean":"2024-03-22T13:08:09.447764+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T13:08:09.447764+0000","last_undegraded":"2024-03-22T13:08:09.447764+0000","last_fullsized":"2024-03-22T13:08:09.447764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.532025+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T13:08:09.532025+0000","last_peered":"2024-03-22T13:08:09.532025+0000","last_clean":"2024-03-22T13:08:09.532025+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T13:08:09.532025+0000","last_undegraded":"2024-03-22T13:08:09.532025+0000","last_fullsized":"2024-03-22T13:08:09.532025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423817+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T13:08:09.423817+0000","last_peered":"2024-03-22T13:08:09.423817+0000","last_clean":"2024-03-22T13:08:09.423817+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T13:08:09.423817+0000","last_undegraded":"2024-03-22T13:08:09.423817+0000","last_fullsized":"2024-03-22T13:08:09.423817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356700+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T13:08:34.356700+0000","last_peered":"2024-03-22T13:08:34.356700+0000","last_clean":"2024-03-22T13:08:34.356700+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T13:08:34.356700+0000","last_undegraded":"2024-03-22T13:08:34.356700+0000","last_fullsized":"2024-03-22T13:08:34.356700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.472507+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T13:08:34.472507+0000","last_peered":"2024-03-22T13:08:34.472507+0000","last_clean":"2024-03-22T13:08:34.472507+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T13:08:34.472507+0000","last_undegraded":"2024-03-22T13:08:34.472507+0000","last_fullsized":"2024-03-22T13:08:34.472507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:23.598956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044564+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T13:08:38.044564+0000","last_peered":"2024-03-22T13:08:38.044564+0000","last_clean":"2024-03-22T13:08:38.044564+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T13:08:38.044564+0000","last_undegraded":"2024-03-22T13:08:38.044564+0000","last_fullsized":"2024-03-22T13:08:38.044564+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044582+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T13:08:38.044582+0000","last_peered":"2024-03-22T13:08:38.044582+0000","last_clean":"2024-03-22T13:08:38.044582+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T13:08:38.044582+0000","last_undegraded":"2024-03-22T13:08:38.044582+0000","last_fullsized":"2024-03-22T13:08:38.044582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091763+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T13:08:38.091763+0000","last_peered":"2024-03-22T13:08:38.091763+0000","last_clean":"2024-03-22T13:08:38.091763+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T13:08:38.091763+0000","last_undegraded":"2024-03-22T13:08:38.091763+0000","last_fullsized":"2024-03-22T13:08:38.091763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360387+0000","last_change":"2024-03-22T12:56:30.309258+0000","last_active":"2024-03-22T13:08:59.360387+0000","last_peered":"2024-03-22T13:08:59.360387+0000","last_clean":"2024-03-22T13:08:59.360387+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T13:08:59.360387+0000","last_undegraded":"2024-03-22T13:08:59.360387+0000","last_fullsized":"2024-03-22T13:08:59.360387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094308+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T13:09:03.094308+0000","last_peered":"2024-03-22T13:09:03.094308+0000","last_clean":"2024-03-22T13:09:03.094308+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T13:09:03.094308+0000","last_undegraded":"2024-03-22T13:09:03.094308+0000","last_fullsized":"2024-03-22T13:09:03.094308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452409+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T13:08:59.452409+0000","last_peered":"2024-03-22T13:08:59.452409+0000","last_clean":"2024-03-22T13:08:59.452409+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T13:08:59.452409+0000","last_undegraded":"2024-03-22T13:08:59.452409+0000","last_fullsized":"2024-03-22T13:08:59.452409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094342+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T13:09:03.094342+0000","last_peered":"2024-03-22T13:09:03.094342+0000","last_clean":"2024-03-22T13:09:03.094342+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T13:09:03.094342+0000","last_undegraded":"2024-03-22T13:09:03.094342+0000","last_fullsized":"2024-03-22T13:09:03.094342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.449798+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T13:08:29.449798+0000","last_peered":"2024-03-22T13:08:29.449798+0000","last_clean":"2024-03-22T13:08:29.449798+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T13:08:29.449798+0000","last_undegraded":"2024-03-22T13:08:29.449798+0000","last_fullsized":"2024-03-22T13:08:29.449798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431526+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T13:08:59.431526+0000","last_peered":"2024-03-22T13:08:59.431526+0000","last_clean":"2024-03-22T13:08:59.431526+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T13:08:59.431526+0000","last_undegraded":"2024-03-22T13:08:59.431526+0000","last_fullsized":"2024-03-22T13:08:59.431526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:14.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539145+0000","last_change":"2024-03-22T12:56:30.303254+0000","last_active":"2024-03-22T13:08:59.539145+0000","last_peered":"2024-03-22T13:08:59.539145+0000","last_clean":"2024-03-22T13:08:59.539145+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T13:08:59.539145+0000","last_undegraded":"2024-03-22T13:08:59.539145+0000","last_fullsized":"2024-03-22T13:08:59.539145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094374+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T13:09:03.094374+0000","last_peered":"2024-03-22T13:09:03.094374+0000","last_clean":"2024-03-22T13:09:03.094374+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T13:09:03.094374+0000","last_undegraded":"2024-03-22T13:09:03.094374+0000","last_fullsized":"2024-03-22T13:09:03.094374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509633,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":142876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509633,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":140274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509633,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":143610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":140476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509632,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":730186,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542339,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":731412,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":143576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542338,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T13:12:04.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T13:12:04.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.675+0000 7fd84a600640 1 Processor -- start 2024-03-22T13:12:04.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.675+0000 7fd84a600640 1 -- start start 2024-03-22T13:12:04.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.676+0000 7fd84a600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd8440ce8a0 0x7fd84406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:04.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.676+0000 7fd84a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd84415ae30 0x7fd8440cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:04.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.676+0000 7fd84a600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84415b240 con 0x7fd8440cc380 2024-03-22T13:12:04.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.676+0000 7fd84a600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8440cee00 con 0x7fd84415ae30 2024-03-22T13:12:04.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.676+0000 7fd84a600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8440cf0c0 con 0x7fd8440ce8a0 2024-03-22T13:12:04.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.677+0000 7fd843e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd8440ce8a0 0x7fd84406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:04.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.677+0000 7fd849c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd84415ae30 0x7fd8440cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:04.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.678+0000 7fd849200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd8440cc380 0x7fd8440cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43782/0 (socket says 172.21.15.6:43782) 2024-03-22T13:12:04.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.678+0000 7fd849200640 1 -- 172.21.15.6:0/2373273836 learned_addr learned my addr 172.21.15.6:0/2373273836 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:04.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.680+0000 7fd849c00640 1 -- 172.21.15.6:0/2373273836 >> v2:172.21.15.92:3300/0 conn(0x7fd8440ce8a0 msgr2=0x7fd84406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:04.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.680+0000 7fd849c00640 1 --2- 172.21.15.6:0/2373273836 >> v2:172.21.15.92:3300/0 conn(0x7fd8440ce8a0 0x7fd84406c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.680+0000 7fd849c00640 1 -- 172.21.15.6:0/2373273836 >> v1:172.21.15.92:6789/0 conn(0x7fd8440cc380 legacy=0x7fd8440cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:04.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.680+0000 7fd849c00640 1 -- 172.21.15.6:0/2373273836 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440cf3a0 con 0x7fd84415ae30 2024-03-22T13:12:04.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.683+0000 7fd849c00640 1 --2- 172.21.15.6:0/2373273836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 0x7fd8440cc760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd8440cb7f0 tx=0x7fd83c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec4a3ed04bd1e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:04.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.688+0000 7fd843400640 1 -- 172.21.15.6:0/2373273836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c039070 con 0x7fd84415ae30 2024-03-22T13:12:04.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.689+0000 7fd843400640 1 -- 172.21.15.6:0/2373273836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c02cc80 con 0x7fd84415ae30 2024-03-22T13:12:04.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.689+0000 7fd843400640 1 -- 172.21.15.6:0/2373273836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c034660 con 0x7fd84415ae30 2024-03-22T13:12:04.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.689+0000 7fd84a600640 1 -- 172.21.15.6:0/2373273836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 msgr2=0x7fd8440cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:04.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.689+0000 7fd84a600640 1 --2- 172.21.15.6:0/2373273836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 0x7fd8440cc760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd8440cb7f0 tx=0x7fd83c02bed0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.690+0000 7fd84a600640 1 -- 172.21.15.6:0/2373273836 shutdown_connections 2024-03-22T13:12:04.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.690+0000 7fd84a600640 1 --2- 172.21.15.6:0/2373273836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 0x7fd8440cc760 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.690+0000 7fd84a600640 1 --2- 172.21.15.6:0/2373273836 >> v2:172.21.15.92:3300/0 conn(0x7fd8440ce8a0 0x7fd84406c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.690+0000 7fd84a600640 1 -- 172.21.15.6:0/2373273836 >> 172.21.15.6:0/2373273836 conn(0x7fd8440b1480 msgr2=0x7fd8440b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:04.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.690+0000 7fd84a600640 1 -- 172.21.15.6:0/2373273836 shutdown_connections 2024-03-22T13:12:04.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.690+0000 7fd84a600640 1 -- 172.21.15.6:0/2373273836 wait complete. 2024-03-22T13:12:04.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.692+0000 7fd84a600640 1 Processor -- start 2024-03-22T13:12:04.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.692+0000 7fd84a600640 1 -- start start 2024-03-22T13:12:04.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.693+0000 7fd84a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8440cc380 0x7fd8441f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:04.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.693+0000 7fd84a600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8440ce8a0 0x7fd8441f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:04.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.693+0000 7fd84a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 0x7fd8441f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:04.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.693+0000 7fd84a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84415a910 con 0x7fd84415ae30 2024-03-22T13:12:04.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.693+0000 7fd84a600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84415a630 con 0x7fd8440cc380 2024-03-22T13:12:04.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.693+0000 7fd84a600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8441549d0 con 0x7fd8440ce8a0 2024-03-22T13:12:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.694+0000 7fd849200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8440cc380 0x7fd8441f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.694+0000 7fd849c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 0x7fd8441f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.694+0000 7fd843e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8440ce8a0 0x7fd8441f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.694+0000 7fd849200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8440cc380 0x7fd8441f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36832/0 (socket says 172.21.15.6:36832) 2024-03-22T13:12:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.694+0000 7fd849200640 1 -- 172.21.15.6:0/1385916030 learned_addr learned my addr 172.21.15.6:0/1385916030 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.697+0000 7fd849200640 1 -- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8440ce8a0 msgr2=0x7fd8441f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.697+0000 7fd849200640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8440ce8a0 0x7fd8441f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.697+0000 7fd849200640 1 -- 172.21.15.6:0/1385916030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 msgr2=0x7fd8441f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.697+0000 7fd849200640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 0x7fd8441f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:04.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.697+0000 7fd849200640 1 -- 172.21.15.6:0/1385916030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd844153d20 con 0x7fd8440cc380 2024-03-22T13:12:04.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.700+0000 7fd849200640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8440cc380 0x7fd8441f1440 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c3c0 tx=0x7fd83800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.702+0000 7fd840c00640 1 -- 172.21.15.6:0/1385916030 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838017070 con 0x7fd8440cc380 2024-03-22T13:12:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.702+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8440b2630 con 0x7fd8440cc380 2024-03-22T13:12:04.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.703+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8440b3bf0 con 0x7fd8440cc380 2024-03-22T13:12:04.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.704+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84406cd90 con 0x7fd8440cc380 2024-03-22T13:12:04.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:04.913+0000 7fd840c00640 1 -- 172.21.15.6:0/1385916030 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd838005ce0 con 0x7fd8440cc380 2024-03-22T13:12:05.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.015+0000 7fd840c00640 1 -- 172.21.15.6:0/1385916030 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838021060 con 0x7fd8440cc380 2024-03-22T13:12:05.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.018+0000 7fd840c00640 1 -- 172.21.15.6:0/1385916030 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd83801a070 con 0x7fd8440cc380 2024-03-22T13:12:05.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.020+0000 7fd840c00640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd82406bc00 0x7fd82406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:05.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.020+0000 7fd843e00640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd82406bc00 0x7fd82406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:05.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.021+0000 7fd840c00640 1 -- 172.21.15.6:0/1385916030 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fd838056a20 con 0x7fd8440cc380 2024-03-22T13:12:05.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.021+0000 7fd843e00640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd82406bc00 0x7fd82406e0c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd8440b1400 tx=0x7fd830009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.045+0000 7fd840c00640 1 -- 172.21.15.6:0/1385916030 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd84406cd90 con 0x7fd8440cc380 2024-03-22T13:12:05.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.269+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd844075650 con 0x7fd82406bc00 2024-03-22T13:12:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.287+0000 7fd840c00640 1 -- 172.21.15.6:0/1385916030 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332703 (secure 0 0 0) 0x7fd844075650 con 0x7fd82406bc00 2024-03-22T13:12:05.291 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:12:05.293 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T13:12:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.292+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd82406bc00 msgr2=0x7fd82406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.292+0000 7fd84a600640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd82406bc00 0x7fd82406e0c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd8440b1400 tx=0x7fd830009190 comp rx=0 tx=0).stop 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.292+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8440cc380 msgr2=0x7fd8441f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.292+0000 7fd84a600640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8440cc380 0x7fd8441f1440 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c3c0 tx=0x7fd83800c880 comp rx=0 tx=0).stop 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 shutdown_connections 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd84415ae30 0x7fd8441f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8440ce8a0 0x7fd8441f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fd82406bc00 0x7fd82406e0c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 --2- 172.21.15.6:0/1385916030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8440cc380 0x7fd8441f1440 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 >> 172.21.15.6:0/1385916030 conn(0x7fd8440b1480 msgr2=0x7fd8440cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 shutdown_connections 2024-03-22T13:12:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.293+0000 7fd84a600640 1 -- 172.21.15.6:0/1385916030 wait complete. 2024-03-22T13:12:05.310 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-03-22T13:12:03.912288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2976,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":3047424,"data_stored":2312816,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003023"},"pg_stats":[{"pgid":"2.1f","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431578+0000","last_change":"2024-03-22T12:56:28.778294+0000","last_active":"2024-03-22T13:08:59.431578+0000","last_peered":"2024-03-22T13:08:59.431578+0000","last_clean":"2024-03-22T13:08:59.431578+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T13:08:59.431578+0000","last_undegraded":"2024-03-22T13:08:59.431578+0000","last_fullsized":"2024-03-22T13:08:59.431578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.355851+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T13:08:29.355851+0000","last_peered":"2024-03-22T13:08:29.355851+0000","last_clean":"2024-03-22T13:08:29.355851+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T13:08:29.355851+0000","last_undegraded":"2024-03-22T13:08:29.355851+0000","last_fullsized":"2024-03-22T13:08:29.355851+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.358314+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T13:08:44.358314+0000","last_peered":"2024-03-22T13:08:44.358314+0000","last_clean":"2024-03-22T13:08:44.358314+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T13:08:44.358314+0000","last_undegraded":"2024-03-22T13:08:44.358314+0000","last_fullsized":"2024-03-22T13:08:44.358314+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:35:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"30'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.355891+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T13:08:29.355891+0000","last_peered":"2024-03-22T13:08:29.355891+0000","last_clean":"2024-03-22T13:08:29.355891+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T13:08:29.355891+0000","last_undegraded":"2024-03-22T13:08:29.355891+0000","last_fullsized":"2024-03-22T13:08:29.355891+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431556+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T13:08:59.431556+0000","last_peered":"2024-03-22T13:08:59.431556+0000","last_clean":"2024-03-22T13:08:59.431556+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T13:08:59.431556+0000","last_undegraded":"2024-03-22T13:08:59.431556+0000","last_fullsized":"2024-03-22T13:08:59.431556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.473319+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T13:08:44.473319+0000","last_peered":"2024-03-22T13:08:44.473319+0000","last_clean":"2024-03-22T13:08:44.473319+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T13:08:44.473319+0000","last_undegraded":"2024-03-22T13:08:44.473319+0000","last_fullsized":"2024-03-22T13:08:44.473319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:40.477732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'12","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093797+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T13:08:58.093797+0000","last_peered":"2024-03-22T13:08:58.093797+0000","last_clean":"2024-03-22T13:08:58.093797+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T13:08:58.093797+0000","last_undegraded":"2024-03-22T13:08:58.093797+0000","last_fullsized":"2024-03-22T13:08:58.093797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452477+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T13:08:59.452477+0000","last_peered":"2024-03-22T13:08:59.452477+0000","last_clean":"2024-03-22T13:08:59.452477+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T13:08:59.452477+0000","last_undegraded":"2024-03-22T13:08:59.452477+0000","last_fullsized":"2024-03-22T13:08:59.452477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475708+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T13:09:09.475708+0000","last_peered":"2024-03-22T13:09:09.475708+0000","last_clean":"2024-03-22T13:09:09.475708+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T13:09:09.475708+0000","last_undegraded":"2024-03-22T13:09:09.475708+0000","last_fullsized":"2024-03-22T13:09:09.475708+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:29.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093761+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T13:08:58.093761+0000","last_peered":"2024-03-22T13:08:58.093761+0000","last_clean":"2024-03-22T13:08:58.093761+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T13:08:58.093761+0000","last_undegraded":"2024-03-22T13:08:58.093761+0000","last_fullsized":"2024-03-22T13:08:58.093761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452502+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T13:08:59.452502+0000","last_peered":"2024-03-22T13:08:59.452502+0000","last_clean":"2024-03-22T13:08:59.452502+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T13:08:59.452502+0000","last_undegraded":"2024-03-22T13:08:59.452502+0000","last_fullsized":"2024-03-22T13:08:59.452502+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475689+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T13:09:09.475689+0000","last_peered":"2024-03-22T13:09:09.475689+0000","last_clean":"2024-03-22T13:09:09.475689+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T13:09:09.475689+0000","last_undegraded":"2024-03-22T13:09:09.475689+0000","last_fullsized":"2024-03-22T13:09:09.475689+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049612+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T13:09:33.049612+0000","last_peered":"2024-03-22T13:09:33.049612+0000","last_clean":"2024-03-22T13:09:33.049612+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T13:09:33.049612+0000","last_undegraded":"2024-03-22T13:09:33.049612+0000","last_fullsized":"2024-03-22T13:09:33.049612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.086340+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T13:07:43.086340+0000","last_peered":"2024-03-22T13:07:43.086340+0000","last_clean":"2024-03-22T13:07:43.086340+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T13:07:43.086340+0000","last_undegraded":"2024-03-22T13:07:43.086340+0000","last_fullsized":"2024-03-22T13:07:43.086340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468904+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T13:07:54.468904+0000","last_peered":"2024-03-22T13:07:54.468904+0000","last_clean":"2024-03-22T13:07:54.468904+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T13:07:54.468904+0000","last_undegraded":"2024-03-22T13:07:54.468904+0000","last_fullsized":"2024-03-22T13:07:54.468904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:29.130909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.350707+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T13:07:44.350707+0000","last_peered":"2024-03-22T13:07:44.350707+0000","last_clean":"2024-03-22T13:07:44.350707+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T13:07:44.350707+0000","last_undegraded":"2024-03-22T13:07:44.350707+0000","last_fullsized":"2024-03-22T13:07:44.350707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"31'4","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542796+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T13:09:29.542796+0000","last_peered":"2024-03-22T13:09:29.542796+0000","last_clean":"2024-03-22T13:09:29.542796+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T13:09:29.542796+0000","last_undegraded":"2024-03-22T13:09:29.542796+0000","last_fullsized":"2024-03-22T13:09:29.542796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468869+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T13:07:54.468869+0000","last_peered":"2024-03-22T13:07:54.468869+0000","last_clean":"2024-03-22T13:07:54.468869+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T13:07:54.468869+0000","last_undegraded":"2024-03-22T13:07:54.468869+0000","last_fullsized":"2024-03-22T13:07:54.468869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:40.776689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363615+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T13:09:29.363615+0000","last_peered":"2024-03-22T13:09:29.363615+0000","last_clean":"2024-03-22T13:09:29.363615+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T13:09:29.363615+0000","last_undegraded":"2024-03-22T13:09:29.363615+0000","last_fullsized":"2024-03-22T13:09:29.363615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"30'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542761+0000","last_change":"2024-03-22T12:56:28.757253+0000","last_active":"2024-03-22T13:09:29.542761+0000","last_peered":"2024-03-22T13:09:29.542761+0000","last_clean":"2024-03-22T13:09:29.542761+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T13:09:29.542761+0000","last_undegraded":"2024-03-22T13:09:29.542761+0000","last_fullsized":"2024-03-22T13:09:29.542761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:48.506816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.529674+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T13:07:54.529674+0000","last_peered":"2024-03-22T13:07:54.529674+0000","last_clean":"2024-03-22T13:07:54.529674+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T13:07:54.529674+0000","last_undegraded":"2024-03-22T13:07:54.529674+0000","last_fullsized":"2024-03-22T13:07:54.529674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.477525+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T13:09:29.477525+0000","last_peered":"2024-03-22T13:09:29.477525+0000","last_clean":"2024-03-22T13:09:29.477525+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T13:09:29.477525+0000","last_undegraded":"2024-03-22T13:09:29.477525+0000","last_fullsized":"2024-03-22T13:09:29.477525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:34.508555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.096649+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T13:09:28.096649+0000","last_peered":"2024-03-22T13:09:28.096649+0000","last_clean":"2024-03-22T13:09:28.096649+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T13:09:28.096649+0000","last_undegraded":"2024-03-22T13:09:28.096649+0000","last_fullsized":"2024-03-22T13:09:28.096649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468920+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T13:07:54.468920+0000","last_peered":"2024-03-22T13:07:54.468920+0000","last_clean":"2024-03-22T13:07:54.468920+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T13:07:54.468920+0000","last_undegraded":"2024-03-22T13:07:54.468920+0000","last_fullsized":"2024-03-22T13:07:54.468920+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:09.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.435947+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T13:09:29.435947+0000","last_peered":"2024-03-22T13:09:29.435947+0000","last_clean":"2024-03-22T13:09:29.435947+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T13:09:29.435947+0000","last_undegraded":"2024-03-22T13:09:29.435947+0000","last_fullsized":"2024-03-22T13:09:29.435947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'7","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.049231+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T13:09:28.049231+0000","last_peered":"2024-03-22T13:09:28.049231+0000","last_clean":"2024-03-22T13:09:28.049231+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T13:09:28.049231+0000","last_undegraded":"2024-03-22T13:09:28.049231+0000","last_fullsized":"2024-03-22T13:09:28.049231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475657+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T13:09:09.475657+0000","last_peered":"2024-03-22T13:09:09.475657+0000","last_clean":"2024-03-22T13:09:09.475657+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T13:09:09.475657+0000","last_undegraded":"2024-03-22T13:09:09.475657+0000","last_fullsized":"2024-03-22T13:09:09.475657+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:13.307688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363715+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T13:09:29.363715+0000","last_peered":"2024-03-22T13:09:29.363715+0000","last_clean":"2024-03-22T13:09:29.363715+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T13:09:29.363715+0000","last_undegraded":"2024-03-22T13:09:29.363715+0000","last_fullsized":"2024-03-22T13:09:29.363715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:06.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"30'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542839+0000","last_change":"2024-03-22T12:56:28.767925+0000","last_active":"2024-03-22T13:09:29.542839+0000","last_peered":"2024-03-22T13:09:29.542839+0000","last_clean":"2024-03-22T13:09:29.542839+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T13:09:29.542839+0000","last_undegraded":"2024-03-22T13:09:29.542839+0000","last_fullsized":"2024-03-22T13:09:29.542839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:39.357442+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T13:08:39.357442+0000","last_peered":"2024-03-22T13:08:39.357442+0000","last_clean":"2024-03-22T13:08:39.357442+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T13:08:39.357442+0000","last_undegraded":"2024-03-22T13:08:39.357442+0000","last_fullsized":"2024-03-22T13:08:39.357442+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:40:09.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539070+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T13:08:59.539070+0000","last_peered":"2024-03-22T13:08:59.539070+0000","last_clean":"2024-03-22T13:08:59.539070+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T13:08:59.539070+0000","last_undegraded":"2024-03-22T13:08:59.539070+0000","last_fullsized":"2024-03-22T13:08:59.539070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093813+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T13:08:58.093813+0000","last_peered":"2024-03-22T13:08:58.093813+0000","last_clean":"2024-03-22T13:08:58.093813+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T13:08:58.093813+0000","last_undegraded":"2024-03-22T13:08:58.093813+0000","last_fullsized":"2024-03-22T13:08:58.093813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.433290+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T13:09:09.433290+0000","last_peered":"2024-03-22T13:09:09.433290+0000","last_clean":"2024-03-22T13:09:09.433290+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T13:09:09.433290+0000","last_undegraded":"2024-03-22T13:09:09.433290+0000","last_fullsized":"2024-03-22T13:09:09.433290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452512+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T13:08:59.452512+0000","last_peered":"2024-03-22T13:08:59.452512+0000","last_clean":"2024-03-22T13:08:59.452512+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T13:08:59.452512+0000","last_undegraded":"2024-03-22T13:08:59.452512+0000","last_fullsized":"2024-03-22T13:08:59.452512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539108+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T13:08:59.539108+0000","last_peered":"2024-03-22T13:08:59.539108+0000","last_clean":"2024-03-22T13:08:59.539108+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T13:08:59.539108+0000","last_undegraded":"2024-03-22T13:08:59.539108+0000","last_fullsized":"2024-03-22T13:08:59.539108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:38.043405+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T13:09:38.043405+0000","last_peered":"2024-03-22T13:09:38.043405+0000","last_clean":"2024-03-22T13:09:38.043405+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T13:09:38.043405+0000","last_undegraded":"2024-03-22T13:09:38.043405+0000","last_fullsized":"2024-03-22T13:09:38.043405+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:45.197847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.474702+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T13:08:59.474702+0000","last_peered":"2024-03-22T13:08:59.474702+0000","last_clean":"2024-03-22T13:08:59.474702+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T13:08:59.474702+0000","last_undegraded":"2024-03-22T13:08:59.474702+0000","last_fullsized":"2024-03-22T13:08:59.474702+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:23.839356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452447+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T13:08:59.452447+0000","last_peered":"2024-03-22T13:08:59.452447+0000","last_clean":"2024-03-22T13:08:59.452447+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T13:08:59.452447+0000","last_undegraded":"2024-03-22T13:08:59.452447+0000","last_fullsized":"2024-03-22T13:08:59.452447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.540600+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T13:09:09.540600+0000","last_peered":"2024-03-22T13:09:09.540600+0000","last_clean":"2024-03-22T13:09:09.540600+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T13:09:09.540600+0000","last_undegraded":"2024-03-22T13:09:09.540600+0000","last_fullsized":"2024-03-22T13:09:09.540600+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.534260+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T13:08:29.534260+0000","last_peered":"2024-03-22T13:08:29.534260+0000","last_clean":"2024-03-22T13:08:29.534260+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T13:08:29.534260+0000","last_undegraded":"2024-03-22T13:08:29.534260+0000","last_fullsized":"2024-03-22T13:08:29.534260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.037760+0000","last_change":"2024-03-22T12:56:28.763384+0000","last_active":"2024-03-22T13:08:58.037760+0000","last_peered":"2024-03-22T13:08:58.037760+0000","last_clean":"2024-03-22T13:08:58.037760+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T13:08:58.037760+0000","last_undegraded":"2024-03-22T13:08:58.037760+0000","last_fullsized":"2024-03-22T13:08:58.037760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:15:45.463056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:58.029027+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T13:07:58.029027+0000","last_peered":"2024-03-22T13:07:58.029027+0000","last_clean":"2024-03-22T13:07:58.029027+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T13:07:58.029027+0000","last_undegraded":"2024-03-22T13:07:58.029027+0000","last_fullsized":"2024-03-22T13:07:58.029027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:48.651995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431440+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T13:08:59.431440+0000","last_peered":"2024-03-22T13:08:59.431440+0000","last_clean":"2024-03-22T13:08:59.431440+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T13:08:59.431440+0000","last_undegraded":"2024-03-22T13:08:59.431440+0000","last_fullsized":"2024-03-22T13:08:59.431440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:36:50.048241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:28.090677+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T13:08:28.090677+0000","last_peered":"2024-03-22T13:08:28.090677+0000","last_clean":"2024-03-22T13:08:28.090677+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T13:08:28.090677+0000","last_undegraded":"2024-03-22T13:08:28.090677+0000","last_fullsized":"2024-03-22T13:08:28.090677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.437421+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T13:09:39.437421+0000","last_peered":"2024-03-22T13:09:39.437421+0000","last_clean":"2024-03-22T13:09:39.437421+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T13:09:39.437421+0000","last_undegraded":"2024-03-22T13:09:39.437421+0000","last_fullsized":"2024-03-22T13:09:39.437421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:29:31.400078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360426+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T13:08:59.360426+0000","last_peered":"2024-03-22T13:08:59.360426+0000","last_clean":"2024-03-22T13:08:59.360426+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T13:08:59.360426+0000","last_undegraded":"2024-03-22T13:08:59.360426+0000","last_fullsized":"2024-03-22T13:08:59.360426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452461+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T13:08:59.452461+0000","last_peered":"2024-03-22T13:08:59.452461+0000","last_clean":"2024-03-22T13:08:59.452461+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T13:08:59.452461+0000","last_undegraded":"2024-03-22T13:08:59.452461+0000","last_fullsized":"2024-03-22T13:08:59.452461+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":194,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:18.089652+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T13:08:18.089652+0000","last_peered":"2024-03-22T13:08:18.089652+0000","last_clean":"2024-03-22T13:08:18.089652+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T13:08:18.089652+0000","last_undegraded":"2024-03-22T13:08:18.089652+0000","last_fullsized":"2024-03-22T13:08:18.089652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047239+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T13:09:08.047239+0000","last_peered":"2024-03-22T13:09:08.047239+0000","last_clean":"2024-03-22T13:09:08.047239+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T13:09:08.047239+0000","last_undegraded":"2024-03-22T13:09:08.047239+0000","last_fullsized":"2024-03-22T13:09:08.047239+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086769+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T13:07:48.086769+0000","last_peered":"2024-03-22T13:07:48.086769+0000","last_clean":"2024-03-22T13:07:48.086769+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T13:07:48.086769+0000","last_undegraded":"2024-03-22T13:07:48.086769+0000","last_fullsized":"2024-03-22T13:07:48.086769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"31'6","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.026555+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T13:07:43.026555+0000","last_peered":"2024-03-22T13:07:43.026555+0000","last_clean":"2024-03-22T13:07:43.026555+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T13:07:43.026555+0000","last_undegraded":"2024-03-22T13:07:43.026555+0000","last_fullsized":"2024-03-22T13:07:43.026555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:38.050005+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T13:09:38.050005+0000","last_peered":"2024-03-22T13:09:38.050005+0000","last_clean":"2024-03-22T13:09:38.050005+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T13:09:38.050005+0000","last_undegraded":"2024-03-22T13:09:38.050005+0000","last_fullsized":"2024-03-22T13:09:38.050005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.445504+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T13:07:44.445504+0000","last_peered":"2024-03-22T13:07:44.445504+0000","last_clean":"2024-03-22T13:07:44.445504+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T13:07:44.445504+0000","last_undegraded":"2024-03-22T13:07:44.445504+0000","last_fullsized":"2024-03-22T13:07:44.445504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"31'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.528186+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T13:07:44.528186+0000","last_peered":"2024-03-22T13:07:44.528186+0000","last_clean":"2024-03-22T13:07:44.528186+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T13:07:44.528186+0000","last_undegraded":"2024-03-22T13:07:44.528186+0000","last_fullsized":"2024-03-22T13:07:44.528186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:22.329365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.421240+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T13:07:54.421240+0000","last_peered":"2024-03-22T13:07:54.421240+0000","last_clean":"2024-03-22T13:07:54.421240+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T13:07:54.421240+0000","last_undegraded":"2024-03-22T13:07:54.421240+0000","last_fullsized":"2024-03-22T13:07:54.421240+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086812+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T13:07:48.086812+0000","last_peered":"2024-03-22T13:07:48.086812+0000","last_clean":"2024-03-22T13:07:48.086812+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T13:07:48.086812+0000","last_undegraded":"2024-03-22T13:07:48.086812+0000","last_fullsized":"2024-03-22T13:07:48.086812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"31'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.026598+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T13:07:43.026598+0000","last_peered":"2024-03-22T13:07:43.026598+0000","last_clean":"2024-03-22T13:07:43.026598+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T13:07:43.026598+0000","last_undegraded":"2024-03-22T13:07:43.026598+0000","last_fullsized":"2024-03-22T13:07:43.026598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.456333+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T13:09:39.456333+0000","last_peered":"2024-03-22T13:09:39.456333+0000","last_clean":"2024-03-22T13:09:39.456333+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T13:09:39.456333+0000","last_undegraded":"2024-03-22T13:09:39.456333+0000","last_fullsized":"2024-03-22T13:09:39.456333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086795+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T13:07:48.086795+0000","last_peered":"2024-03-22T13:07:48.086795+0000","last_clean":"2024-03-22T13:07:48.086795+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T13:07:48.086795+0000","last_undegraded":"2024-03-22T13:07:48.086795+0000","last_fullsized":"2024-03-22T13:07:48.086795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"31'2","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:15.533469+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T13:08:15.533469+0000","last_peered":"2024-03-22T13:08:15.533469+0000","last_clean":"2024-03-22T13:08:15.533469+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T13:08:15.533469+0000","last_undegraded":"2024-03-22T13:08:15.533469+0000","last_fullsized":"2024-03-22T13:08:15.533469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:24:46.021828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.478501+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T13:09:39.478501+0000","last_peered":"2024-03-22T13:09:39.478501+0000","last_clean":"2024-03-22T13:09:39.478501+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T13:09:39.478501+0000","last_undegraded":"2024-03-22T13:09:39.478501+0000","last_fullsized":"2024-03-22T13:09:39.478501+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:38.316319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089125+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T13:08:13.089125+0000","last_peered":"2024-03-22T13:08:13.089125+0000","last_clean":"2024-03-22T13:08:13.089125+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T13:08:13.089125+0000","last_undegraded":"2024-03-22T13:08:13.089125+0000","last_fullsized":"2024-03-22T13:08:13.089125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"31'18","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.030746+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T13:08:08.030746+0000","last_peered":"2024-03-22T13:08:08.030746+0000","last_clean":"2024-03-22T13:08:08.030746+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T13:08:08.030746+0000","last_undegraded":"2024-03-22T13:08:08.030746+0000","last_fullsized":"2024-03-22T13:08:08.030746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:20:57.722062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.429460+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T13:08:44.429460+0000","last_peered":"2024-03-22T13:08:44.429460+0000","last_clean":"2024-03-22T13:08:44.429460+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T13:08:44.429460+0000","last_undegraded":"2024-03-22T13:08:44.429460+0000","last_fullsized":"2024-03-22T13:08:44.429460+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.042504+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T13:08:13.042504+0000","last_peered":"2024-03-22T13:08:13.042504+0000","last_clean":"2024-03-22T13:08:13.042504+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T13:08:13.042504+0000","last_undegraded":"2024-03-22T13:08:13.042504+0000","last_fullsized":"2024-03-22T13:08:13.042504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.041967+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T13:08:08.041967+0000","last_peered":"2024-03-22T13:08:08.041967+0000","last_clean":"2024-03-22T13:08:08.041967+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T13:08:08.041967+0000","last_undegraded":"2024-03-22T13:08:08.041967+0000","last_fullsized":"2024-03-22T13:08:08.041967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.358426+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T13:08:44.358426+0000","last_peered":"2024-03-22T13:08:44.358426+0000","last_clean":"2024-03-22T13:08:44.358426+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T13:08:44.358426+0000","last_undegraded":"2024-03-22T13:08:44.358426+0000","last_fullsized":"2024-03-22T13:08:44.358426+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:51.788457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089153+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T13:08:13.089153+0000","last_peered":"2024-03-22T13:08:13.089153+0000","last_clean":"2024-03-22T13:08:13.089153+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T13:08:13.089153+0000","last_undegraded":"2024-03-22T13:08:13.089153+0000","last_fullsized":"2024-03-22T13:08:13.089153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447799+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T13:08:09.447799+0000","last_peered":"2024-03-22T13:08:09.447799+0000","last_clean":"2024-03-22T13:08:09.447799+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T13:08:09.447799+0000","last_undegraded":"2024-03-22T13:08:09.447799+0000","last_fullsized":"2024-03-22T13:08:09.447799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354809+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T13:08:19.354809+0000","last_peered":"2024-03-22T13:08:19.354809+0000","last_clean":"2024-03-22T13:08:19.354809+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T13:08:19.354809+0000","last_undegraded":"2024-03-22T13:08:19.354809+0000","last_fullsized":"2024-03-22T13:08:19.354809+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:08:12.860933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423855+0000","last_change":"2024-03-22T12:56:30.299969+0000","last_active":"2024-03-22T13:08:09.423855+0000","last_peered":"2024-03-22T13:08:09.423855+0000","last_clean":"2024-03-22T13:08:09.423855+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T13:08:09.423855+0000","last_undegraded":"2024-03-22T13:08:09.423855+0000","last_fullsized":"2024-03-22T13:08:09.423855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'9","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423914+0000","last_change":"2024-03-22T12:56:28.777142+0000","last_active":"2024-03-22T13:08:09.423914+0000","last_peered":"2024-03-22T13:08:09.423914+0000","last_clean":"2024-03-22T13:08:09.423914+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T13:08:09.423914+0000","last_undegraded":"2024-03-22T13:08:09.423914+0000","last_fullsized":"2024-03-22T13:08:09.423914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:49.411808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.451139+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T13:08:44.451139+0000","last_peered":"2024-03-22T13:08:44.451139+0000","last_clean":"2024-03-22T13:08:44.451139+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T13:08:44.451139+0000","last_undegraded":"2024-03-22T13:08:44.451139+0000","last_fullsized":"2024-03-22T13:08:44.451139+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091724+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T13:08:38.091724+0000","last_peered":"2024-03-22T13:08:38.091724+0000","last_clean":"2024-03-22T13:08:38.091724+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T13:08:38.091724+0000","last_undegraded":"2024-03-22T13:08:38.091724+0000","last_fullsized":"2024-03-22T13:08:38.091724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'14","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.042011+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T13:08:08.042011+0000","last_peered":"2024-03-22T13:08:08.042011+0000","last_clean":"2024-03-22T13:08:08.042011+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T13:08:08.042011+0000","last_undegraded":"2024-03-22T13:08:08.042011+0000","last_fullsized":"2024-03-22T13:08:08.042011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533275+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T13:08:19.533275+0000","last_peered":"2024-03-22T13:08:19.533275+0000","last_clean":"2024-03-22T13:08:19.533275+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T13:08:19.533275+0000","last_undegraded":"2024-03-22T13:08:19.533275+0000","last_fullsized":"2024-03-22T13:08:19.533275+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.470116+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T13:08:09.470116+0000","last_peered":"2024-03-22T13:08:09.470116+0000","last_clean":"2024-03-22T13:08:09.470116+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T13:08:09.470116+0000","last_undegraded":"2024-03-22T13:08:09.470116+0000","last_fullsized":"2024-03-22T13:08:09.470116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:49.776407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.535102+0000","last_change":"2024-03-22T12:56:28.769041+0000","last_active":"2024-03-22T13:08:34.535102+0000","last_peered":"2024-03-22T13:08:34.535102+0000","last_clean":"2024-03-22T13:08:34.535102+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T13:08:34.535102+0000","last_undegraded":"2024-03-22T13:08:34.535102+0000","last_fullsized":"2024-03-22T13:08:34.535102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.471072+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T13:08:19.471072+0000","last_peered":"2024-03-22T13:08:19.471072+0000","last_clean":"2024-03-22T13:08:19.471072+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T13:08:19.471072+0000","last_undegraded":"2024-03-22T13:08:19.471072+0000","last_fullsized":"2024-03-22T13:08:19.471072+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:56.815720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.428126+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T13:08:34.428126+0000","last_peered":"2024-03-22T13:08:34.428126+0000","last_clean":"2024-03-22T13:08:34.428126+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T13:08:34.428126+0000","last_undegraded":"2024-03-22T13:08:34.428126+0000","last_fullsized":"2024-03-22T13:08:34.428126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450282+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T13:08:34.450282+0000","last_peered":"2024-03-22T13:08:34.450282+0000","last_clean":"2024-03-22T13:08:34.450282+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T13:08:34.450282+0000","last_undegraded":"2024-03-22T13:08:34.450282+0000","last_fullsized":"2024-03-22T13:08:34.450282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354747+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T13:08:19.354747+0000","last_peered":"2024-03-22T13:08:19.354747+0000","last_clean":"2024-03-22T13:08:19.354747+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T13:08:19.354747+0000","last_undegraded":"2024-03-22T13:08:19.354747+0000","last_fullsized":"2024-03-22T13:08:19.354747+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:06:47.716384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450270+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T13:08:34.450270+0000","last_peered":"2024-03-22T13:08:34.450270+0000","last_clean":"2024-03-22T13:08:34.450270+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T13:08:34.450270+0000","last_undegraded":"2024-03-22T13:08:34.450270+0000","last_fullsized":"2024-03-22T13:08:34.450270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:33.044141+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T13:08:33.044141+0000","last_peered":"2024-03-22T13:08:33.044141+0000","last_clean":"2024-03-22T13:08:33.044141+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T13:08:33.044141+0000","last_undegraded":"2024-03-22T13:08:33.044141+0000","last_fullsized":"2024-03-22T13:08:33.044141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354781+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T13:08:19.354781+0000","last_peered":"2024-03-22T13:08:19.354781+0000","last_clean":"2024-03-22T13:08:19.354781+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T13:08:19.354781+0000","last_undegraded":"2024-03-22T13:08:19.354781+0000","last_fullsized":"2024-03-22T13:08:19.354781+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:39.428904+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T13:08:39.428904+0000","last_peered":"2024-03-22T13:08:39.428904+0000","last_clean":"2024-03-22T13:08:39.428904+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T13:08:39.428904+0000","last_undegraded":"2024-03-22T13:08:39.428904+0000","last_fullsized":"2024-03-22T13:08:39.428904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.534984+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T13:08:34.534984+0000","last_peered":"2024-03-22T13:08:34.534984+0000","last_clean":"2024-03-22T13:08:34.534984+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T13:08:34.534984+0000","last_undegraded":"2024-03-22T13:08:34.534984+0000","last_fullsized":"2024-03-22T13:08:34.534984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450228+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T13:08:34.450228+0000","last_peered":"2024-03-22T13:08:34.450228+0000","last_clean":"2024-03-22T13:08:34.450228+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T13:08:34.450228+0000","last_undegraded":"2024-03-22T13:08:34.450228+0000","last_fullsized":"2024-03-22T13:08:34.450228+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047276+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T13:09:08.047276+0000","last_peered":"2024-03-22T13:09:08.047276+0000","last_clean":"2024-03-22T13:09:08.047276+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T13:09:08.047276+0000","last_undegraded":"2024-03-22T13:09:08.047276+0000","last_fullsized":"2024-03-22T13:09:08.047276+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356605+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T13:08:34.356605+0000","last_peered":"2024-03-22T13:08:34.356605+0000","last_clean":"2024-03-22T13:08:34.356605+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T13:08:34.356605+0000","last_undegraded":"2024-03-22T13:08:34.356605+0000","last_fullsized":"2024-03-22T13:08:34.356605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356655+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T13:08:34.356655+0000","last_peered":"2024-03-22T13:08:34.356655+0000","last_clean":"2024-03-22T13:08:34.356655+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T13:08:34.356655+0000","last_undegraded":"2024-03-22T13:08:34.356655+0000","last_fullsized":"2024-03-22T13:08:34.356655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:31:41.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.046243+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T13:08:58.046243+0000","last_peered":"2024-03-22T13:08:58.046243+0000","last_clean":"2024-03-22T13:08:58.046243+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T13:08:58.046243+0000","last_undegraded":"2024-03-22T13:08:58.046243+0000","last_fullsized":"2024-03-22T13:08:58.046243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:08.021699+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T13:07:08.021699+0000","last_peered":"2024-03-22T13:07:08.021699+0000","last_clean":"2024-03-22T13:07:08.021699+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T13:07:08.021699+0000","last_undegraded":"2024-03-22T13:07:08.021699+0000","last_fullsized":"2024-03-22T13:07:08.021699+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.039286+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T13:09:08.039286+0000","last_peered":"2024-03-22T13:09:08.039286+0000","last_clean":"2024-03-22T13:09:08.039286+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T13:09:08.039286+0000","last_undegraded":"2024-03-22T13:09:08.039286+0000","last_fullsized":"2024-03-22T13:09:08.039286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:43.256977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360446+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T13:08:59.360446+0000","last_peered":"2024-03-22T13:08:59.360446+0000","last_clean":"2024-03-22T13:08:59.360446+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T13:08:59.360446+0000","last_undegraded":"2024-03-22T13:08:59.360446+0000","last_fullsized":"2024-03-22T13:08:59.360446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.474675+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T13:08:59.474675+0000","last_peered":"2024-03-22T13:08:59.474675+0000","last_clean":"2024-03-22T13:08:59.474675+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T13:08:59.474675+0000","last_undegraded":"2024-03-22T13:08:59.474675+0000","last_fullsized":"2024-03-22T13:08:59.474675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:48.882251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.361490+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T13:09:09.361490+0000","last_peered":"2024-03-22T13:09:09.361490+0000","last_clean":"2024-03-22T13:09:09.361490+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T13:09:09.361490+0000","last_undegraded":"2024-03-22T13:09:09.361490+0000","last_fullsized":"2024-03-22T13:09:09.361490+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.046714+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T13:09:03.046714+0000","last_peered":"2024-03-22T13:09:03.046714+0000","last_clean":"2024-03-22T13:09:03.046714+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T13:09:03.046714+0000","last_undegraded":"2024-03-22T13:09:03.046714+0000","last_fullsized":"2024-03-22T13:09:03.046714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.038633+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T13:09:03.038633+0000","last_peered":"2024-03-22T13:09:03.038633+0000","last_clean":"2024-03-22T13:09:03.038633+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T13:09:03.038633+0000","last_undegraded":"2024-03-22T13:09:03.038633+0000","last_fullsized":"2024-03-22T13:09:03.038633+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452433+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T13:08:59.452433+0000","last_peered":"2024-03-22T13:08:59.452433+0000","last_clean":"2024-03-22T13:08:59.452433+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T13:08:59.452433+0000","last_undegraded":"2024-03-22T13:08:59.452433+0000","last_fullsized":"2024-03-22T13:08:59.452433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431501+0000","last_change":"2024-03-22T12:56:30.300244+0000","last_active":"2024-03-22T13:08:59.431501+0000","last_peered":"2024-03-22T13:08:59.431501+0000","last_clean":"2024-03-22T13:08:59.431501+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T13:08:59.431501+0000","last_undegraded":"2024-03-22T13:08:59.431501+0000","last_fullsized":"2024-03-22T13:08:59.431501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363660+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T13:09:29.363660+0000","last_peered":"2024-03-22T13:09:29.363660+0000","last_clean":"2024-03-22T13:09:29.363660+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T13:09:29.363660+0000","last_undegraded":"2024-03-22T13:09:29.363660+0000","last_fullsized":"2024-03-22T13:09:29.363660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542819+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T13:09:29.542819+0000","last_peered":"2024-03-22T13:09:29.542819+0000","last_clean":"2024-03-22T13:09:29.542819+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T13:09:29.542819+0000","last_undegraded":"2024-03-22T13:09:29.542819+0000","last_fullsized":"2024-03-22T13:09:29.542819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049650+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T13:09:33.049650+0000","last_peered":"2024-03-22T13:09:33.049650+0000","last_clean":"2024-03-22T13:09:33.049650+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T13:09:33.049650+0000","last_undegraded":"2024-03-22T13:09:33.049650+0000","last_fullsized":"2024-03-22T13:09:33.049650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363696+0000","last_change":"2024-03-22T12:56:30.308903+0000","last_active":"2024-03-22T13:09:29.363696+0000","last_peered":"2024-03-22T13:09:29.363696+0000","last_clean":"2024-03-22T13:09:29.363696+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T13:09:29.363696+0000","last_undegraded":"2024-03-22T13:09:29.363696+0000","last_fullsized":"2024-03-22T13:09:29.363696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:58.028924+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T13:07:58.028924+0000","last_peered":"2024-03-22T13:07:58.028924+0000","last_clean":"2024-03-22T13:07:58.028924+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T13:07:58.028924+0000","last_undegraded":"2024-03-22T13:07:58.028924+0000","last_fullsized":"2024-03-22T13:07:58.028924+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:13:57.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533242+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T13:08:19.533242+0000","last_peered":"2024-03-22T13:08:19.533242+0000","last_clean":"2024-03-22T13:08:19.533242+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T13:08:19.533242+0000","last_undegraded":"2024-03-22T13:08:19.533242+0000","last_fullsized":"2024-03-22T13:08:19.533242+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:19.797978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.445455+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T13:07:44.445455+0000","last_peered":"2024-03-22T13:07:44.445455+0000","last_clean":"2024-03-22T13:07:44.445455+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T13:07:44.445455+0000","last_undegraded":"2024-03-22T13:07:44.445455+0000","last_fullsized":"2024-03-22T13:07:44.445455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.419780+0000","last_change":"2024-03-22T12:56:30.299214+0000","last_active":"2024-03-22T13:07:44.419780+0000","last_peered":"2024-03-22T13:07:44.419780+0000","last_clean":"2024-03-22T13:07:44.419780+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T13:07:44.419780+0000","last_undegraded":"2024-03-22T13:07:44.419780+0000","last_fullsized":"2024-03-22T13:07:44.419780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:35.356723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.040052+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T13:07:48.040052+0000","last_peered":"2024-03-22T13:07:48.040052+0000","last_clean":"2024-03-22T13:07:48.040052+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T13:07:48.040052+0000","last_undegraded":"2024-03-22T13:07:48.040052+0000","last_fullsized":"2024-03-22T13:07:48.040052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533301+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T13:08:19.533301+0000","last_peered":"2024-03-22T13:08:19.533301+0000","last_clean":"2024-03-22T13:08:19.533301+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T13:08:19.533301+0000","last_undegraded":"2024-03-22T13:08:19.533301+0000","last_fullsized":"2024-03-22T13:08:19.533301+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:57:33.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.031478+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T13:08:13.031478+0000","last_peered":"2024-03-22T13:08:13.031478+0000","last_clean":"2024-03-22T13:08:13.031478+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T13:08:13.031478+0000","last_undegraded":"2024-03-22T13:08:13.031478+0000","last_fullsized":"2024-03-22T13:08:13.031478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:38.987151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447831+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T13:08:09.447831+0000","last_peered":"2024-03-22T13:08:09.447831+0000","last_clean":"2024-03-22T13:08:09.447831+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T13:08:09.447831+0000","last_undegraded":"2024-03-22T13:08:09.447831+0000","last_fullsized":"2024-03-22T13:08:09.447831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447764+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T13:08:09.447764+0000","last_peered":"2024-03-22T13:08:09.447764+0000","last_clean":"2024-03-22T13:08:09.447764+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T13:08:09.447764+0000","last_undegraded":"2024-03-22T13:08:09.447764+0000","last_fullsized":"2024-03-22T13:08:09.447764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.532025+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T13:08:09.532025+0000","last_peered":"2024-03-22T13:08:09.532025+0000","last_clean":"2024-03-22T13:08:09.532025+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T13:08:09.532025+0000","last_undegraded":"2024-03-22T13:08:09.532025+0000","last_fullsized":"2024-03-22T13:08:09.532025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423817+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T13:08:09.423817+0000","last_peered":"2024-03-22T13:08:09.423817+0000","last_clean":"2024-03-22T13:08:09.423817+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T13:08:09.423817+0000","last_undegraded":"2024-03-22T13:08:09.423817+0000","last_fullsized":"2024-03-22T13:08:09.423817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356700+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T13:08:34.356700+0000","last_peered":"2024-03-22T13:08:34.356700+0000","last_clean":"2024-03-22T13:08:34.356700+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T13:08:34.356700+0000","last_undegraded":"2024-03-22T13:08:34.356700+0000","last_fullsized":"2024-03-22T13:08:34.356700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.472507+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T13:08:34.472507+0000","last_peered":"2024-03-22T13:08:34.472507+0000","last_clean":"2024-03-22T13:08:34.472507+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T13:08:34.472507+0000","last_undegraded":"2024-03-22T13:08:34.472507+0000","last_fullsized":"2024-03-22T13:08:34.472507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:23.598956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044564+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T13:08:38.044564+0000","last_peered":"2024-03-22T13:08:38.044564+0000","last_clean":"2024-03-22T13:08:38.044564+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T13:08:38.044564+0000","last_undegraded":"2024-03-22T13:08:38.044564+0000","last_fullsized":"2024-03-22T13:08:38.044564+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044582+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T13:08:38.044582+0000","last_peered":"2024-03-22T13:08:38.044582+0000","last_clean":"2024-03-22T13:08:38.044582+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T13:08:38.044582+0000","last_undegraded":"2024-03-22T13:08:38.044582+0000","last_fullsized":"2024-03-22T13:08:38.044582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091763+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T13:08:38.091763+0000","last_peered":"2024-03-22T13:08:38.091763+0000","last_clean":"2024-03-22T13:08:38.091763+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T13:08:38.091763+0000","last_undegraded":"2024-03-22T13:08:38.091763+0000","last_fullsized":"2024-03-22T13:08:38.091763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360387+0000","last_change":"2024-03-22T12:56:30.309258+0000","last_active":"2024-03-22T13:08:59.360387+0000","last_peered":"2024-03-22T13:08:59.360387+0000","last_clean":"2024-03-22T13:08:59.360387+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T13:08:59.360387+0000","last_undegraded":"2024-03-22T13:08:59.360387+0000","last_fullsized":"2024-03-22T13:08:59.360387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094308+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T13:09:03.094308+0000","last_peered":"2024-03-22T13:09:03.094308+0000","last_clean":"2024-03-22T13:09:03.094308+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T13:09:03.094308+0000","last_undegraded":"2024-03-22T13:09:03.094308+0000","last_fullsized":"2024-03-22T13:09:03.094308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452409+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T13:08:59.452409+0000","last_peered":"2024-03-22T13:08:59.452409+0000","last_clean":"2024-03-22T13:08:59.452409+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T13:08:59.452409+0000","last_undegraded":"2024-03-22T13:08:59.452409+0000","last_fullsized":"2024-03-22T13:08:59.452409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094342+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T13:09:03.094342+0000","last_peered":"2024-03-22T13:09:03.094342+0000","last_clean":"2024-03-22T13:09:03.094342+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T13:09:03.094342+0000","last_undegraded":"2024-03-22T13:09:03.094342+0000","last_fullsized":"2024-03-22T13:09:03.094342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.449798+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T13:08:29.449798+0000","last_peered":"2024-03-22T13:08:29.449798+0000","last_clean":"2024-03-22T13:08:29.449798+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T13:08:29.449798+0000","last_undegraded":"2024-03-22T13:08:29.449798+0000","last_fullsized":"2024-03-22T13:08:29.449798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431526+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T13:08:59.431526+0000","last_peered":"2024-03-22T13:08:59.431526+0000","last_clean":"2024-03-22T13:08:59.431526+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T13:08:59.431526+0000","last_undegraded":"2024-03-22T13:08:59.431526+0000","last_fullsized":"2024-03-22T13:08:59.431526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:14.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539145+0000","last_change":"2024-03-22T12:56:30.303254+0000","last_active":"2024-03-22T13:08:59.539145+0000","last_peered":"2024-03-22T13:08:59.539145+0000","last_clean":"2024-03-22T13:08:59.539145+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T13:08:59.539145+0000","last_undegraded":"2024-03-22T13:08:59.539145+0000","last_fullsized":"2024-03-22T13:08:59.539145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094374+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T13:09:03.094374+0000","last_peered":"2024-03-22T13:09:03.094374+0000","last_clean":"2024-03-22T13:09:03.094374+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T13:09:03.094374+0000","last_undegraded":"2024-03-22T13:09:03.094374+0000","last_fullsized":"2024-03-22T13:09:03.094374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509633,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":142876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509633,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":140274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509633,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":143610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":140476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509632,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":730186,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542339,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":731412,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":143576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542338,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T13:12:05.315 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-22T13:12:05.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T13:12:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.546+0000 7f3223200640 1 Processor -- start 2024-03-22T13:12:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.546+0000 7f3223200640 1 -- start start 2024-03-22T13:12:05.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.547+0000 7f3223200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f321c0c7ee0 0x7f321c15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:05.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.547+0000 7f3223200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c06b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:05.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.547+0000 7f3223200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c0c4740 con 0x7f321c154240 2024-03-22T13:12:05.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.547+0000 7f3223200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321c15e130 con 0x7f321c0c7ee0 2024-03-22T13:12:05.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.547+0000 7f3223200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c154620 con 0x7f321c0c68c0 2024-03-22T13:12:05.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.548+0000 7f3221200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c06b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:05.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.548+0000 7f321b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f321c0c7ee0 0x7f321c15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:05.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.548+0000 7f3221200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c06b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54248/0 (socket says 172.21.15.6:54248) 2024-03-22T13:12:05.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.548+0000 7f3221200640 1 -- 172.21.15.6:0/75901285 learned_addr learned my addr 172.21.15.6:0/75901285 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.550+0000 7f3221200640 1 -- 172.21.15.6:0/75901285 >> v2:172.21.15.6:3300/0 conn(0x7f321c0c7ee0 msgr2=0x7f321c15ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.550+0000 7f3221200640 1 --2- 172.21.15.6:0/75901285 >> v2:172.21.15.6:3300/0 conn(0x7f321c0c7ee0 0x7f321c15ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.550+0000 7f3221200640 1 -- 172.21.15.6:0/75901285 >> v1:172.21.15.6:6789/0 conn(0x7f321c154240 legacy=0x7f321c0c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.550+0000 7f3221200640 1 -- 172.21.15.6:0/75901285 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c0c6e00 con 0x7f321c0c68c0 2024-03-22T13:12:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.552+0000 7f3221200640 1 --2- 172.21.15.6:0/75901285 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c06b080 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3214009870 tx=0x7f321402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f4754906ae5bb30 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.555+0000 7f321aa00640 1 -- 172.21.15.6:0/75901285 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321402f840 con 0x7f321c0c68c0 2024-03-22T13:12:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.557+0000 7f321aa00640 1 -- 172.21.15.6:0/75901285 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321402f9e0 con 0x7f321c0c68c0 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f321aa00640 1 -- 172.21.15.6:0/75901285 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321402fdb0 con 0x7f321c0c68c0 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 -- 172.21.15.6:0/75901285 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 msgr2=0x7f321c06b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 --2- 172.21.15.6:0/75901285 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c06b080 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3214009870 tx=0x7f321402ba20 comp rx=0 tx=0).stop 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 -- 172.21.15.6:0/75901285 shutdown_connections 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 --2- 172.21.15.6:0/75901285 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c06b080 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 --2- 172.21.15.6:0/75901285 >> v2:172.21.15.6:3300/0 conn(0x7f321c0c7ee0 0x7f321c15ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 -- 172.21.15.6:0/75901285 >> 172.21.15.6:0/75901285 conn(0x7f321c0cb930 msgr2=0x7f321c0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 -- 172.21.15.6:0/75901285 shutdown_connections 2024-03-22T13:12:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.558+0000 7f3223200640 1 -- 172.21.15.6:0/75901285 wait complete. 2024-03-22T13:12:05.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.561+0000 7f3223200640 1 Processor -- start 2024-03-22T13:12:05.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.561+0000 7f3223200640 1 -- start start 2024-03-22T13:12:05.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.562+0000 7f3223200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:05.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.562+0000 7f3223200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f321c0c7ee0 0x7f321c15a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:05.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.562+0000 7f3223200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f321c154240 0x7f321c150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:05.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.562+0000 7f3223200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c15dae0 con 0x7f321c0c7ee0 2024-03-22T13:12:05.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.562+0000 7f3223200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c15d800 con 0x7f321c154240 2024-03-22T13:12:05.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.562+0000 7f3223200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c15ddc0 con 0x7f321c0c68c0 2024-03-22T13:12:05.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.563+0000 7f321be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:05.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.563+0000 7f321b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f321c0c7ee0 0x7f321c15a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.563+0000 7f321be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54260/0 (socket says 172.21.15.6:54260) 2024-03-22T13:12:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.563+0000 7f321be00640 1 -- 172.21.15.6:0/229768178 learned_addr learned my addr 172.21.15.6:0/229768178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.563+0000 7f3221200640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f321c154240 0x7f321c150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.567+0000 7f321b400640 1 -- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 msgr2=0x7f321c153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.567+0000 7f321b400640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.567+0000 7f321b400640 1 -- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f321c154240 msgr2=0x7f321c150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.567+0000 7f321b400640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f321c154240 0x7f321c150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.567+0000 7f321b400640 1 -- 172.21.15.6:0/229768178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c15ae30 con 0x7f321c0c7ee0 2024-03-22T13:12:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.570+0000 7f321b400640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f321c0c7ee0 0x7f321c15a4f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f320400e820 tx=0x7f320400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.575+0000 7f3213e00640 1 -- 172.21.15.6:0/229768178 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204017070 con 0x7f321c0c7ee0 2024-03-22T13:12:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.575+0000 7f3213e00640 1 -- 172.21.15.6:0/229768178 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320400c940 con 0x7f321c0c7ee0 2024-03-22T13:12:05.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.575+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c1e3d20 con 0x7f321c0c7ee0 2024-03-22T13:12:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.575+0000 7f3213e00640 1 -- 172.21.15.6:0/229768178 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320400cd30 con 0x7f321c0c7ee0 2024-03-22T13:12:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.576+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c1e42e0 con 0x7f321c0c7ee0 2024-03-22T13:12:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.577+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c0756c0 con 0x7f321c0c7ee0 2024-03-22T13:12:05.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.609+0000 7f3213e00640 1 -- 172.21.15.6:0/229768178 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f320401a070 con 0x7f321c0c7ee0 2024-03-22T13:12:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.610+0000 7f3213e00640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31f806bc20 0x7f31f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.611+0000 7f321be00640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31f806bc20 0x7f31f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.612+0000 7f321be00640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31f806bc20 0x7f31f806e0e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f320c007fd0 tx=0x7f320c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.615+0000 7f3213e00640 1 -- 172.21.15.6:0/229768178 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3204057640 con 0x7f321c0c7ee0 2024-03-22T13:12:05.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:05.933+0000 7f3213e00640 1 -- 172.21.15.6:0/229768178 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f32040595a0 con 0x7f321c0c7ee0 2024-03-22T13:12:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.156+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f321c0c7ab0 con 0x7f31f806bc20 2024-03-22T13:12:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.173+0000 7f3213e00640 1 -- 172.21.15.6:0/229768178 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332703 (secure 0 0 0) 0x7f321c0c7ab0 con 0x7f31f806bc20 2024-03-22T13:12:06.177 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:12:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31f806bc20 msgr2=0x7f31f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31f806bc20 0x7f31f806e0e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f320c007fd0 tx=0x7f320c009040 comp rx=0 tx=0).stop 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f321c0c7ee0 msgr2=0x7f321c15a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f321c0c7ee0 0x7f321c15a4f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f320400e820 tx=0x7f320400ece0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 shutdown_connections 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f31f806bc20 0x7f31f806e0e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f321c154240 0x7f321c150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f321c0c7ee0 0x7f321c15a4f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 --2- 172.21.15.6:0/229768178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f321c0c68c0 0x7f321c153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.179+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 >> 172.21.15.6:0/229768178 conn(0x7f321c0cb930 msgr2=0x7f321c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:06.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.180+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 shutdown_connections 2024-03-22T13:12:06.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.180+0000 7f3223200640 1 -- 172.21.15.6:0/229768178 wait complete. 2024-03-22T13:12:06.197 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-03-22T13:12:05.912805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2976,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":3047424,"data_stored":2312816,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"2.1f","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431578+0000","last_change":"2024-03-22T12:56:28.778294+0000","last_active":"2024-03-22T13:08:59.431578+0000","last_peered":"2024-03-22T13:08:59.431578+0000","last_clean":"2024-03-22T13:08:59.431578+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T13:08:59.431578+0000","last_undegraded":"2024-03-22T13:08:59.431578+0000","last_fullsized":"2024-03-22T13:08:59.431578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.355851+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T13:08:29.355851+0000","last_peered":"2024-03-22T13:08:29.355851+0000","last_clean":"2024-03-22T13:08:29.355851+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T13:08:29.355851+0000","last_undegraded":"2024-03-22T13:08:29.355851+0000","last_fullsized":"2024-03-22T13:08:29.355851+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.358314+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T13:08:44.358314+0000","last_peered":"2024-03-22T13:08:44.358314+0000","last_clean":"2024-03-22T13:08:44.358314+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T13:08:44.358314+0000","last_undegraded":"2024-03-22T13:08:44.358314+0000","last_fullsized":"2024-03-22T13:08:44.358314+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:35:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"30'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.355891+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T13:08:29.355891+0000","last_peered":"2024-03-22T13:08:29.355891+0000","last_clean":"2024-03-22T13:08:29.355891+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T13:08:29.355891+0000","last_undegraded":"2024-03-22T13:08:29.355891+0000","last_fullsized":"2024-03-22T13:08:29.355891+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431556+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T13:08:59.431556+0000","last_peered":"2024-03-22T13:08:59.431556+0000","last_clean":"2024-03-22T13:08:59.431556+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T13:08:59.431556+0000","last_undegraded":"2024-03-22T13:08:59.431556+0000","last_fullsized":"2024-03-22T13:08:59.431556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.473319+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T13:08:44.473319+0000","last_peered":"2024-03-22T13:08:44.473319+0000","last_clean":"2024-03-22T13:08:44.473319+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T13:08:44.473319+0000","last_undegraded":"2024-03-22T13:08:44.473319+0000","last_fullsized":"2024-03-22T13:08:44.473319+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:40.477732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'12","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093797+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T13:08:58.093797+0000","last_peered":"2024-03-22T13:08:58.093797+0000","last_clean":"2024-03-22T13:08:58.093797+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T13:08:58.093797+0000","last_undegraded":"2024-03-22T13:08:58.093797+0000","last_fullsized":"2024-03-22T13:08:58.093797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452477+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T13:08:59.452477+0000","last_peered":"2024-03-22T13:08:59.452477+0000","last_clean":"2024-03-22T13:08:59.452477+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T13:08:59.452477+0000","last_undegraded":"2024-03-22T13:08:59.452477+0000","last_fullsized":"2024-03-22T13:08:59.452477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475708+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T13:09:09.475708+0000","last_peered":"2024-03-22T13:09:09.475708+0000","last_clean":"2024-03-22T13:09:09.475708+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T13:09:09.475708+0000","last_undegraded":"2024-03-22T13:09:09.475708+0000","last_fullsized":"2024-03-22T13:09:09.475708+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:29.078349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093761+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T13:08:58.093761+0000","last_peered":"2024-03-22T13:08:58.093761+0000","last_clean":"2024-03-22T13:08:58.093761+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T13:08:58.093761+0000","last_undegraded":"2024-03-22T13:08:58.093761+0000","last_fullsized":"2024-03-22T13:08:58.093761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452502+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T13:08:59.452502+0000","last_peered":"2024-03-22T13:08:59.452502+0000","last_clean":"2024-03-22T13:08:59.452502+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T13:08:59.452502+0000","last_undegraded":"2024-03-22T13:08:59.452502+0000","last_fullsized":"2024-03-22T13:08:59.452502+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475689+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T13:09:09.475689+0000","last_peered":"2024-03-22T13:09:09.475689+0000","last_clean":"2024-03-22T13:09:09.475689+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T13:09:09.475689+0000","last_undegraded":"2024-03-22T13:09:09.475689+0000","last_fullsized":"2024-03-22T13:09:09.475689+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049612+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T13:09:33.049612+0000","last_peered":"2024-03-22T13:09:33.049612+0000","last_clean":"2024-03-22T13:09:33.049612+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T13:09:33.049612+0000","last_undegraded":"2024-03-22T13:09:33.049612+0000","last_fullsized":"2024-03-22T13:09:33.049612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.086340+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T13:07:43.086340+0000","last_peered":"2024-03-22T13:07:43.086340+0000","last_clean":"2024-03-22T13:07:43.086340+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T13:07:43.086340+0000","last_undegraded":"2024-03-22T13:07:43.086340+0000","last_fullsized":"2024-03-22T13:07:43.086340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468904+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T13:07:54.468904+0000","last_peered":"2024-03-22T13:07:54.468904+0000","last_clean":"2024-03-22T13:07:54.468904+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T13:07:54.468904+0000","last_undegraded":"2024-03-22T13:07:54.468904+0000","last_fullsized":"2024-03-22T13:07:54.468904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:29.130909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.350707+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T13:07:44.350707+0000","last_peered":"2024-03-22T13:07:44.350707+0000","last_clean":"2024-03-22T13:07:44.350707+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T13:07:44.350707+0000","last_undegraded":"2024-03-22T13:07:44.350707+0000","last_fullsized":"2024-03-22T13:07:44.350707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"31'4","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542796+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T13:09:29.542796+0000","last_peered":"2024-03-22T13:09:29.542796+0000","last_clean":"2024-03-22T13:09:29.542796+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T13:09:29.542796+0000","last_undegraded":"2024-03-22T13:09:29.542796+0000","last_fullsized":"2024-03-22T13:09:29.542796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468869+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T13:07:54.468869+0000","last_peered":"2024-03-22T13:07:54.468869+0000","last_clean":"2024-03-22T13:07:54.468869+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T13:07:54.468869+0000","last_undegraded":"2024-03-22T13:07:54.468869+0000","last_fullsized":"2024-03-22T13:07:54.468869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:40.776689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363615+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T13:09:29.363615+0000","last_peered":"2024-03-22T13:09:29.363615+0000","last_clean":"2024-03-22T13:09:29.363615+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T13:09:29.363615+0000","last_undegraded":"2024-03-22T13:09:29.363615+0000","last_fullsized":"2024-03-22T13:09:29.363615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"30'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542761+0000","last_change":"2024-03-22T12:56:28.757253+0000","last_active":"2024-03-22T13:09:29.542761+0000","last_peered":"2024-03-22T13:09:29.542761+0000","last_clean":"2024-03-22T13:09:29.542761+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T13:09:29.542761+0000","last_undegraded":"2024-03-22T13:09:29.542761+0000","last_fullsized":"2024-03-22T13:09:29.542761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:48.506816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.529674+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T13:07:54.529674+0000","last_peered":"2024-03-22T13:07:54.529674+0000","last_clean":"2024-03-22T13:07:54.529674+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T13:07:54.529674+0000","last_undegraded":"2024-03-22T13:07:54.529674+0000","last_fullsized":"2024-03-22T13:07:54.529674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.477525+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T13:09:29.477525+0000","last_peered":"2024-03-22T13:09:29.477525+0000","last_clean":"2024-03-22T13:09:29.477525+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T13:09:29.477525+0000","last_undegraded":"2024-03-22T13:09:29.477525+0000","last_fullsized":"2024-03-22T13:09:29.477525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:34.508555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.096649+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T13:09:28.096649+0000","last_peered":"2024-03-22T13:09:28.096649+0000","last_clean":"2024-03-22T13:09:28.096649+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T13:09:28.096649+0000","last_undegraded":"2024-03-22T13:09:28.096649+0000","last_fullsized":"2024-03-22T13:09:28.096649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.468920+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T13:07:54.468920+0000","last_peered":"2024-03-22T13:07:54.468920+0000","last_clean":"2024-03-22T13:07:54.468920+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T13:07:54.468920+0000","last_undegraded":"2024-03-22T13:07:54.468920+0000","last_fullsized":"2024-03-22T13:07:54.468920+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:09.592208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.435947+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T13:09:29.435947+0000","last_peered":"2024-03-22T13:09:29.435947+0000","last_clean":"2024-03-22T13:09:29.435947+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T13:09:29.435947+0000","last_undegraded":"2024-03-22T13:09:29.435947+0000","last_fullsized":"2024-03-22T13:09:29.435947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'7","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.049231+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T13:09:28.049231+0000","last_peered":"2024-03-22T13:09:28.049231+0000","last_clean":"2024-03-22T13:09:28.049231+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T13:09:28.049231+0000","last_undegraded":"2024-03-22T13:09:28.049231+0000","last_fullsized":"2024-03-22T13:09:28.049231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.475657+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T13:09:09.475657+0000","last_peered":"2024-03-22T13:09:09.475657+0000","last_clean":"2024-03-22T13:09:09.475657+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T13:09:09.475657+0000","last_undegraded":"2024-03-22T13:09:09.475657+0000","last_fullsized":"2024-03-22T13:09:09.475657+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:13.307688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363715+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T13:09:29.363715+0000","last_peered":"2024-03-22T13:09:29.363715+0000","last_clean":"2024-03-22T13:09:29.363715+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T13:09:29.363715+0000","last_undegraded":"2024-03-22T13:09:29.363715+0000","last_fullsized":"2024-03-22T13:09:29.363715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:30:06.289170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"30'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542839+0000","last_change":"2024-03-22T12:56:28.767925+0000","last_active":"2024-03-22T13:09:29.542839+0000","last_peered":"2024-03-22T13:09:29.542839+0000","last_clean":"2024-03-22T13:09:29.542839+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T13:09:29.542839+0000","last_undegraded":"2024-03-22T13:09:29.542839+0000","last_fullsized":"2024-03-22T13:09:29.542839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:39.357442+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T13:08:39.357442+0000","last_peered":"2024-03-22T13:08:39.357442+0000","last_clean":"2024-03-22T13:08:39.357442+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T13:08:39.357442+0000","last_undegraded":"2024-03-22T13:08:39.357442+0000","last_fullsized":"2024-03-22T13:08:39.357442+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:40:09.217719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539070+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T13:08:59.539070+0000","last_peered":"2024-03-22T13:08:59.539070+0000","last_clean":"2024-03-22T13:08:59.539070+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T13:08:59.539070+0000","last_undegraded":"2024-03-22T13:08:59.539070+0000","last_fullsized":"2024-03-22T13:08:59.539070+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093813+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T13:08:58.093813+0000","last_peered":"2024-03-22T13:08:58.093813+0000","last_clean":"2024-03-22T13:08:58.093813+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T13:08:58.093813+0000","last_undegraded":"2024-03-22T13:08:58.093813+0000","last_fullsized":"2024-03-22T13:08:58.093813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.433290+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T13:09:09.433290+0000","last_peered":"2024-03-22T13:09:09.433290+0000","last_clean":"2024-03-22T13:09:09.433290+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T13:09:09.433290+0000","last_undegraded":"2024-03-22T13:09:09.433290+0000","last_fullsized":"2024-03-22T13:09:09.433290+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:28.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452512+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T13:08:59.452512+0000","last_peered":"2024-03-22T13:08:59.452512+0000","last_clean":"2024-03-22T13:08:59.452512+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T13:08:59.452512+0000","last_undegraded":"2024-03-22T13:08:59.452512+0000","last_fullsized":"2024-03-22T13:08:59.452512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539108+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T13:08:59.539108+0000","last_peered":"2024-03-22T13:08:59.539108+0000","last_clean":"2024-03-22T13:08:59.539108+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T13:08:59.539108+0000","last_undegraded":"2024-03-22T13:08:59.539108+0000","last_fullsized":"2024-03-22T13:08:59.539108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:38.043405+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T13:09:38.043405+0000","last_peered":"2024-03-22T13:09:38.043405+0000","last_clean":"2024-03-22T13:09:38.043405+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T13:09:38.043405+0000","last_undegraded":"2024-03-22T13:09:38.043405+0000","last_fullsized":"2024-03-22T13:09:38.043405+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:45.197847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.474702+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T13:08:59.474702+0000","last_peered":"2024-03-22T13:08:59.474702+0000","last_clean":"2024-03-22T13:08:59.474702+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T13:08:59.474702+0000","last_undegraded":"2024-03-22T13:08:59.474702+0000","last_fullsized":"2024-03-22T13:08:59.474702+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:23.839356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452447+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T13:08:59.452447+0000","last_peered":"2024-03-22T13:08:59.452447+0000","last_clean":"2024-03-22T13:08:59.452447+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T13:08:59.452447+0000","last_undegraded":"2024-03-22T13:08:59.452447+0000","last_fullsized":"2024-03-22T13:08:59.452447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.540600+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T13:09:09.540600+0000","last_peered":"2024-03-22T13:09:09.540600+0000","last_clean":"2024-03-22T13:09:09.540600+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T13:09:09.540600+0000","last_undegraded":"2024-03-22T13:09:09.540600+0000","last_fullsized":"2024-03-22T13:09:09.540600+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.534260+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T13:08:29.534260+0000","last_peered":"2024-03-22T13:08:29.534260+0000","last_clean":"2024-03-22T13:08:29.534260+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T13:08:29.534260+0000","last_undegraded":"2024-03-22T13:08:29.534260+0000","last_fullsized":"2024-03-22T13:08:29.534260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.037760+0000","last_change":"2024-03-22T12:56:28.763384+0000","last_active":"2024-03-22T13:08:58.037760+0000","last_peered":"2024-03-22T13:08:58.037760+0000","last_clean":"2024-03-22T13:08:58.037760+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T13:08:58.037760+0000","last_undegraded":"2024-03-22T13:08:58.037760+0000","last_fullsized":"2024-03-22T13:08:58.037760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:15:45.463056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:58.029027+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T13:07:58.029027+0000","last_peered":"2024-03-22T13:07:58.029027+0000","last_clean":"2024-03-22T13:07:58.029027+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T13:07:58.029027+0000","last_undegraded":"2024-03-22T13:07:58.029027+0000","last_fullsized":"2024-03-22T13:07:58.029027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:48.651995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431440+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T13:08:59.431440+0000","last_peered":"2024-03-22T13:08:59.431440+0000","last_clean":"2024-03-22T13:08:59.431440+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T13:08:59.431440+0000","last_undegraded":"2024-03-22T13:08:59.431440+0000","last_fullsized":"2024-03-22T13:08:59.431440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:36:50.048241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:28.090677+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T13:08:28.090677+0000","last_peered":"2024-03-22T13:08:28.090677+0000","last_clean":"2024-03-22T13:08:28.090677+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T13:08:28.090677+0000","last_undegraded":"2024-03-22T13:08:28.090677+0000","last_fullsized":"2024-03-22T13:08:28.090677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.437421+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T13:09:39.437421+0000","last_peered":"2024-03-22T13:09:39.437421+0000","last_clean":"2024-03-22T13:09:39.437421+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T13:09:39.437421+0000","last_undegraded":"2024-03-22T13:09:39.437421+0000","last_fullsized":"2024-03-22T13:09:39.437421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:29:31.400078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360426+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T13:08:59.360426+0000","last_peered":"2024-03-22T13:08:59.360426+0000","last_clean":"2024-03-22T13:08:59.360426+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T13:08:59.360426+0000","last_undegraded":"2024-03-22T13:08:59.360426+0000","last_fullsized":"2024-03-22T13:08:59.360426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:11.480778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452461+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T13:08:59.452461+0000","last_peered":"2024-03-22T13:08:59.452461+0000","last_clean":"2024-03-22T13:08:59.452461+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T13:08:59.452461+0000","last_undegraded":"2024-03-22T13:08:59.452461+0000","last_fullsized":"2024-03-22T13:08:59.452461+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":194,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:18.089652+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T13:08:18.089652+0000","last_peered":"2024-03-22T13:08:18.089652+0000","last_clean":"2024-03-22T13:08:18.089652+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T13:08:18.089652+0000","last_undegraded":"2024-03-22T13:08:18.089652+0000","last_fullsized":"2024-03-22T13:08:18.089652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047239+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T13:09:08.047239+0000","last_peered":"2024-03-22T13:09:08.047239+0000","last_clean":"2024-03-22T13:09:08.047239+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T13:09:08.047239+0000","last_undegraded":"2024-03-22T13:09:08.047239+0000","last_fullsized":"2024-03-22T13:09:08.047239+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086769+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T13:07:48.086769+0000","last_peered":"2024-03-22T13:07:48.086769+0000","last_clean":"2024-03-22T13:07:48.086769+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T13:07:48.086769+0000","last_undegraded":"2024-03-22T13:07:48.086769+0000","last_fullsized":"2024-03-22T13:07:48.086769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"31'6","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.026555+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T13:07:43.026555+0000","last_peered":"2024-03-22T13:07:43.026555+0000","last_clean":"2024-03-22T13:07:43.026555+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T13:07:43.026555+0000","last_undegraded":"2024-03-22T13:07:43.026555+0000","last_fullsized":"2024-03-22T13:07:43.026555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:38.050005+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T13:09:38.050005+0000","last_peered":"2024-03-22T13:09:38.050005+0000","last_clean":"2024-03-22T13:09:38.050005+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T13:09:38.050005+0000","last_undegraded":"2024-03-22T13:09:38.050005+0000","last_fullsized":"2024-03-22T13:09:38.050005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.445504+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T13:07:44.445504+0000","last_peered":"2024-03-22T13:07:44.445504+0000","last_clean":"2024-03-22T13:07:44.445504+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T13:07:44.445504+0000","last_undegraded":"2024-03-22T13:07:44.445504+0000","last_fullsized":"2024-03-22T13:07:44.445504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"31'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.528186+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T13:07:44.528186+0000","last_peered":"2024-03-22T13:07:44.528186+0000","last_clean":"2024-03-22T13:07:44.528186+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T13:07:44.528186+0000","last_undegraded":"2024-03-22T13:07:44.528186+0000","last_fullsized":"2024-03-22T13:07:44.528186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:22.329365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:54.421240+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T13:07:54.421240+0000","last_peered":"2024-03-22T13:07:54.421240+0000","last_clean":"2024-03-22T13:07:54.421240+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T13:07:54.421240+0000","last_undegraded":"2024-03-22T13:07:54.421240+0000","last_fullsized":"2024-03-22T13:07:54.421240+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086812+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T13:07:48.086812+0000","last_peered":"2024-03-22T13:07:48.086812+0000","last_clean":"2024-03-22T13:07:48.086812+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T13:07:48.086812+0000","last_undegraded":"2024-03-22T13:07:48.086812+0000","last_fullsized":"2024-03-22T13:07:48.086812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"31'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:43.026598+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T13:07:43.026598+0000","last_peered":"2024-03-22T13:07:43.026598+0000","last_clean":"2024-03-22T13:07:43.026598+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T13:07:43.026598+0000","last_undegraded":"2024-03-22T13:07:43.026598+0000","last_fullsized":"2024-03-22T13:07:43.026598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.456333+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T13:09:39.456333+0000","last_peered":"2024-03-22T13:09:39.456333+0000","last_clean":"2024-03-22T13:09:39.456333+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T13:09:39.456333+0000","last_undegraded":"2024-03-22T13:09:39.456333+0000","last_fullsized":"2024-03-22T13:09:39.456333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086795+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T13:07:48.086795+0000","last_peered":"2024-03-22T13:07:48.086795+0000","last_clean":"2024-03-22T13:07:48.086795+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T13:07:48.086795+0000","last_undegraded":"2024-03-22T13:07:48.086795+0000","last_fullsized":"2024-03-22T13:07:48.086795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"31'2","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:15.533469+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T13:08:15.533469+0000","last_peered":"2024-03-22T13:08:15.533469+0000","last_clean":"2024-03-22T13:08:15.533469+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T13:08:15.533469+0000","last_undegraded":"2024-03-22T13:08:15.533469+0000","last_fullsized":"2024-03-22T13:08:15.533469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:24:46.021828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.478501+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T13:09:39.478501+0000","last_peered":"2024-03-22T13:09:39.478501+0000","last_clean":"2024-03-22T13:09:39.478501+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T13:09:39.478501+0000","last_undegraded":"2024-03-22T13:09:39.478501+0000","last_fullsized":"2024-03-22T13:09:39.478501+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:38.316319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089125+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T13:08:13.089125+0000","last_peered":"2024-03-22T13:08:13.089125+0000","last_clean":"2024-03-22T13:08:13.089125+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T13:08:13.089125+0000","last_undegraded":"2024-03-22T13:08:13.089125+0000","last_fullsized":"2024-03-22T13:08:13.089125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"31'18","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.030746+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T13:08:08.030746+0000","last_peered":"2024-03-22T13:08:08.030746+0000","last_clean":"2024-03-22T13:08:08.030746+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T13:08:08.030746+0000","last_undegraded":"2024-03-22T13:08:08.030746+0000","last_fullsized":"2024-03-22T13:08:08.030746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:20:57.722062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.429460+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T13:08:44.429460+0000","last_peered":"2024-03-22T13:08:44.429460+0000","last_clean":"2024-03-22T13:08:44.429460+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T13:08:44.429460+0000","last_undegraded":"2024-03-22T13:08:44.429460+0000","last_fullsized":"2024-03-22T13:08:44.429460+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:43.029094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.042504+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T13:08:13.042504+0000","last_peered":"2024-03-22T13:08:13.042504+0000","last_clean":"2024-03-22T13:08:13.042504+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T13:08:13.042504+0000","last_undegraded":"2024-03-22T13:08:13.042504+0000","last_fullsized":"2024-03-22T13:08:13.042504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.041967+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T13:08:08.041967+0000","last_peered":"2024-03-22T13:08:08.041967+0000","last_clean":"2024-03-22T13:08:08.041967+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T13:08:08.041967+0000","last_undegraded":"2024-03-22T13:08:08.041967+0000","last_fullsized":"2024-03-22T13:08:08.041967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.358426+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T13:08:44.358426+0000","last_peered":"2024-03-22T13:08:44.358426+0000","last_clean":"2024-03-22T13:08:44.358426+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T13:08:44.358426+0000","last_undegraded":"2024-03-22T13:08:44.358426+0000","last_fullsized":"2024-03-22T13:08:44.358426+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:51.788457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089153+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T13:08:13.089153+0000","last_peered":"2024-03-22T13:08:13.089153+0000","last_clean":"2024-03-22T13:08:13.089153+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T13:08:13.089153+0000","last_undegraded":"2024-03-22T13:08:13.089153+0000","last_fullsized":"2024-03-22T13:08:13.089153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447799+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T13:08:09.447799+0000","last_peered":"2024-03-22T13:08:09.447799+0000","last_clean":"2024-03-22T13:08:09.447799+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T13:08:09.447799+0000","last_undegraded":"2024-03-22T13:08:09.447799+0000","last_fullsized":"2024-03-22T13:08:09.447799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354809+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T13:08:19.354809+0000","last_peered":"2024-03-22T13:08:19.354809+0000","last_clean":"2024-03-22T13:08:19.354809+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T13:08:19.354809+0000","last_undegraded":"2024-03-22T13:08:19.354809+0000","last_fullsized":"2024-03-22T13:08:19.354809+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:08:12.860933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423855+0000","last_change":"2024-03-22T12:56:30.299969+0000","last_active":"2024-03-22T13:08:09.423855+0000","last_peered":"2024-03-22T13:08:09.423855+0000","last_clean":"2024-03-22T13:08:09.423855+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T13:08:09.423855+0000","last_undegraded":"2024-03-22T13:08:09.423855+0000","last_fullsized":"2024-03-22T13:08:09.423855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'9","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423914+0000","last_change":"2024-03-22T12:56:28.777142+0000","last_active":"2024-03-22T13:08:09.423914+0000","last_peered":"2024-03-22T13:08:09.423914+0000","last_clean":"2024-03-22T13:08:09.423914+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T13:08:09.423914+0000","last_undegraded":"2024-03-22T13:08:09.423914+0000","last_fullsized":"2024-03-22T13:08:09.423914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:49.411808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.451139+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T13:08:44.451139+0000","last_peered":"2024-03-22T13:08:44.451139+0000","last_clean":"2024-03-22T13:08:44.451139+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T13:08:44.451139+0000","last_undegraded":"2024-03-22T13:08:44.451139+0000","last_fullsized":"2024-03-22T13:08:44.451139+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091724+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T13:08:38.091724+0000","last_peered":"2024-03-22T13:08:38.091724+0000","last_clean":"2024-03-22T13:08:38.091724+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T13:08:38.091724+0000","last_undegraded":"2024-03-22T13:08:38.091724+0000","last_fullsized":"2024-03-22T13:08:38.091724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'14","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.042011+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T13:08:08.042011+0000","last_peered":"2024-03-22T13:08:08.042011+0000","last_clean":"2024-03-22T13:08:08.042011+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T13:08:08.042011+0000","last_undegraded":"2024-03-22T13:08:08.042011+0000","last_fullsized":"2024-03-22T13:08:08.042011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533275+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T13:08:19.533275+0000","last_peered":"2024-03-22T13:08:19.533275+0000","last_clean":"2024-03-22T13:08:19.533275+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T13:08:19.533275+0000","last_undegraded":"2024-03-22T13:08:19.533275+0000","last_fullsized":"2024-03-22T13:08:19.533275+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:26:02.964217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.470116+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T13:08:09.470116+0000","last_peered":"2024-03-22T13:08:09.470116+0000","last_clean":"2024-03-22T13:08:09.470116+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T13:08:09.470116+0000","last_undegraded":"2024-03-22T13:08:09.470116+0000","last_fullsized":"2024-03-22T13:08:09.470116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:30:49.776407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.535102+0000","last_change":"2024-03-22T12:56:28.769041+0000","last_active":"2024-03-22T13:08:34.535102+0000","last_peered":"2024-03-22T13:08:34.535102+0000","last_clean":"2024-03-22T13:08:34.535102+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T13:08:34.535102+0000","last_undegraded":"2024-03-22T13:08:34.535102+0000","last_fullsized":"2024-03-22T13:08:34.535102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.471072+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T13:08:19.471072+0000","last_peered":"2024-03-22T13:08:19.471072+0000","last_clean":"2024-03-22T13:08:19.471072+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T13:08:19.471072+0000","last_undegraded":"2024-03-22T13:08:19.471072+0000","last_fullsized":"2024-03-22T13:08:19.471072+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:56.815720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.428126+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T13:08:34.428126+0000","last_peered":"2024-03-22T13:08:34.428126+0000","last_clean":"2024-03-22T13:08:34.428126+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T13:08:34.428126+0000","last_undegraded":"2024-03-22T13:08:34.428126+0000","last_fullsized":"2024-03-22T13:08:34.428126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450282+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T13:08:34.450282+0000","last_peered":"2024-03-22T13:08:34.450282+0000","last_clean":"2024-03-22T13:08:34.450282+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T13:08:34.450282+0000","last_undegraded":"2024-03-22T13:08:34.450282+0000","last_fullsized":"2024-03-22T13:08:34.450282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354747+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T13:08:19.354747+0000","last_peered":"2024-03-22T13:08:19.354747+0000","last_clean":"2024-03-22T13:08:19.354747+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T13:08:19.354747+0000","last_undegraded":"2024-03-22T13:08:19.354747+0000","last_fullsized":"2024-03-22T13:08:19.354747+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:06:47.716384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450270+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T13:08:34.450270+0000","last_peered":"2024-03-22T13:08:34.450270+0000","last_clean":"2024-03-22T13:08:34.450270+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T13:08:34.450270+0000","last_undegraded":"2024-03-22T13:08:34.450270+0000","last_fullsized":"2024-03-22T13:08:34.450270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:33.044141+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T13:08:33.044141+0000","last_peered":"2024-03-22T13:08:33.044141+0000","last_clean":"2024-03-22T13:08:33.044141+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T13:08:33.044141+0000","last_undegraded":"2024-03-22T13:08:33.044141+0000","last_fullsized":"2024-03-22T13:08:33.044141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.354781+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T13:08:19.354781+0000","last_peered":"2024-03-22T13:08:19.354781+0000","last_clean":"2024-03-22T13:08:19.354781+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T13:08:19.354781+0000","last_undegraded":"2024-03-22T13:08:19.354781+0000","last_fullsized":"2024-03-22T13:08:19.354781+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:39.428904+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T13:08:39.428904+0000","last_peered":"2024-03-22T13:08:39.428904+0000","last_clean":"2024-03-22T13:08:39.428904+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T13:08:39.428904+0000","last_undegraded":"2024-03-22T13:08:39.428904+0000","last_fullsized":"2024-03-22T13:08:39.428904+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:19.126603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.534984+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T13:08:34.534984+0000","last_peered":"2024-03-22T13:08:34.534984+0000","last_clean":"2024-03-22T13:08:34.534984+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T13:08:34.534984+0000","last_undegraded":"2024-03-22T13:08:34.534984+0000","last_fullsized":"2024-03-22T13:08:34.534984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450228+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T13:08:34.450228+0000","last_peered":"2024-03-22T13:08:34.450228+0000","last_clean":"2024-03-22T13:08:34.450228+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T13:08:34.450228+0000","last_undegraded":"2024-03-22T13:08:34.450228+0000","last_fullsized":"2024-03-22T13:08:34.450228+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047276+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T13:09:08.047276+0000","last_peered":"2024-03-22T13:09:08.047276+0000","last_clean":"2024-03-22T13:09:08.047276+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T13:09:08.047276+0000","last_undegraded":"2024-03-22T13:09:08.047276+0000","last_fullsized":"2024-03-22T13:09:08.047276+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356605+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T13:08:34.356605+0000","last_peered":"2024-03-22T13:08:34.356605+0000","last_clean":"2024-03-22T13:08:34.356605+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T13:08:34.356605+0000","last_undegraded":"2024-03-22T13:08:34.356605+0000","last_fullsized":"2024-03-22T13:08:34.356605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:42:14.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356655+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T13:08:34.356655+0000","last_peered":"2024-03-22T13:08:34.356655+0000","last_clean":"2024-03-22T13:08:34.356655+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T13:08:34.356655+0000","last_undegraded":"2024-03-22T13:08:34.356655+0000","last_fullsized":"2024-03-22T13:08:34.356655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:31:41.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.046243+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T13:08:58.046243+0000","last_peered":"2024-03-22T13:08:58.046243+0000","last_clean":"2024-03-22T13:08:58.046243+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T13:08:58.046243+0000","last_undegraded":"2024-03-22T13:08:58.046243+0000","last_fullsized":"2024-03-22T13:08:58.046243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:08.021699+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T13:07:08.021699+0000","last_peered":"2024-03-22T13:07:08.021699+0000","last_clean":"2024-03-22T13:07:08.021699+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T13:07:08.021699+0000","last_undegraded":"2024-03-22T13:07:08.021699+0000","last_fullsized":"2024-03-22T13:07:08.021699+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.039286+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T13:09:08.039286+0000","last_peered":"2024-03-22T13:09:08.039286+0000","last_clean":"2024-03-22T13:09:08.039286+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T13:09:08.039286+0000","last_undegraded":"2024-03-22T13:09:08.039286+0000","last_fullsized":"2024-03-22T13:09:08.039286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:43.256977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360446+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T13:08:59.360446+0000","last_peered":"2024-03-22T13:08:59.360446+0000","last_clean":"2024-03-22T13:08:59.360446+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T13:08:59.360446+0000","last_undegraded":"2024-03-22T13:08:59.360446+0000","last_fullsized":"2024-03-22T13:08:59.360446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:12:49.535163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.474675+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T13:08:59.474675+0000","last_peered":"2024-03-22T13:08:59.474675+0000","last_clean":"2024-03-22T13:08:59.474675+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T13:08:59.474675+0000","last_undegraded":"2024-03-22T13:08:59.474675+0000","last_fullsized":"2024-03-22T13:08:59.474675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:48.882251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:09.361490+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T13:09:09.361490+0000","last_peered":"2024-03-22T13:09:09.361490+0000","last_clean":"2024-03-22T13:09:09.361490+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T13:09:09.361490+0000","last_undegraded":"2024-03-22T13:09:09.361490+0000","last_fullsized":"2024-03-22T13:09:09.361490+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.046714+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T13:09:03.046714+0000","last_peered":"2024-03-22T13:09:03.046714+0000","last_clean":"2024-03-22T13:09:03.046714+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T13:09:03.046714+0000","last_undegraded":"2024-03-22T13:09:03.046714+0000","last_fullsized":"2024-03-22T13:09:03.046714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.038633+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T13:09:03.038633+0000","last_peered":"2024-03-22T13:09:03.038633+0000","last_clean":"2024-03-22T13:09:03.038633+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T13:09:03.038633+0000","last_undegraded":"2024-03-22T13:09:03.038633+0000","last_fullsized":"2024-03-22T13:09:03.038633+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452433+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T13:08:59.452433+0000","last_peered":"2024-03-22T13:08:59.452433+0000","last_clean":"2024-03-22T13:08:59.452433+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T13:08:59.452433+0000","last_undegraded":"2024-03-22T13:08:59.452433+0000","last_fullsized":"2024-03-22T13:08:59.452433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431501+0000","last_change":"2024-03-22T12:56:30.300244+0000","last_active":"2024-03-22T13:08:59.431501+0000","last_peered":"2024-03-22T13:08:59.431501+0000","last_clean":"2024-03-22T13:08:59.431501+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T13:08:59.431501+0000","last_undegraded":"2024-03-22T13:08:59.431501+0000","last_fullsized":"2024-03-22T13:08:59.431501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:42:11.091814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363660+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T13:09:29.363660+0000","last_peered":"2024-03-22T13:09:29.363660+0000","last_clean":"2024-03-22T13:09:29.363660+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T13:09:29.363660+0000","last_undegraded":"2024-03-22T13:09:29.363660+0000","last_fullsized":"2024-03-22T13:09:29.363660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.542819+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T13:09:29.542819+0000","last_peered":"2024-03-22T13:09:29.542819+0000","last_clean":"2024-03-22T13:09:29.542819+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T13:09:29.542819+0000","last_undegraded":"2024-03-22T13:09:29.542819+0000","last_fullsized":"2024-03-22T13:09:29.542819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:23:24.028635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049650+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T13:09:33.049650+0000","last_peered":"2024-03-22T13:09:33.049650+0000","last_clean":"2024-03-22T13:09:33.049650+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T13:09:33.049650+0000","last_undegraded":"2024-03-22T13:09:33.049650+0000","last_fullsized":"2024-03-22T13:09:33.049650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:29.363696+0000","last_change":"2024-03-22T12:56:30.308903+0000","last_active":"2024-03-22T13:09:29.363696+0000","last_peered":"2024-03-22T13:09:29.363696+0000","last_clean":"2024-03-22T13:09:29.363696+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T13:09:29.363696+0000","last_undegraded":"2024-03-22T13:09:29.363696+0000","last_fullsized":"2024-03-22T13:09:29.363696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:58.028924+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T13:07:58.028924+0000","last_peered":"2024-03-22T13:07:58.028924+0000","last_clean":"2024-03-22T13:07:58.028924+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T13:07:58.028924+0000","last_undegraded":"2024-03-22T13:07:58.028924+0000","last_fullsized":"2024-03-22T13:07:58.028924+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:13:57.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533242+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T13:08:19.533242+0000","last_peered":"2024-03-22T13:08:19.533242+0000","last_clean":"2024-03-22T13:08:19.533242+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T13:08:19.533242+0000","last_undegraded":"2024-03-22T13:08:19.533242+0000","last_fullsized":"2024-03-22T13:08:19.533242+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:19.797978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.445455+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T13:07:44.445455+0000","last_peered":"2024-03-22T13:07:44.445455+0000","last_clean":"2024-03-22T13:07:44.445455+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T13:07:44.445455+0000","last_undegraded":"2024-03-22T13:07:44.445455+0000","last_fullsized":"2024-03-22T13:07:44.445455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:44.419780+0000","last_change":"2024-03-22T12:56:30.299214+0000","last_active":"2024-03-22T13:07:44.419780+0000","last_peered":"2024-03-22T13:07:44.419780+0000","last_clean":"2024-03-22T13:07:44.419780+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T13:07:44.419780+0000","last_undegraded":"2024-03-22T13:07:44.419780+0000","last_fullsized":"2024-03-22T13:07:44.419780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:35.356723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.040052+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T13:07:48.040052+0000","last_peered":"2024-03-22T13:07:48.040052+0000","last_clean":"2024-03-22T13:07:48.040052+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T13:07:48.040052+0000","last_undegraded":"2024-03-22T13:07:48.040052+0000","last_fullsized":"2024-03-22T13:07:48.040052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:19.533301+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T13:08:19.533301+0000","last_peered":"2024-03-22T13:08:19.533301+0000","last_clean":"2024-03-22T13:08:19.533301+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T13:08:19.533301+0000","last_undegraded":"2024-03-22T13:08:19.533301+0000","last_fullsized":"2024-03-22T13:08:19.533301+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:57:33.808269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.031478+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T13:08:13.031478+0000","last_peered":"2024-03-22T13:08:13.031478+0000","last_clean":"2024-03-22T13:08:13.031478+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T13:08:13.031478+0000","last_undegraded":"2024-03-22T13:08:13.031478+0000","last_fullsized":"2024-03-22T13:08:13.031478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:38.987151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447831+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T13:08:09.447831+0000","last_peered":"2024-03-22T13:08:09.447831+0000","last_clean":"2024-03-22T13:08:09.447831+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T13:08:09.447831+0000","last_undegraded":"2024-03-22T13:08:09.447831+0000","last_fullsized":"2024-03-22T13:08:09.447831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447764+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T13:08:09.447764+0000","last_peered":"2024-03-22T13:08:09.447764+0000","last_clean":"2024-03-22T13:08:09.447764+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T13:08:09.447764+0000","last_undegraded":"2024-03-22T13:08:09.447764+0000","last_fullsized":"2024-03-22T13:08:09.447764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.532025+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T13:08:09.532025+0000","last_peered":"2024-03-22T13:08:09.532025+0000","last_clean":"2024-03-22T13:08:09.532025+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T13:08:09.532025+0000","last_undegraded":"2024-03-22T13:08:09.532025+0000","last_fullsized":"2024-03-22T13:08:09.532025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.423817+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T13:08:09.423817+0000","last_peered":"2024-03-22T13:08:09.423817+0000","last_clean":"2024-03-22T13:08:09.423817+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T13:08:09.423817+0000","last_undegraded":"2024-03-22T13:08:09.423817+0000","last_fullsized":"2024-03-22T13:08:09.423817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.356700+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T13:08:34.356700+0000","last_peered":"2024-03-22T13:08:34.356700+0000","last_clean":"2024-03-22T13:08:34.356700+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T13:08:34.356700+0000","last_undegraded":"2024-03-22T13:08:34.356700+0000","last_fullsized":"2024-03-22T13:08:34.356700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:27:18.226099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.472507+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T13:08:34.472507+0000","last_peered":"2024-03-22T13:08:34.472507+0000","last_clean":"2024-03-22T13:08:34.472507+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T13:08:34.472507+0000","last_undegraded":"2024-03-22T13:08:34.472507+0000","last_fullsized":"2024-03-22T13:08:34.472507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:28:23.598956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044564+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T13:08:38.044564+0000","last_peered":"2024-03-22T13:08:38.044564+0000","last_clean":"2024-03-22T13:08:38.044564+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T13:08:38.044564+0000","last_undegraded":"2024-03-22T13:08:38.044564+0000","last_fullsized":"2024-03-22T13:08:38.044564+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044582+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T13:08:38.044582+0000","last_peered":"2024-03-22T13:08:38.044582+0000","last_clean":"2024-03-22T13:08:38.044582+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T13:08:38.044582+0000","last_undegraded":"2024-03-22T13:08:38.044582+0000","last_fullsized":"2024-03-22T13:08:38.044582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091763+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T13:08:38.091763+0000","last_peered":"2024-03-22T13:08:38.091763+0000","last_clean":"2024-03-22T13:08:38.091763+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T13:08:38.091763+0000","last_undegraded":"2024-03-22T13:08:38.091763+0000","last_fullsized":"2024-03-22T13:08:38.091763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.360387+0000","last_change":"2024-03-22T12:56:30.309258+0000","last_active":"2024-03-22T13:08:59.360387+0000","last_peered":"2024-03-22T13:08:59.360387+0000","last_clean":"2024-03-22T13:08:59.360387+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T13:08:59.360387+0000","last_undegraded":"2024-03-22T13:08:59.360387+0000","last_fullsized":"2024-03-22T13:08:59.360387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:15.204508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094308+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T13:09:03.094308+0000","last_peered":"2024-03-22T13:09:03.094308+0000","last_clean":"2024-03-22T13:09:03.094308+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T13:09:03.094308+0000","last_undegraded":"2024-03-22T13:09:03.094308+0000","last_fullsized":"2024-03-22T13:09:03.094308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452409+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T13:08:59.452409+0000","last_peered":"2024-03-22T13:08:59.452409+0000","last_clean":"2024-03-22T13:08:59.452409+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T13:08:59.452409+0000","last_undegraded":"2024-03-22T13:08:59.452409+0000","last_fullsized":"2024-03-22T13:08:59.452409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094342+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T13:09:03.094342+0000","last_peered":"2024-03-22T13:09:03.094342+0000","last_clean":"2024-03-22T13:09:03.094342+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T13:09:03.094342+0000","last_undegraded":"2024-03-22T13:09:03.094342+0000","last_fullsized":"2024-03-22T13:09:03.094342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.449798+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T13:08:29.449798+0000","last_peered":"2024-03-22T13:08:29.449798+0000","last_clean":"2024-03-22T13:08:29.449798+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T13:08:29.449798+0000","last_undegraded":"2024-03-22T13:08:29.449798+0000","last_fullsized":"2024-03-22T13:08:29.449798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.431526+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T13:08:59.431526+0000","last_peered":"2024-03-22T13:08:59.431526+0000","last_clean":"2024-03-22T13:08:59.431526+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T13:08:59.431526+0000","last_undegraded":"2024-03-22T13:08:59.431526+0000","last_fullsized":"2024-03-22T13:08:59.431526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:14.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.539145+0000","last_change":"2024-03-22T12:56:30.303254+0000","last_active":"2024-03-22T13:08:59.539145+0000","last_peered":"2024-03-22T13:08:59.539145+0000","last_clean":"2024-03-22T13:08:59.539145+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T13:08:59.539145+0000","last_undegraded":"2024-03-22T13:08:59.539145+0000","last_fullsized":"2024-03-22T13:08:59.539145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094374+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T13:09:03.094374+0000","last_peered":"2024-03-22T13:09:03.094374+0000","last_clean":"2024-03-22T13:09:03.094374+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T13:09:03.094374+0000","last_undegraded":"2024-03-22T13:09:03.094374+0000","last_fullsized":"2024-03-22T13:09:03.094374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509634,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":142876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509634,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":140274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509634,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":143610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509634,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":140476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509633,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":730186,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542339,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":731412,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":143576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542338,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T13:12:06.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T13:12:06.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.428+0000 7fe663000640 1 Processor -- start 2024-03-22T13:12:06.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.429+0000 7fe663000640 1 -- start start 2024-03-22T13:12:06.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.430+0000 7fe663000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c154700 0x7fe65c0c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:06.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.430+0000 7fe663000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c15abf0 con 0x7fe65c0c9900 2024-03-22T13:12:06.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.430+0000 7fe663000640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c15b150 con 0x7fe65c0706b0 2024-03-22T13:12:06.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.430+0000 7fe663000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c070c90 con 0x7fe65c154700 2024-03-22T13:12:06.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.431+0000 7fe661c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c154700 0x7fe65c0c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:06.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.431+0000 7fe661200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe65c0706b0 0x7fe65c0c91f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43784/0 (socket says 172.21.15.6:43784) 2024-03-22T13:12:06.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.431+0000 7fe661200640 1 -- 172.21.15.6:0/3098865911 learned_addr learned my addr 172.21.15.6:0/3098865911 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:06.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.434+0000 7fe65be00640 1 -- 172.21.15.6:0/3098865911 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360263442 0 0) 0x7fe65c15b150 con 0x7fe65c0706b0 2024-03-22T13:12:06.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.434+0000 7fe661c00640 1 -- 172.21.15.6:0/3098865911 >> v1:172.21.15.92:6789/0 conn(0x7fe65c0706b0 legacy=0x7fe65c0c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:06.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.434+0000 7fe661c00640 1 -- 172.21.15.6:0/3098865911 >> v1:172.21.15.6:6789/0 conn(0x7fe65c0c9900 legacy=0x7fe65c164b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:06.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.434+0000 7fe661c00640 1 -- 172.21.15.6:0/3098865911 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c15d8c0 con 0x7fe65c154700 2024-03-22T13:12:06.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.439+0000 7fe661c00640 1 --2- 172.21.15.6:0/3098865911 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c154700 0x7fe65c0c73b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6480097d0 tx=0x7fe64802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ded44fc50e3494a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.444+0000 7fe65be00640 1 -- 172.21.15.6:0/3098865911 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648035020 con 0x7fe65c154700 2024-03-22T13:12:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.444+0000 7fe65be00640 1 -- 172.21.15.6:0/3098865911 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe648031800 con 0x7fe65c154700 2024-03-22T13:12:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.444+0000 7fe65be00640 1 -- 172.21.15.6:0/3098865911 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648031bd0 con 0x7fe65c154700 2024-03-22T13:12:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.445+0000 7fe663000640 1 -- 172.21.15.6:0/3098865911 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c154700 msgr2=0x7fe65c0c73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.445+0000 7fe663000640 1 --2- 172.21.15.6:0/3098865911 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c154700 0x7fe65c0c73b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6480097d0 tx=0x7fe64802b8a0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.445+0000 7fe663000640 1 -- 172.21.15.6:0/3098865911 shutdown_connections 2024-03-22T13:12:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.445+0000 7fe663000640 1 --2- 172.21.15.6:0/3098865911 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c154700 0x7fe65c0c73b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.445+0000 7fe663000640 1 -- 172.21.15.6:0/3098865911 >> 172.21.15.6:0/3098865911 conn(0x7fe65c06c190 msgr2=0x7fe65c06c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.445+0000 7fe663000640 1 -- 172.21.15.6:0/3098865911 shutdown_connections 2024-03-22T13:12:06.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.445+0000 7fe663000640 1 -- 172.21.15.6:0/3098865911 wait complete. 2024-03-22T13:12:06.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.449+0000 7fe663000640 1 Processor -- start 2024-03-22T13:12:06.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.449+0000 7fe663000640 1 -- start start 2024-03-22T13:12:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.451+0000 7fe663000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0706b0 0x7fe65c0cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.451+0000 7fe663000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0c9900 0x7fe65c0c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.451+0000 7fe663000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe65c154700 0x7fe65c158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.451+0000 7fe663000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c15de40 con 0x7fe65c154700 2024-03-22T13:12:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.451+0000 7fe663000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c15db60 con 0x7fe65c0c9900 2024-03-22T13:12:06.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.451+0000 7fe663000640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c15e120 con 0x7fe65c0706b0 2024-03-22T13:12:06.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.452+0000 7fe661c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0706b0 0x7fe65c0cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:06.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.452+0000 7fe661200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0c9900 0x7fe65c0c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:06.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.452+0000 7fe661c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0706b0 0x7fe65c0cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54288/0 (socket says 172.21.15.6:54288) 2024-03-22T13:12:06.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.452+0000 7fe661c00640 1 -- 172.21.15.6:0/1308138628 learned_addr learned my addr 172.21.15.6:0/1308138628 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:06.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.456+0000 7fe662600640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe65c154700 0x7fe65c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.456+0000 7fe661200640 1 -- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0706b0 msgr2=0x7fe65c0cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.456+0000 7fe661200640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0706b0 0x7fe65c0cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.456+0000 7fe661200640 1 -- 172.21.15.6:0/1308138628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe65c154700 msgr2=0x7fe65c158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.456+0000 7fe661200640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe65c154700 0x7fe65c158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:06.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.456+0000 7fe661200640 1 -- 172.21.15.6:0/1308138628 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c1532e0 con 0x7fe65c0c9900 2024-03-22T13:12:06.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.467+0000 7fe661200640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0c9900 0x7fe65c0c8ea0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe65c070ab0 tx=0x7fe65000c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:06.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.547+0000 7fe659600640 1 -- 172.21.15.6:0/1308138628 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6500074e0 con 0x7fe65c0c9900 2024-03-22T13:12:06.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.547+0000 7fe659600640 1 -- 172.21.15.6:0/1308138628 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe650007680 con 0x7fe65c0c9900 2024-03-22T13:12:06.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.547+0000 7fe659600640 1 -- 172.21.15.6:0/1308138628 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe650004a20 con 0x7fe65c0c9900 2024-03-22T13:12:06.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.547+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c154c60 con 0x7fe65c0c9900 2024-03-22T13:12:06.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.547+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c153830 con 0x7fe65c0c9900 2024-03-22T13:12:06.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.549+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c075650 con 0x7fe65c0c9900 2024-03-22T13:12:06.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.576+0000 7fe659600640 1 -- 172.21.15.6:0/1308138628 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe650011070 con 0x7fe65c0c9900 2024-03-22T13:12:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.578+0000 7fe659600640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe63c06bc20 0x7fe63c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.579+0000 7fe661c00640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe63c06bc20 0x7fe63c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:06.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.580+0000 7fe661c00640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe63c06bc20 0x7fe63c06e0e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe648000c00 tx=0x7fe64803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:06.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.833+0000 7fe659600640 1 -- 172.21.15.6:0/1308138628 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fe6500570c0 con 0x7fe65c0c9900 2024-03-22T13:12:06.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:06.845+0000 7fe659600640 1 -- 172.21.15.6:0/1308138628 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe650059720 con 0x7fe65c0c9900 2024-03-22T13:12:07.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.058+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe65c158010 con 0x7fe65c0c9900 2024-03-22T13:12:07.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.121+0000 7fe659600640 1 -- 172.21.15.6:0/1308138628 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+19030 (secure 0 0 0) 0x7fe650059020 con 0x7fe65c0c9900 2024-03-22T13:12:07.121 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:12:07.122 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":31,"fsid":"1722e499-f2bc-49ac-a1c9-7ab0d7863db9","created":"2024-03-22T12:55:55.276483+0000","modified":"2024-03-22T12:58:17.973475+0000","last_up_change":"2024-03-22T12:56:13.373035+0000","last_in_change":"2024-03-22T12:56:03.396760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T12:56:14.196164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T12:56:26.857509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T12:56:28.394044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T12:56:34.623081+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"256bea72-2eca-4d53-a283-029ac721281b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6801","nonce":2278862740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6803","nonce":2278862740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6809","nonce":2278862740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2278862740},{"type":"v1","addr":"172.21.15.6:6805","nonce":2278862740}]},"public_addr":"172.21.15.6:6801/2278862740","cluster_addr":"172.21.15.6:6803/2278862740","heartbeat_back_addr":"172.21.15.6:6809/2278862740","heartbeat_front_addr":"172.21.15.6:6805/2278862740","state":["exists","up"]},{"osd":1,"uuid":"1dad1564-b377-49fa-91e8-ddbe1febfeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6808","nonce":3168070181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6811","nonce":3168070181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6817","nonce":3168070181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3168070181},{"type":"v1","addr":"172.21.15.6:6814","nonce":3168070181}]},"public_addr":"172.21.15.6:6808/3168070181","cluster_addr":"172.21.15.6:6811/3168070181","heartbeat_back_addr":"172.21.15.6:6817/3168070181","heartbeat_front_addr":"172.21.15.6:6814/3168070181","state":["exists","up"]},{"osd":2,"uuid":"b7e700d8-a624-4a3c-b5ed-4b2db1e5a4e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6816","nonce":2211174308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6819","nonce":2211174308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6823","nonce":2211174308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2211174308},{"type":"v1","addr":"172.21.15.6:6821","nonce":2211174308}]},"public_addr":"172.21.15.6:6816/2211174308","cluster_addr":"172.21.15.6:6819/2211174308","heartbeat_back_addr":"172.21.15.6:6823/2211174308","heartbeat_front_addr":"172.21.15.6:6821/2211174308","state":["exists","up"]},{"osd":3,"uuid":"9ca78a23-26be-4a7a-8b71-0b17aece4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6825","nonce":484261108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6827","nonce":484261108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6831","nonce":484261108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":484261108},{"type":"v1","addr":"172.21.15.6:6829","nonce":484261108}]},"public_addr":"172.21.15.6:6825/484261108","cluster_addr":"172.21.15.6:6827/484261108","heartbeat_back_addr":"172.21.15.6:6831/484261108","heartbeat_front_addr":"172.21.15.6:6829/484261108","state":["exists","up"]},{"osd":4,"uuid":"f1ba8f52-e2b1-4dc2-a330-27f2b026e60d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6821","nonce":2540113998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6826","nonce":2540113998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6833","nonce":2540113998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2540113998},{"type":"v1","addr":"172.21.15.92:6830","nonce":2540113998}]},"public_addr":"172.21.15.92:6821/2540113998","cluster_addr":"172.21.15.92:6826/2540113998","heartbeat_back_addr":"172.21.15.92:6833/2540113998","heartbeat_front_addr":"172.21.15.92:6830/2540113998","state":["exists","up"]},{"osd":5,"uuid":"69f533c6-b98b-40bd-ac72-fea32148cb6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6811","nonce":1588636001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6813","nonce":1588636001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6817","nonce":1588636001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1588636001},{"type":"v1","addr":"172.21.15.92:6815","nonce":1588636001}]},"public_addr":"172.21.15.92:6811/1588636001","cluster_addr":"172.21.15.92:6813/1588636001","heartbeat_back_addr":"172.21.15.92:6817/1588636001","heartbeat_front_addr":"172.21.15.92:6815/1588636001","state":["exists","up"]},{"osd":6,"uuid":"5b75948a-a510-4418-b902-87cd6d84f34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6803","nonce":3646124871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6805","nonce":3646124871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6809","nonce":3646124871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3646124871},{"type":"v1","addr":"172.21.15.92:6807","nonce":3646124871}]},"public_addr":"172.21.15.92:6803/3646124871","cluster_addr":"172.21.15.92:6805/3646124871","heartbeat_back_addr":"172.21.15.92:6809/3646124871","heartbeat_front_addr":"172.21.15.92:6807/3646124871","state":["exists","up"]},{"osd":7,"uuid":"e27c5d29-4343-4274-9e28-9d3cd5c3f32b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6820","nonce":1078860650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6823","nonce":1078860650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6829","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6831","nonce":1078860650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":1078860650},{"type":"v1","addr":"172.21.15.92:6827","nonce":1078860650}]},"public_addr":"172.21.15.92:6820/1078860650","cluster_addr":"172.21.15.92:6823/1078860650","heartbeat_back_addr":"172.21.15.92:6831/1078860650","heartbeat_front_addr":"172.21.15.92:6827/1078860650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.293642+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.338970+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.088179+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.300324+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.129282+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.106939+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.044023+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T12:56:09.160283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:6837/2825293724":"2024-03-23T12:58:17.951750+0000","172.21.15.92:6836/2825293724":"2024-03-23T12:58:17.951750+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe63c06bc20 msgr2=0x7fe63c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe63c06bc20 0x7fe63c06e0e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe648000c00 tx=0x7fe64803d040 comp rx=0 tx=0).stop 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0c9900 msgr2=0x7fe65c0c8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0c9900 0x7fe65c0c8ea0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe65c070ab0 tx=0x7fe65000c870 comp rx=0 tx=0).stop 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 shutdown_connections 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe65c154700 0x7fe65c158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fe63c06bc20 0x7fe63c06e0e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:07.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0c9900 0x7fe65c0c8ea0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:07.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 --2- 172.21.15.6:0/1308138628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0706b0 0x7fe65c0cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:07.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 >> 172.21.15.6:0/1308138628 conn(0x7fe65c06c190 msgr2=0x7fe65c0c50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:07.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 shutdown_connections 2024-03-22T13:12:07.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:07.124+0000 7fe663000640 1 -- 172.21.15.6:0/1308138628 wait complete. 2024-03-22T13:12:08.144 INFO:tasks.ceph:Scrubbing osd.0 2024-03-22T13:12:08.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.347+0000 7f21ce400640 1 Processor -- start 2024-03-22T13:12:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.348+0000 7f21ce400640 1 -- start start 2024-03-22T13:12:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.349+0000 7f21ce400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c80cea70 0x7f21c815b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.349+0000 7f21ce400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f21c80ce520 0x7f21c806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.349+0000 7f21ce400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c80cc9b0 con 0x7f21c81549a0 2024-03-22T13:12:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.349+0000 7f21ce400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c815aca0 con 0x7f21c80ce520 2024-03-22T13:12:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.349+0000 7f21ce400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c80ccc90 con 0x7f21c80cea70 2024-03-22T13:12:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.350+0000 7f21c6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c80cea70 0x7f21c815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.350+0000 7f21c7e00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f21c80ce520 0x7f21c806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.350+0000 7f21c6a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c80cea70 0x7f21c815b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54296/0 (socket says 172.21.15.6:54296) 2024-03-22T13:12:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.350+0000 7f21c6a00640 1 -- 172.21.15.6:0/1304395297 learned_addr learned my addr 172.21.15.6:0/1304395297 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:08.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.353+0000 7f21c6a00640 1 -- 172.21.15.6:0/1304395297 >> v2:172.21.15.92:3300/0 conn(0x7f21c80ce520 msgr2=0x7f21c806c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:08.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.353+0000 7f21c6a00640 1 --2- 172.21.15.6:0/1304395297 >> v2:172.21.15.92:3300/0 conn(0x7f21c80ce520 0x7f21c806c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.353+0000 7f21c6a00640 1 -- 172.21.15.6:0/1304395297 >> v1:172.21.15.92:6789/0 conn(0x7f21c81549a0 legacy=0x7f21c80cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.354+0000 7f21c6a00640 1 -- 172.21.15.6:0/1304395297 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c815a650 con 0x7f21c80cea70 2024-03-22T13:12:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.354+0000 7f21c7e00640 1 --2- 172.21.15.6:0/1304395297 >> v2:172.21.15.92:3300/0 conn(0x7f21c80ce520 0x7f21c806c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:12:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.356+0000 7f21c6a00640 1 --2- 172.21.15.6:0/1304395297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c80cea70 0x7f21c815b130 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009870 tx=0x7f21b402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55be854cc74a4dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.361+0000 7f21c6000640 1 -- 172.21.15.6:0/1304395297 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b402f840 con 0x7f21c80cea70 2024-03-22T13:12:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.361+0000 7f21c6000640 1 -- 172.21.15.6:0/1304395297 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b402f9e0 con 0x7f21c80cea70 2024-03-22T13:12:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.361+0000 7f21c6000640 1 -- 172.21.15.6:0/1304395297 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b402fdb0 con 0x7f21c80cea70 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 -- 172.21.15.6:0/1304395297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c80cea70 msgr2=0x7f21c815b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 --2- 172.21.15.6:0/1304395297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c80cea70 0x7f21c815b130 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009870 tx=0x7f21b402ba20 comp rx=0 tx=0).stop 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 -- 172.21.15.6:0/1304395297 shutdown_connections 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 --2- 172.21.15.6:0/1304395297 >> v2:172.21.15.92:3300/0 conn(0x7f21c80ce520 0x7f21c806c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 --2- 172.21.15.6:0/1304395297 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c80cea70 0x7f21c815b130 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 -- 172.21.15.6:0/1304395297 >> 172.21.15.6:0/1304395297 conn(0x7f21c80b1700 msgr2=0x7f21c80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 -- 172.21.15.6:0/1304395297 shutdown_connections 2024-03-22T13:12:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.362+0000 7f21ce400640 1 -- 172.21.15.6:0/1304395297 wait complete. 2024-03-22T13:12:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.364+0000 7f21ce400640 1 Processor -- start 2024-03-22T13:12:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.364+0000 7f21ce400640 1 -- start start 2024-03-22T13:12:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.365+0000 7f21ce400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c80ce520 0x7f21c814def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.365+0000 7f21ce400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c80cea70 0x7f21c814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.365+0000 7f21ce400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c81549a0 0x7f21c80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.365+0000 7f21ce400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c80cd430 con 0x7f21c80cea70 2024-03-22T13:12:08.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.365+0000 7f21ce400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c80ccf70 con 0x7f21c80ce520 2024-03-22T13:12:08.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.365+0000 7f21ce400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c80cd5b0 con 0x7f21c81549a0 2024-03-22T13:12:08.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.366+0000 7f21c6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c80cea70 0x7f21c814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.366+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c81549a0 0x7f21c80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.366+0000 7f21c7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c80ce520 0x7f21c814def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.366+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c81549a0 0x7f21c80b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54298/0 (socket says 172.21.15.6:54298) 2024-03-22T13:12:08.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.366+0000 7f21c7e00640 1 -- 172.21.15.6:0/1892672791 learned_addr learned my addr 172.21.15.6:0/1892672791 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.369+0000 7f21c7e00640 1 -- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c80ce520 msgr2=0x7f21c814def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.369+0000 7f21c7e00640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c80ce520 0x7f21c814def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.369+0000 7f21c7e00640 1 -- 172.21.15.6:0/1892672791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c80cea70 msgr2=0x7f21c814e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.370+0000 7f21c7e00640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21c80cea70 0x7f21c814e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.370+0000 7f21c7e00640 1 -- 172.21.15.6:0/1892672791 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c81527f0 con 0x7f21c81549a0 2024-03-22T13:12:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.372+0000 7f21c7e00640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c81549a0 0x7f21c80b2e90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009e60 tx=0x7f21c000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.376+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0018030 con 0x7f21c81549a0 2024-03-22T13:12:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.376+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c000cdf0 con 0x7f21c81549a0 2024-03-22T13:12:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.376+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0003bb0 con 0x7f21c81549a0 2024-03-22T13:12:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.376+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c81f0c50 con 0x7f21c81549a0 2024-03-22T13:12:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.377+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c80b5590 con 0x7f21c81549a0 2024-03-22T13:12:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.377+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21c80757d0 con 0x7f21c81549a0 2024-03-22T13:12:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.410+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f21c0005d10 con 0x7f21c81549a0 2024-03-22T13:12:08.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.411+0000 7f21bf400640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a806bc50 0x7f21a806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.412+0000 7f21c7400640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a806bc50 0x7f21a806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.413+0000 7f21c7400640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a806bc50 0x7f21a806e110 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f21b8001780 tx=0x7f21b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.414+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f21c0058310 con 0x7f21c81549a0 2024-03-22T13:12:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.414+0000 7f21bf400640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7f21a80757f0 0x7f21a8077c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.414+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 --> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21c0009a50 con 0x7f21a80757f0 2024-03-22T13:12:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.414+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21c0056510 con 0x7f21c81549a0 2024-03-22T13:12:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.415+0000 7f21c6a00640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7f21a80757f0 0x7f21a8077c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.415+0000 7f21c6a00640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7f21a80757f0 0x7f21a8077c90 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.417+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== osd.0 v2:172.21.15.6:6800/2278862740 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21c0009a50 con 0x7f21a80757f0 2024-03-22T13:12:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.434+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 --> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f21c80b5b50 con 0x7f21a80757f0 2024-03-22T13:12:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.434+0000 7f21bf400640 1 -- 172.21.15.6:0/1892672791 <== osd.0 v2:172.21.15.6:6800/2278862740 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f21c80b5b50 con 0x7f21a80757f0 2024-03-22T13:12:08.434 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:08.434 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:08.435 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.435+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7f21a80757f0 msgr2=0x7f21a8077c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.435+0000 7f21ce400640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.6:6800/2278862740,v1:172.21.15.6:6801/2278862740] conn(0x7f21a80757f0 0x7f21a8077c90 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.435+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a806bc50 msgr2=0x7f21a806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.435+0000 7f21ce400640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f21a806bc50 0x7f21a806e110 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f21b8001780 tx=0x7f21b8009040 comp rx=0 tx=0).stop 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.435+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c81549a0 msgr2=0x7f21c80b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.435+0000 7f21ce400640 1 --2- 172.21.15.6:0/1892672791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f21c81549a0 0x7f21c80b2e90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009e60 tx=0x7f21c000c490 comp rx=0 tx=0).stop 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.435+0000 7f21c7e00640 1 -- 172.21.15.6:0/1892672791 reap_dead start 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.436+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 shutdown_connections 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.436+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 >> 172.21.15.6:0/1892672791 conn(0x7f21c80b1700 msgr2=0x7f21c80af670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.436+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 shutdown_connections 2024-03-22T13:12:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.436+0000 7f21ce400640 1 -- 172.21.15.6:0/1892672791 wait complete. 2024-03-22T13:12:08.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-22T13:12:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.635+0000 7fb012200640 1 Processor -- start 2024-03-22T13:12:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.635+0000 7fb012200640 1 -- start start 2024-03-22T13:12:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.636+0000 7fb012200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb00c06c200 0x7fb00c15b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.636+0000 7fb012200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c0d0110 0x7fb00c06c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.637+0000 7fb012200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c0cdbb0 con 0x7fb00c0cef30 2024-03-22T13:12:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.637+0000 7fb012200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c0ceaa0 con 0x7fb00c06c200 2024-03-22T13:12:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.637+0000 7fb012200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0cde90 con 0x7fb00c0d0110 2024-03-22T13:12:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.683+0000 7fb00be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c0d0110 0x7fb00c06c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.683+0000 7fb00be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c0d0110 0x7fb00c06c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54310/0 (socket says 172.21.15.6:54310) 2024-03-22T13:12:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.683+0000 7fb00be00640 1 -- 172.21.15.6:0/4130778400 learned_addr learned my addr 172.21.15.6:0/4130778400 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.687+0000 7fb00be00640 1 -- 172.21.15.6:0/4130778400 >> v2:172.21.15.6:3300/0 conn(0x7fb00c06c200 msgr2=0x7fb00c15b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.687+0000 7fb00be00640 1 --2- 172.21.15.6:0/4130778400 >> v2:172.21.15.6:3300/0 conn(0x7fb00c06c200 0x7fb00c15b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.687+0000 7fb00be00640 1 -- 172.21.15.6:0/4130778400 >> v1:172.21.15.6:6789/0 conn(0x7fb00c0cef30 legacy=0x7fb00c15d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.687+0000 7fb00be00640 1 -- 172.21.15.6:0/4130778400 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c0ce170 con 0x7fb00c0d0110 2024-03-22T13:12:08.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.691+0000 7fb00be00640 1 --2- 172.21.15.6:0/4130778400 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c0d0110 0x7fb00c06c5e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faffc009870 tx=0x7faffc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d174c9d6cf0ec47d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.695+0000 7fb00b400640 1 -- 172.21.15.6:0/4130778400 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc034070 con 0x7fb00c0d0110 2024-03-22T13:12:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.695+0000 7fb00b400640 1 -- 172.21.15.6:0/4130778400 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc030840 con 0x7fb00c0d0110 2024-03-22T13:12:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.695+0000 7fb00b400640 1 -- 172.21.15.6:0/4130778400 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc030c10 con 0x7fb00c0d0110 2024-03-22T13:12:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 -- 172.21.15.6:0/4130778400 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c0d0110 msgr2=0x7fb00c06c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 --2- 172.21.15.6:0/4130778400 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c0d0110 0x7fb00c06c5e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faffc009870 tx=0x7faffc02b980 comp rx=0 tx=0).stop 2024-03-22T13:12:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 -- 172.21.15.6:0/4130778400 shutdown_connections 2024-03-22T13:12:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 --2- 172.21.15.6:0/4130778400 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c0d0110 0x7fb00c06c5e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 --2- 172.21.15.6:0/4130778400 >> v2:172.21.15.6:3300/0 conn(0x7fb00c06c200 0x7fb00c15b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 -- 172.21.15.6:0/4130778400 >> 172.21.15.6:0/4130778400 conn(0x7fb00c0b1b60 msgr2=0x7fb00c0cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 -- 172.21.15.6:0/4130778400 shutdown_connections 2024-03-22T13:12:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.697+0000 7fb012200640 1 -- 172.21.15.6:0/4130778400 wait complete. 2024-03-22T13:12:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.699+0000 7fb012200640 1 Processor -- start 2024-03-22T13:12:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.700+0000 7fb012200640 1 -- start start 2024-03-22T13:12:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.701+0000 7fb012200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c06c200 0x7fb00c15a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.701+0000 7fb012200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb00c0cef30 0x7fb00c15c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.701+0000 7fb012200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb00c0d0110 0x7fb00c15cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.701+0000 7fb012200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c154a80 con 0x7fb00c0d0110 2024-03-22T13:12:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.701+0000 7fb012200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0ce730 con 0x7fb00c0cef30 2024-03-22T13:12:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.701+0000 7fb012200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0cd800 con 0x7fb00c06c200 2024-03-22T13:12:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.701+0000 7fb010e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c06c200 0x7fb00c15a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.702+0000 7fb010e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c06c200 0x7fb00c15a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54316/0 (socket says 172.21.15.6:54316) 2024-03-22T13:12:08.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.702+0000 7fb010e00640 1 -- 172.21.15.6:0/1725376991 learned_addr learned my addr 172.21.15.6:0/1725376991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:08.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.702+0000 7fb00be00640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb00c0cef30 0x7fb00c15c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.706+0000 7fb010e00640 1 -- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb00c0cef30 msgr2=0x7fb00c15c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.706+0000 7fb010e00640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb00c0cef30 0x7fb00c15c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.706+0000 7fb010e00640 1 -- 172.21.15.6:0/1725376991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb00c0d0110 msgr2=0x7fb00c15cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.706+0000 7fb010e00640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb00c0d0110 0x7fb00c15cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.706+0000 7fb010e00640 1 -- 172.21.15.6:0/1725376991 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c1f07b0 con 0x7fb00c06c200 2024-03-22T13:12:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.709+0000 7fb010e00640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c06c200 0x7fb00c15a620 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b440 tx=0x7fb00000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.716+0000 7fb008c00640 1 -- 172.21.15.6:0/1725376991 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000015020 con 0x7fb00c06c200 2024-03-22T13:12:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.716+0000 7fb008c00640 1 -- 172.21.15.6:0/1725376991 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb000011890 con 0x7fb00c06c200 2024-03-22T13:12:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.716+0000 7fb008c00640 1 -- 172.21.15.6:0/1725376991 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000011c80 con 0x7fb00c06c200 2024-03-22T13:12:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.716+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c1f0a90 con 0x7fb00c06c200 2024-03-22T13:12:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.716+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c1f1050 con 0x7fb00c06c200 2024-03-22T13:12:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.717+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c075650 con 0x7fb00c06c200 2024-03-22T13:12:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.748+0000 7fb008c00640 1 -- 172.21.15.6:0/1725376991 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb000018020 con 0x7fb00c06c200 2024-03-22T13:12:08.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.750+0000 7fb008c00640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7faff006bbd0 0x7faff006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:08.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.750+0000 7fb00be00640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7faff006bbd0 0x7faff006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:08.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.752+0000 7fb00be00640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7faff006bbd0 0x7faff006e090 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7faffc036000 tx=0x7faffc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:08.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:08.753+0000 7fb008c00640 1 -- 172.21.15.6:0/1725376991 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb000057620 con 0x7fb00c06c200 2024-03-22T13:12:09.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.024+0000 7fb008c00640 1 -- 172.21.15.6:0/1725376991 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb00001f020 con 0x7fb00c06c200 2024-03-22T13:12:09.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.235+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb00c1f3b80 con 0x7faff006bbd0 2024-03-22T13:12:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.236+0000 7fb008c00640 1 -- 172.21.15.6:0/1725376991 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb00c1f3b80 con 0x7faff006bbd0 2024-03-22T13:12:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-03-22T13:12:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7faff006bbd0 msgr2=0x7faff006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7faff006bbd0 0x7faff006e090 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7faffc036000 tx=0x7faffc008030 comp rx=0 tx=0).stop 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c06c200 msgr2=0x7fb00c15a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c06c200 0x7fb00c15a620 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b440 tx=0x7fb00000b900 comp rx=0 tx=0).stop 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 shutdown_connections 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb00c0d0110 0x7fb00c15cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7faff006bbd0 0x7faff006e090 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb00c0cef30 0x7fb00c15c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 --2- 172.21.15.6:0/1725376991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb00c06c200 0x7fb00c15a620 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 >> 172.21.15.6:0/1725376991 conn(0x7fb00c0b1b60 msgr2=0x7fb00c1e56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.238+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 shutdown_connections 2024-03-22T13:12:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.239+0000 7fb012200640 1 -- 172.21.15.6:0/1725376991 wait complete. 2024-03-22T13:12:09.255 INFO:tasks.ceph:Scrubbing osd.1 2024-03-22T13:12:09.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:09.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.443+0000 7fb743a00640 1 Processor -- start 2024-03-22T13:12:09.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.443+0000 7fb743a00640 1 -- start start 2024-03-22T13:12:09.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.444+0000 7fb743a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb73c1549a0 0x7fb73c15b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.445+0000 7fb743a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c06c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.445+0000 7fb743a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c0cc9b0 con 0x7fb73c0ce520 2024-03-22T13:12:09.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.445+0000 7fb743a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73c15aca0 con 0x7fb73c1549a0 2024-03-22T13:12:09.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.445+0000 7fb743a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0ccc90 con 0x7fb73c0cea70 2024-03-22T13:12:09.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.446+0000 7fb73be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c06c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.446+0000 7fb741000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb73c1549a0 0x7fb73c15b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.446+0000 7fb73be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c06c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54332/0 (socket says 172.21.15.6:54332) 2024-03-22T13:12:09.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.446+0000 7fb73be00640 1 -- 172.21.15.6:0/2659466267 learned_addr learned my addr 172.21.15.6:0/2659466267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.449+0000 7fb73be00640 1 -- 172.21.15.6:0/2659466267 >> v2:172.21.15.6:3300/0 conn(0x7fb73c1549a0 msgr2=0x7fb73c15b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.449+0000 7fb73be00640 1 --2- 172.21.15.6:0/2659466267 >> v2:172.21.15.6:3300/0 conn(0x7fb73c1549a0 0x7fb73c15b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.449+0000 7fb73be00640 1 -- 172.21.15.6:0/2659466267 >> v1:172.21.15.6:6789/0 conn(0x7fb73c0ce520 legacy=0x7fb73c0cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.449+0000 7fb73be00640 1 -- 172.21.15.6:0/2659466267 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c15a650 con 0x7fb73c0cea70 2024-03-22T13:12:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.453+0000 7fb73be00640 1 --2- 172.21.15.6:0/2659466267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c06c6c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb724009870 tx=0x7fb72402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5707f5963b2119d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:09.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.458+0000 7fb73b400640 1 -- 172.21.15.6:0/2659466267 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72402f840 con 0x7fb73c0cea70 2024-03-22T13:12:09.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.458+0000 7fb73b400640 1 -- 172.21.15.6:0/2659466267 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72402f9e0 con 0x7fb73c0cea70 2024-03-22T13:12:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.458+0000 7fb73b400640 1 -- 172.21.15.6:0/2659466267 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72402fdb0 con 0x7fb73c0cea70 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 -- 172.21.15.6:0/2659466267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 msgr2=0x7fb73c06c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 --2- 172.21.15.6:0/2659466267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c06c6c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb724009870 tx=0x7fb72402ba20 comp rx=0 tx=0).stop 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 -- 172.21.15.6:0/2659466267 shutdown_connections 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 --2- 172.21.15.6:0/2659466267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c06c6c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 --2- 172.21.15.6:0/2659466267 >> v2:172.21.15.6:3300/0 conn(0x7fb73c1549a0 0x7fb73c15b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 -- 172.21.15.6:0/2659466267 >> 172.21.15.6:0/2659466267 conn(0x7fb73c0b1700 msgr2=0x7fb73c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 -- 172.21.15.6:0/2659466267 shutdown_connections 2024-03-22T13:12:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.459+0000 7fb743a00640 1 -- 172.21.15.6:0/2659466267 wait complete. 2024-03-22T13:12:09.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.462+0000 7fb743a00640 1 Processor -- start 2024-03-22T13:12:09.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.462+0000 7fb743a00640 1 -- start start 2024-03-22T13:12:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.463+0000 7fb743a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb73c0ce520 0x7fb73c150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.463+0000 7fb743a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.463+0000 7fb743a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb73c1549a0 0x7fb73c1e57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.463+0000 7fb743a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0cd430 con 0x7fb73c0ce520 2024-03-22T13:12:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.463+0000 7fb743a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0ccf70 con 0x7fb73c1549a0 2024-03-22T13:12:09.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.463+0000 7fb743a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0cd5b0 con 0x7fb73c0cea70 2024-03-22T13:12:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.464+0000 7fb73be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.464+0000 7fb741a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb73c1549a0 0x7fb73c1e57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.464+0000 7fb741000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb73c0ce520 0x7fb73c150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.464+0000 7fb73be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c0c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54338/0 (socket says 172.21.15.6:54338) 2024-03-22T13:12:09.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.464+0000 7fb73be00640 1 -- 172.21.15.6:0/449419860 learned_addr learned my addr 172.21.15.6:0/449419860 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.468+0000 7fb741000640 1 -- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 msgr2=0x7fb73c0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.468+0000 7fb741000640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb73c0cea70 0x7fb73c0c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.468+0000 7fb741000640 1 -- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb73c1549a0 msgr2=0x7fb73c1e57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.468+0000 7fb741000640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb73c1549a0 0x7fb73c1e57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.468+0000 7fb741000640 1 -- 172.21.15.6:0/449419860 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c0c4ae0 con 0x7fb73c0ce520 2024-03-22T13:12:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.471+0000 7fb741000640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb73c0ce520 0x7fb73c150410 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009db0 tx=0x7fb72c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:09.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.477+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c018020 con 0x7fb73c0ce520 2024-03-22T13:12:09.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.477+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c014480 con 0x7fb73c0ce520 2024-03-22T13:12:09.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.477+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c1f1c40 con 0x7fb73c0ce520 2024-03-22T13:12:09.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.477+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c014850 con 0x7fb73c0ce520 2024-03-22T13:12:09.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.477+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c1f2200 con 0x7fb73c0ce520 2024-03-22T13:12:09.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.478+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb73c0757d0 con 0x7fb73c0ce520 2024-03-22T13:12:09.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.506+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb72c005d10 con 0x7fb73c0ce520 2024-03-22T13:12:09.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.508+0000 7fb738c00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb71806bc50 0x7fb71806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.508+0000 7fb73be00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb71806bc50 0x7fb71806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.509+0000 7fb73be00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb71806bc50 0x7fb71806e110 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb724009870 tx=0x7fb724008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:09.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.510+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb72c0585c0 con 0x7fb73c0ce520 2024-03-22T13:12:09.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.510+0000 7fb738c00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7fb7180757f0 0x7fb718077c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.510+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 --> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb72c00be50 con 0x7fb7180757f0 2024-03-22T13:12:09.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.510+0000 7fb741a00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7fb7180757f0 0x7fb718077c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.511+0000 7fb741a00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7fb7180757f0 0x7fb718077c90 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:09.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.512+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== osd.1 v2:172.21.15.6:6807/3168070181 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb72c00be50 con 0x7fb7180757f0 2024-03-22T13:12:09.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.512+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb72c0567c0 con 0x7fb73c0ce520 2024-03-22T13:12:09.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.533+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 --> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb73c074c10 con 0x7fb7180757f0 2024-03-22T13:12:09.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.533+0000 7fb738c00640 1 -- 172.21.15.6:0/449419860 <== osd.1 v2:172.21.15.6:6807/3168070181 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb73c074c10 con 0x7fb7180757f0 2024-03-22T13:12:09.533 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:09.533 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:09.533 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:09.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.534+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7fb7180757f0 msgr2=0x7fb718077c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.534+0000 7fb743a00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:6807/3168070181,v1:172.21.15.6:6808/3168070181] conn(0x7fb7180757f0 0x7fb718077c90 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.534+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb71806bc50 msgr2=0x7fb71806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.534+0000 7fb743a00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb71806bc50 0x7fb71806e110 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb724009870 tx=0x7fb724008030 comp rx=0 tx=0).stop 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.534+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb73c0ce520 msgr2=0x7fb73c150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.534+0000 7fb743a00640 1 --2- 172.21.15.6:0/449419860 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb73c0ce520 0x7fb73c150410 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009db0 tx=0x7fb72c00c490 comp rx=0 tx=0).stop 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.534+0000 7fb741a00640 1 -- 172.21.15.6:0/449419860 reap_dead start 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.535+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 shutdown_connections 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.535+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 >> 172.21.15.6:0/449419860 conn(0x7fb73c0b1700 msgr2=0x7fb73c0cef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.535+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 shutdown_connections 2024-03-22T13:12:09.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.535+0000 7fb743a00640 1 -- 172.21.15.6:0/449419860 wait complete. 2024-03-22T13:12:09.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-22T13:12:09.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.736+0000 7f62fe400640 1 Processor -- start 2024-03-22T13:12:09.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.736+0000 7f62fe400640 1 -- start start 2024-03-22T13:12:09.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.738+0000 7f62fe400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f62f8152280 0x7f62f8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.738+0000 7f62fe400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8159070 con 0x7f62f8158c20 2024-03-22T13:12:09.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.738+0000 7f62fe400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f80c1450 con 0x7f62f80c0ea0 2024-03-22T13:12:09.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.738+0000 7f62fe400640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f8158790 con 0x7f62f8152280 2024-03-22T13:12:09.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.738+0000 7f62f7400640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f62f8152280 0x7f62f8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.739+0000 7f62f7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62f8158c20 0x7f62f80c65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60174/0 (socket says 172.21.15.6:60174) 2024-03-22T13:12:09.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.739+0000 7f62f7e00640 1 -- 172.21.15.6:0/3590306900 learned_addr learned my addr 172.21.15.6:0/3590306900 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:09.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.742+0000 7f62f6a00640 1 -- 172.21.15.6:0/3590306900 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960786709 0 0) 0x7f62f8159070 con 0x7f62f8158c20 2024-03-22T13:12:09.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.742+0000 7f62f6a00640 1 -- 172.21.15.6:0/3590306900 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62dc003520 con 0x7f62f8158c20 2024-03-22T13:12:09.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.745+0000 7f62f7400640 1 -- 172.21.15.6:0/3590306900 >> v1:172.21.15.92:6789/0 conn(0x7f62f80c0ea0 legacy=0x7f62f806b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.745+0000 7f62f7400640 1 -- 172.21.15.6:0/3590306900 >> v1:172.21.15.6:6789/0 conn(0x7f62f8158c20 legacy=0x7f62f80c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.745+0000 7f62f7400640 1 -- 172.21.15.6:0/3590306900 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f80c1730 con 0x7f62f8152280 2024-03-22T13:12:09.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.750+0000 7f62f7400640 1 --2- 172.21.15.6:0/3590306900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8071920 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f62e80097a0 tx=0x7f62e802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fc508a8c263c090 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.755+0000 7f62f6a00640 1 -- 172.21.15.6:0/3590306900 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8035020 con 0x7f62f8152280 2024-03-22T13:12:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.755+0000 7f62f6a00640 1 -- 172.21.15.6:0/3590306900 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e8031830 con 0x7f62f8152280 2024-03-22T13:12:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.755+0000 7f62f6a00640 1 -- 172.21.15.6:0/3590306900 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8031c00 con 0x7f62f8152280 2024-03-22T13:12:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.756+0000 7f62fe400640 1 -- 172.21.15.6:0/3590306900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 msgr2=0x7f62f8071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.756+0000 7f62fe400640 1 --2- 172.21.15.6:0/3590306900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8071920 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f62e80097a0 tx=0x7f62e802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.756+0000 7f62fe400640 1 -- 172.21.15.6:0/3590306900 shutdown_connections 2024-03-22T13:12:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.756+0000 7f62fe400640 1 --2- 172.21.15.6:0/3590306900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8071920 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.756+0000 7f62fe400640 1 -- 172.21.15.6:0/3590306900 >> 172.21.15.6:0/3590306900 conn(0x7f62f806e310 msgr2=0x7f62f806f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.756+0000 7f62fe400640 1 -- 172.21.15.6:0/3590306900 shutdown_connections 2024-03-22T13:12:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.756+0000 7f62fe400640 1 -- 172.21.15.6:0/3590306900 wait complete. 2024-03-22T13:12:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.758+0000 7f62fe400640 1 Processor -- start 2024-03-22T13:12:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.758+0000 7f62fe400640 1 -- start start 2024-03-22T13:12:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.759+0000 7f62fe400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62f80c0ea0 0x7f62f80cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.759+0000 7f62fe400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.759+0000 7f62fe400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62f8158c20 0x7f62f81598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.759+0000 7f62fe400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f81580d0 con 0x7f62f8158c20 2024-03-22T13:12:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.759+0000 7f62fe400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8158420 con 0x7f62f8152280 2024-03-22T13:12:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.759+0000 7f62fe400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8152660 con 0x7f62f80c0ea0 2024-03-22T13:12:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.760+0000 7f62efe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.760+0000 7f62f7400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62f80c0ea0 0x7f62f80cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.760+0000 7f62f7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62f8158c20 0x7f62f81598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.760+0000 7f62efe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36922/0 (socket says 172.21.15.6:36922) 2024-03-22T13:12:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.760+0000 7f62efe00640 1 -- 172.21.15.6:0/1426602779 learned_addr learned my addr 172.21.15.6:0/1426602779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.765+0000 7f62efe00640 1 -- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62f80c0ea0 msgr2=0x7f62f80cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.765+0000 7f62efe00640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62f80c0ea0 0x7f62f80cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.765+0000 7f62efe00640 1 -- 172.21.15.6:0/1426602779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62f8158c20 msgr2=0x7f62f81598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.765+0000 7f62efe00640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62f8158c20 0x7f62f81598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:09.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.765+0000 7f62efe00640 1 -- 172.21.15.6:0/1426602779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8147480 con 0x7f62f8152280 2024-03-22T13:12:09.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.769+0000 7f62efe00640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8159390 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f62e000c4d0 tx=0x7f62e000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:09.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.773+0000 7f62ef400640 1 -- 172.21.15.6:0/1426602779 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e00052d0 con 0x7f62f8152280 2024-03-22T13:12:09.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.773+0000 7f62ef400640 1 -- 172.21.15.6:0/1426602779 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e0005470 con 0x7f62f8152280 2024-03-22T13:12:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.773+0000 7f62ef400640 1 -- 172.21.15.6:0/1426602779 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0012850 con 0x7f62f8152280 2024-03-22T13:12:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.773+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f80c4cb0 con 0x7f62f8152280 2024-03-22T13:12:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.774+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f81479d0 con 0x7f62f8152280 2024-03-22T13:12:09.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.775+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f80757d0 con 0x7f62f8152280 2024-03-22T13:12:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.806+0000 7f62ef400640 1 -- 172.21.15.6:0/1426602779 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f62e00129f0 con 0x7f62f8152280 2024-03-22T13:12:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.808+0000 7f62ef400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f62d406bc50 0x7f62d406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.808+0000 7f62f7400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f62d406bc50 0x7f62d406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:09.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:09.809+0000 7f62f7400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f62d406bc50 0x7f62d406e110 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f62e8036000 tx=0x7f62e8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.050+0000 7f62ef400640 1 -- 172.21.15.6:0/1426602779 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f62e0057500 con 0x7f62f8152280 2024-03-22T13:12:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.068+0000 7f62ef400640 1 -- 172.21.15.6:0/1426602779 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f62e0059460 con 0x7f62f8152280 2024-03-22T13:12:10.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.289+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f62f80c4a50 con 0x7f62d406bc50 2024-03-22T13:12:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.290+0000 7f62ef400640 1 -- 172.21.15.6:0/1426602779 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f62e8031800 con 0x7f62d406bc50 2024-03-22T13:12:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f62d406bc50 msgr2=0x7f62d406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f62d406bc50 0x7f62d406e110 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f62e8036000 tx=0x7f62e8008030 comp rx=0 tx=0).stop 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 msgr2=0x7f62f8159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8159390 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f62e000c4d0 tx=0x7f62e000c990 comp rx=0 tx=0).stop 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 shutdown_connections 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62f8158c20 0x7f62f81598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f62d406bc50 0x7f62d406e110 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f8152280 0x7f62f8159390 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 --2- 172.21.15.6:0/1426602779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62f80c0ea0 0x7f62f80cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 >> 172.21.15.6:0/1426602779 conn(0x7f62f806e310 msgr2=0x7f62f80c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 shutdown_connections 2024-03-22T13:12:10.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.295+0000 7f62fe400640 1 -- 172.21.15.6:0/1426602779 wait complete. 2024-03-22T13:12:10.313 INFO:tasks.ceph:Scrubbing osd.2 2024-03-22T13:12:10.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:10.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.502+0000 7feefa400640 1 Processor -- start 2024-03-22T13:12:10.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.502+0000 7feefa400640 1 -- start start 2024-03-22T13:12:10.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.504+0000 7feefa400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef40c7a30 0x7feef406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.504+0000 7feefa400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef415da40 con 0x7feef4154220 2024-03-22T13:12:10.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.504+0000 7feefa400640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef415dce0 con 0x7feef406c850 2024-03-22T13:12:10.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.504+0000 7feefa400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef40c7e70 con 0x7feef40c7a30 2024-03-22T13:12:10.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.505+0000 7feef2a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef40c7a30 0x7feef406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.505+0000 7feef3e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7feef406c850 0x7feef40c8df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43808/0 (socket says 172.21.15.6:43808) 2024-03-22T13:12:10.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.505+0000 7feef3e00640 1 -- 172.21.15.6:0/1676955305 learned_addr learned my addr 172.21.15.6:0/1676955305 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.508+0000 7feef2a00640 1 -- 172.21.15.6:0/1676955305 >> v1:172.21.15.92:6789/0 conn(0x7feef406c850 legacy=0x7feef40c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.508+0000 7feef2a00640 1 -- 172.21.15.6:0/1676955305 >> v1:172.21.15.6:6789/0 conn(0x7feef4154220 legacy=0x7feef415ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.508+0000 7feef2000640 1 -- 172.21.15.6:0/1676955305 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716124150 0 0) 0x7feef415da40 con 0x7feef4154220 2024-03-22T13:12:10.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.509+0000 7feef2a00640 1 -- 172.21.15.6:0/1676955305 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef415dfc0 con 0x7feef40c7a30 2024-03-22T13:12:10.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.511+0000 7feef2a00640 1 --2- 172.21.15.6:0/1676955305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef40c7a30 0x7feef406c310 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feee0009750 tx=0x7feee002b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e18290387f8e5349 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.516+0000 7feef2000640 1 -- 172.21.15.6:0/1676955305 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0037070 con 0x7feef40c7a30 2024-03-22T13:12:10.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.516+0000 7feef2000640 1 -- 172.21.15.6:0/1676955305 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee0030710 con 0x7feef40c7a30 2024-03-22T13:12:10.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.516+0000 7feef2000640 1 -- 172.21.15.6:0/1676955305 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0030ae0 con 0x7feef40c7a30 2024-03-22T13:12:10.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.517+0000 7feefa400640 1 -- 172.21.15.6:0/1676955305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef40c7a30 msgr2=0x7feef406c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.517+0000 7feefa400640 1 --2- 172.21.15.6:0/1676955305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef40c7a30 0x7feef406c310 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feee0009750 tx=0x7feee002b730 comp rx=0 tx=0).stop 2024-03-22T13:12:10.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.517+0000 7feefa400640 1 -- 172.21.15.6:0/1676955305 shutdown_connections 2024-03-22T13:12:10.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.517+0000 7feefa400640 1 --2- 172.21.15.6:0/1676955305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef40c7a30 0x7feef406c310 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.517+0000 7feefa400640 1 -- 172.21.15.6:0/1676955305 >> 172.21.15.6:0/1676955305 conn(0x7feef40727e0 msgr2=0x7feef4072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:10.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.517+0000 7feefa400640 1 -- 172.21.15.6:0/1676955305 shutdown_connections 2024-03-22T13:12:10.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.517+0000 7feefa400640 1 -- 172.21.15.6:0/1676955305 wait complete. 2024-03-22T13:12:10.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.520+0000 7feefa400640 1 Processor -- start 2024-03-22T13:12:10.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.520+0000 7feefa400640 1 -- start start 2024-03-22T13:12:10.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.522+0000 7feefa400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef406c850 0x7feef40cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.522+0000 7feefa400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feef40c7a30 0x7feef40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.522+0000 7feefa400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feef4154220 0x7feef415c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.522+0000 7feefa400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef406cfc0 con 0x7feef40c7a30 2024-03-22T13:12:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.522+0000 7feefa400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef415e580 con 0x7feef4154220 2024-03-22T13:12:10.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.522+0000 7feefa400640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef4154600 con 0x7feef406c850 2024-03-22T13:12:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.522+0000 7feef3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef406c850 0x7feef40cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.523+0000 7feef3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feef4154220 0x7feef415c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.523+0000 7feef2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feef40c7a30 0x7feef40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.523+0000 7feef3400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef406c850 0x7feef40cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54360/0 (socket says 172.21.15.6:54360) 2024-03-22T13:12:10.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.523+0000 7feef3400640 1 -- 172.21.15.6:0/64391384 learned_addr learned my addr 172.21.15.6:0/64391384 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:10.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.527+0000 7feef3e00640 1 -- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef406c850 msgr2=0x7feef40cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.527+0000 7feef3e00640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feef406c850 0x7feef40cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.527+0000 7feef3e00640 1 -- 172.21.15.6:0/64391384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feef40c7a30 msgr2=0x7feef40ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.527+0000 7feef3e00640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feef40c7a30 0x7feef40ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.527+0000 7feef3e00640 1 -- 172.21.15.6:0/64391384 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef40cd490 con 0x7feef4154220 2024-03-22T13:12:10.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.529+0000 7feef3e00640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feef4154220 0x7feef415c110 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feeec00c820 tx=0x7feeec00cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.534+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec018070 con 0x7feef4154220 2024-03-22T13:12:10.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.534+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec00a640 con 0x7feef4154220 2024-03-22T13:12:10.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.534+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec00aa30 con 0x7feef4154220 2024-03-22T13:12:10.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.534+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef40cd770 con 0x7feef4154220 2024-03-22T13:12:10.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.534+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef415b910 con 0x7feef4154220 2024-03-22T13:12:10.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.535+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feef40757d0 con 0x7feef4154220 2024-03-22T13:12:10.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.562+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feeec01b070 con 0x7feef4154220 2024-03-22T13:12:10.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.563+0000 7feeeb400640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feed406bc20 0x7feed406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.563+0000 7feef3400640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feed406bc20 0x7feed406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.564+0000 7feef3400640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feed406bc20 0x7feed406e0e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7feee4001120 tx=0x7feee4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.564+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7feeec056d10 con 0x7feef4154220 2024-03-22T13:12:10.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.564+0000 7feeeb400640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7feed40757c0 0x7feed4077c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.564+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 --> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feeec014e50 con 0x7feed40757c0 2024-03-22T13:12:10.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.565+0000 7feef2a00640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7feed40757c0 0x7feed4077c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.565+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeec054f10 con 0x7feef4154220 2024-03-22T13:12:10.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.566+0000 7feef2a00640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7feed40757c0 0x7feed4077c60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.567+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== osd.2 v2:172.21.15.6:6813/2211174308 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7feeec014e50 con 0x7feed40757c0 2024-03-22T13:12:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.584+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 --> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feef415b440 con 0x7feed40757c0 2024-03-22T13:12:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.585+0000 7feeeb400640 1 -- 172.21.15.6:0/64391384 <== osd.2 v2:172.21.15.6:6813/2211174308 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7feef415b440 con 0x7feed40757c0 2024-03-22T13:12:10.585 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:10.585 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:10.585 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.585+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7feed40757c0 msgr2=0x7feed4077c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.585+0000 7feefa400640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.6:6813/2211174308,v1:172.21.15.6:6816/2211174308] conn(0x7feed40757c0 0x7feed4077c60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feed406bc20 msgr2=0x7feed406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feed406bc20 0x7feed406e0e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7feee4001120 tx=0x7feee4009040 comp rx=0 tx=0).stop 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feef4154220 msgr2=0x7feef415c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 --2- 172.21.15.6:0/64391384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feef4154220 0x7feef415c110 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feeec00c820 tx=0x7feeec00cce0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feef3e00640 1 -- 172.21.15.6:0/64391384 reap_dead start 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 shutdown_connections 2024-03-22T13:12:10.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 >> 172.21.15.6:0/64391384 conn(0x7feef40727e0 msgr2=0x7feef4072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 shutdown_connections 2024-03-22T13:12:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.586+0000 7feefa400640 1 -- 172.21.15.6:0/64391384 wait complete. 2024-03-22T13:12:10.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-22T13:12:10.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.806+0000 7f0667800640 1 Processor -- start 2024-03-22T13:12:10.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.806+0000 7f0667800640 1 -- start start 2024-03-22T13:12:10.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.807+0000 7f0667800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601548c0 0x7f06600c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.808+0000 7f0667800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0660073520 con 0x7f06600c80a0 2024-03-22T13:12:10.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.808+0000 7f0667800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066015a550 con 0x7f06601544e0 2024-03-22T13:12:10.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.808+0000 7f0667800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066015a830 con 0x7f06601548c0 2024-03-22T13:12:10.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.808+0000 7f065fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601548c0 0x7f06600c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.809+0000 7f0664e00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f06601544e0 0x7f066015d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43824/0 (socket says 172.21.15.6:43824) 2024-03-22T13:12:10.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.809+0000 7f0664e00640 1 -- 172.21.15.6:0/794792049 learned_addr learned my addr 172.21.15.6:0/794792049 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:10.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.812+0000 7f065fe00640 1 -- 172.21.15.6:0/794792049 >> v1:172.21.15.92:6789/0 conn(0x7f06601544e0 legacy=0x7f066015d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.812+0000 7f065fe00640 1 -- 172.21.15.6:0/794792049 >> v1:172.21.15.6:6789/0 conn(0x7f06600c80a0 legacy=0x7f0660072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:10.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.812+0000 7f065fe00640 1 -- 172.21.15.6:0/794792049 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066015ab10 con 0x7f06601548c0 2024-03-22T13:12:10.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.815+0000 7f065fe00640 1 --2- 172.21.15.6:0/794792049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601548c0 0x7f06600c69b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f064c0097d0 tx=0x7f064c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=871994c9248bfed server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.821+0000 7f065f400640 1 -- 172.21.15.6:0/794792049 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c035020 con 0x7f06601548c0 2024-03-22T13:12:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.821+0000 7f065f400640 1 -- 172.21.15.6:0/794792049 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c031890 con 0x7f06601548c0 2024-03-22T13:12:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.821+0000 7f065f400640 1 -- 172.21.15.6:0/794792049 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c031c60 con 0x7f06601548c0 2024-03-22T13:12:10.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.821+0000 7f0667800640 1 -- 172.21.15.6:0/794792049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601548c0 msgr2=0x7f06600c69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.821+0000 7f0667800640 1 --2- 172.21.15.6:0/794792049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601548c0 0x7f06600c69b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f064c0097d0 tx=0x7f064c02b930 comp rx=0 tx=0).stop 2024-03-22T13:12:10.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.822+0000 7f0667800640 1 -- 172.21.15.6:0/794792049 shutdown_connections 2024-03-22T13:12:10.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.822+0000 7f0667800640 1 --2- 172.21.15.6:0/794792049 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601548c0 0x7f06600c69b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.822+0000 7f0667800640 1 -- 172.21.15.6:0/794792049 >> 172.21.15.6:0/794792049 conn(0x7f066006e310 msgr2=0x7f066006f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:10.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.822+0000 7f0667800640 1 -- 172.21.15.6:0/794792049 shutdown_connections 2024-03-22T13:12:10.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.822+0000 7f0667800640 1 -- 172.21.15.6:0/794792049 wait complete. 2024-03-22T13:12:10.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.824+0000 7f0667800640 1 Processor -- start 2024-03-22T13:12:10.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.824+0000 7f0667800640 1 -- start start 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.825+0000 7f0667800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06600c80a0 0x7f066015c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.825+0000 7f0667800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601544e0 0x7f066014df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.825+0000 7f0667800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066014d810 0x7f06601f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.825+0000 7f0667800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066015e1c0 con 0x7f066014d810 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.825+0000 7f0667800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066015e040 con 0x7f06600c80a0 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.825+0000 7f0667800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0660073010 con 0x7f06601544e0 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.825+0000 7f065fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601544e0 0x7f066014df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.826+0000 7f065fe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601544e0 0x7f066014df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54372/0 (socket says 172.21.15.6:54372) 2024-03-22T13:12:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.826+0000 7f065fe00640 1 -- 172.21.15.6:0/3664986691 learned_addr learned my addr 172.21.15.6:0/3664986691 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:10.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.827+0000 7f0664e00640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06600c80a0 0x7f066015c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.831+0000 7f065fe00640 1 -- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06600c80a0 msgr2=0x7f066015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.831+0000 7f065fe00640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06600c80a0 0x7f066015c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.831+0000 7f065fe00640 1 -- 172.21.15.6:0/3664986691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066014d810 msgr2=0x7f06601f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.831+0000 7f065fe00640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066014d810 0x7f06601f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:10.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.831+0000 7f065fe00640 1 -- 172.21.15.6:0/3664986691 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06601f1610 con 0x7f06601544e0 2024-03-22T13:12:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.834+0000 7f065fe00640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601544e0 0x7f066014df10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f064c0097d0 tx=0x7f064c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.840+0000 7f065cc00640 1 -- 172.21.15.6:0/3664986691 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c03ae60 con 0x7f06601544e0 2024-03-22T13:12:10.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.840+0000 7f065cc00640 1 -- 172.21.15.6:0/3664986691 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c0443f0 con 0x7f06601544e0 2024-03-22T13:12:10.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.840+0000 7f065cc00640 1 -- 172.21.15.6:0/3664986691 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c0447c0 con 0x7f06601544e0 2024-03-22T13:12:10.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.840+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066015d2d0 con 0x7f06601544e0 2024-03-22T13:12:10.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.840+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066015beb0 con 0x7f06601544e0 2024-03-22T13:12:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.841+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06600757d0 con 0x7f06601544e0 2024-03-22T13:12:10.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.877+0000 7f065cc00640 1 -- 172.21.15.6:0/3664986691 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f064c04b060 con 0x7f06601544e0 2024-03-22T13:12:10.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.879+0000 7f065cc00640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f064006bc20 0x7f064006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.879+0000 7f0664e00640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f064006bc20 0x7f064006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.880+0000 7f0664e00640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f064006bc20 0x7f064006e0e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0654005880 tx=0x7f0654009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:10.880+0000 7f065cc00640 1 -- 172.21.15.6:0/3664986691 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f064c031830 con 0x7f06601544e0 2024-03-22T13:12:11.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.172+0000 7f065cc00640 1 -- 172.21.15.6:0/3664986691 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f064c03f070 con 0x7f06601544e0 2024-03-22T13:12:11.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.419+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f06600cc9e0 con 0x7f064006bc20 2024-03-22T13:12:11.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.420+0000 7f065cc00640 1 -- 172.21.15.6:0/3664986691 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f06600cc9e0 con 0x7f064006bc20 2024-03-22T13:12:11.420 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f064006bc20 msgr2=0x7f064006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f064006bc20 0x7f064006e0e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0654005880 tx=0x7f0654009040 comp rx=0 tx=0).stop 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601544e0 msgr2=0x7f066014df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601544e0 0x7f066014df10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f064c0097d0 tx=0x7f064c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 shutdown_connections 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066014d810 0x7f06601f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06601544e0 0x7f066014df10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f064006bc20 0x7f064006e0e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 --2- 172.21.15.6:0/3664986691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06600c80a0 0x7f066015c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 >> 172.21.15.6:0/3664986691 conn(0x7f066006e310 msgr2=0x7f0660152f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:11.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 shutdown_connections 2024-03-22T13:12:11.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.424+0000 7f0667800640 1 -- 172.21.15.6:0/3664986691 wait complete. 2024-03-22T13:12:11.450 INFO:tasks.ceph:Scrubbing osd.3 2024-03-22T13:12:11.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:11.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.637+0000 7f5ff3a00640 1 Processor -- start 2024-03-22T13:12:11.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.637+0000 7f5ff3a00640 1 -- start start 2024-03-22T13:12:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.639+0000 7f5ff3a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5fec151410 0x7f5fec0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.639+0000 7f5ff3a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0c5710 0x7f5fec15a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.639+0000 7f5ff3a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0cb700 con 0x7f5fec0c4b70 2024-03-22T13:12:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.639+0000 7f5ff3a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec15e3a0 con 0x7f5fec151410 2024-03-22T13:12:11.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.639+0000 7f5ff3a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec15e8d0 con 0x7f5fec0c5710 2024-03-22T13:12:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.640+0000 7f5febe00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0c5710 0x7f5fec15a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.640+0000 7f5ff1000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5fec151410 0x7f5fec0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.640+0000 7f5ff1a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5fec0c4b70 0x7f5fec15a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60200/0 (socket says 172.21.15.6:60200) 2024-03-22T13:12:11.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.640+0000 7f5ff1a00640 1 -- 172.21.15.6:0/391422734 learned_addr learned my addr 172.21.15.6:0/391422734 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:11.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.643+0000 7f5feb400640 1 -- 172.21.15.6:0/391422734 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346796118 0 0) 0x7f5fec0cb700 con 0x7f5fec0c4b70 2024-03-22T13:12:11.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.643+0000 7f5feb400640 1 -- 172.21.15.6:0/391422734 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd4003520 con 0x7f5fec0c4b70 2024-03-22T13:12:11.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.643+0000 7f5febe00640 1 -- 172.21.15.6:0/391422734 >> v2:172.21.15.92:3300/0 conn(0x7f5fec151410 msgr2=0x7f5fec0cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:11.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.643+0000 7f5febe00640 1 --2- 172.21.15.6:0/391422734 >> v2:172.21.15.92:3300/0 conn(0x7f5fec151410 0x7f5fec0cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.643+0000 7f5febe00640 1 -- 172.21.15.6:0/391422734 >> v1:172.21.15.6:6789/0 conn(0x7f5fec0c4b70 legacy=0x7f5fec15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.643+0000 7f5febe00640 1 -- 172.21.15.6:0/391422734 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec15e5f0 con 0x7f5fec0c5710 2024-03-22T13:12:11.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.646+0000 7f5febe00640 1 --2- 172.21.15.6:0/391422734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0c5710 0x7f5fec15a3e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80097a0 tx=0x7f5fd802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5e7166235445c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:11.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.651+0000 7f5feb400640 1 -- 172.21.15.6:0/391422734 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8035020 con 0x7f5fec0c5710 2024-03-22T13:12:11.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.651+0000 7f5feb400640 1 -- 172.21.15.6:0/391422734 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd8031830 con 0x7f5fec0c5710 2024-03-22T13:12:11.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.651+0000 7f5feb400640 1 -- 172.21.15.6:0/391422734 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8031c00 con 0x7f5fec0c5710 2024-03-22T13:12:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 -- 172.21.15.6:0/391422734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0c5710 msgr2=0x7f5fec15a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 --2- 172.21.15.6:0/391422734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0c5710 0x7f5fec15a3e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80097a0 tx=0x7f5fd802b8d0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 -- 172.21.15.6:0/391422734 shutdown_connections 2024-03-22T13:12:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 --2- 172.21.15.6:0/391422734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0c5710 0x7f5fec15a3e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 --2- 172.21.15.6:0/391422734 >> v2:172.21.15.92:3300/0 conn(0x7f5fec151410 0x7f5fec0cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 -- 172.21.15.6:0/391422734 >> 172.21.15.6:0/391422734 conn(0x7f5fec066e40 msgr2=0x7f5fec067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:11.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 -- 172.21.15.6:0/391422734 shutdown_connections 2024-03-22T13:12:11.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.652+0000 7f5ff3a00640 1 -- 172.21.15.6:0/391422734 wait complete. 2024-03-22T13:12:11.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.654+0000 7f5ff3a00640 1 Processor -- start 2024-03-22T13:12:11.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.654+0000 7f5ff3a00640 1 -- start start 2024-03-22T13:12:11.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff3a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0c4b70 0x7f5fec14f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff3a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fec0c5710 0x7f5fec0b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff3a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec151410 0x7f5fec1f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff3a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0caca0 con 0x7f5fec0c5710 2024-03-22T13:12:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff3a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0ca9c0 con 0x7f5fec0c4b70 2024-03-22T13:12:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff3a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec15b030 con 0x7f5fec151410 2024-03-22T13:12:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff1a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec151410 0x7f5fec1f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0c4b70 0x7f5fec14f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5febe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fec0c5710 0x7f5fec0b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.655+0000 7f5ff1a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec151410 0x7f5fec1f0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54392/0 (socket says 172.21.15.6:54392) 2024-03-22T13:12:11.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.656+0000 7f5ff1a00640 1 -- 172.21.15.6:0/1084262045 learned_addr learned my addr 172.21.15.6:0/1084262045 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:11.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.660+0000 7f5ff1a00640 1 -- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0c4b70 msgr2=0x7f5fec14f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.660+0000 7f5ff1a00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0c4b70 0x7f5fec14f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.660+0000 7f5ff1a00640 1 -- 172.21.15.6:0/1084262045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fec0c5710 msgr2=0x7f5fec0b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.660+0000 7f5ff1a00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5fec0c5710 0x7f5fec0b53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.661+0000 7f5ff1a00640 1 -- 172.21.15.6:0/1084262045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec14fb30 con 0x7f5fec151410 2024-03-22T13:12:11.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.664+0000 7f5ff1a00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec151410 0x7f5fec1f0210 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400cbb0 tx=0x7f5fe400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.669+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4017070 con 0x7f5fec151410 2024-03-22T13:12:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.669+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe400ad40 con 0x7f5fec151410 2024-03-22T13:12:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.669+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4012600 con 0x7f5fec151410 2024-03-22T13:12:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.669+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec14fe10 con 0x7f5fec151410 2024-03-22T13:12:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.669+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec0b2ee0 con 0x7f5fec151410 2024-03-22T13:12:11.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.670+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fec0757d0 con 0x7f5fec151410 2024-03-22T13:12:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.696+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5fe401a070 con 0x7f5fec151410 2024-03-22T13:12:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.697+0000 7f5fe8c00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fcc06bc20 0x7f5fcc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.698+0000 7f5ff1000640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fcc06bc20 0x7f5fcc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.699+0000 7f5ff1000640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fcc06bc20 0x7f5fcc06e0e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009aa0 tx=0x7f5fe0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.700+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5fe4057510 con 0x7f5fec151410 2024-03-22T13:12:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.700+0000 7f5fe8c00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f5fcc0757d0 0x7f5fcc077c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.700+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 --> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fcc078340 con 0x7f5fcc0757d0 2024-03-22T13:12:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.700+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fe4055710 con 0x7f5fec151410 2024-03-22T13:12:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.700+0000 7f5febe00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f5fcc0757d0 0x7f5fcc077c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.701+0000 7f5febe00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f5fcc0757d0 0x7f5fcc077c70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.703+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== osd.3 v2:172.21.15.6:6824/484261108 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5fcc078340 con 0x7f5fcc0757d0 2024-03-22T13:12:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.719+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 --> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5fec074eb0 con 0x7f5fcc0757d0 2024-03-22T13:12:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5fe8c00640 1 -- 172.21.15.6:0/1084262045 <== osd.3 v2:172.21.15.6:6824/484261108 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5fec074eb0 con 0x7f5fcc0757d0 2024-03-22T13:12:11.720 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:11.720 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:11.720 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:11.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f5fcc0757d0 msgr2=0x7f5fcc077c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.6:6824/484261108,v1:172.21.15.6:6825/484261108] conn(0x7f5fcc0757d0 0x7f5fcc077c70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fcc06bc20 msgr2=0x7f5fcc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5fcc06bc20 0x7f5fcc06e0e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009aa0 tx=0x7f5fe0009190 comp rx=0 tx=0).stop 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec151410 msgr2=0x7f5fec1f0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 --2- 172.21.15.6:0/1084262045 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec151410 0x7f5fec1f0210 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400cbb0 tx=0x7f5fe400a430 comp rx=0 tx=0).stop 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff1a00640 1 -- 172.21.15.6:0/1084262045 reap_dead start 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 shutdown_connections 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.720+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 >> 172.21.15.6:0/1084262045 conn(0x7f5fec066e40 msgr2=0x7f5fec14eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.721+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 shutdown_connections 2024-03-22T13:12:11.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.721+0000 7f5ff3a00640 1 -- 172.21.15.6:0/1084262045 wait complete. 2024-03-22T13:12:11.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-22T13:12:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.928+0000 7fce8b600640 1 Processor -- start 2024-03-22T13:12:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.928+0000 7fce8b600640 1 -- start start 2024-03-22T13:12:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.929+0000 7fce8b600640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce840cf660 0x7fce840cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.929+0000 7fce8b600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce840ccf20 con 0x7fce840cdf00 2024-03-22T13:12:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.930+0000 7fce8b600640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8415a6b0 con 0x7fce840ceae0 2024-03-22T13:12:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.930+0000 7fce8b600640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce840cfa40 con 0x7fce840cf660 2024-03-22T13:12:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.930+0000 7fce88c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce840cf660 0x7fce840cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.931+0000 7fce88c00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce840cf660 0x7fce840cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36972/0 (socket says 172.21.15.6:36972) 2024-03-22T13:12:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.931+0000 7fce88c00640 1 -- 172.21.15.6:0/1937876744 learned_addr learned my addr 172.21.15.6:0/1937876744 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.934+0000 7fce88c00640 1 -- 172.21.15.6:0/1937876744 >> v1:172.21.15.92:6789/0 conn(0x7fce840ceae0 legacy=0x7fce840cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.934+0000 7fce88c00640 1 -- 172.21.15.6:0/1937876744 >> v1:172.21.15.6:6789/0 conn(0x7fce840cdf00 legacy=0x7fce840ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.934+0000 7fce88c00640 1 -- 172.21.15.6:0/1937876744 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8415a990 con 0x7fce840cf660 2024-03-22T13:12:11.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.939+0000 7fce88c00640 1 --2- 172.21.15.6:0/1937876744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cf660 0x7fce840cd9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fce74009e50 tx=0x7fce7402c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2e1708a11c4f129 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.944+0000 7fce83400640 1 -- 172.21.15.6:0/1937876744 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7402cbb0 con 0x7fce840cf660 2024-03-22T13:12:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.944+0000 7fce83400640 1 -- 172.21.15.6:0/1937876744 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7402cd50 con 0x7fce840cf660 2024-03-22T13:12:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.944+0000 7fce83400640 1 -- 172.21.15.6:0/1937876744 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce74034610 con 0x7fce840cf660 2024-03-22T13:12:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.945+0000 7fce8b600640 1 -- 172.21.15.6:0/1937876744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cf660 msgr2=0x7fce840cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.945+0000 7fce8b600640 1 --2- 172.21.15.6:0/1937876744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cf660 0x7fce840cd9c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fce74009e50 tx=0x7fce7402c010 comp rx=0 tx=0).stop 2024-03-22T13:12:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.945+0000 7fce8b600640 1 -- 172.21.15.6:0/1937876744 shutdown_connections 2024-03-22T13:12:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.945+0000 7fce8b600640 1 --2- 172.21.15.6:0/1937876744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cf660 0x7fce840cd9c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.945+0000 7fce8b600640 1 -- 172.21.15.6:0/1937876744 >> 172.21.15.6:0/1937876744 conn(0x7fce840b16d0 msgr2=0x7fce840b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.945+0000 7fce8b600640 1 -- 172.21.15.6:0/1937876744 shutdown_connections 2024-03-22T13:12:11.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.945+0000 7fce8b600640 1 -- 172.21.15.6:0/1937876744 wait complete. 2024-03-22T13:12:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.947+0000 7fce8b600640 1 Processor -- start 2024-03-22T13:12:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.947+0000 7fce8b600640 1 -- start start 2024-03-22T13:12:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.949+0000 7fce8b600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cdf00 0x7fce840c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.949+0000 7fce8b600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce840ceae0 0x7fce840c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.949+0000 7fce8b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce840cf660 0x7fce841e5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.949+0000 7fce8b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8415afe0 con 0x7fce840cf660 2024-03-22T13:12:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.949+0000 7fce8b600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8415b470 con 0x7fce840cdf00 2024-03-22T13:12:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.949+0000 7fce8b600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce840cac50 con 0x7fce840ceae0 2024-03-22T13:12:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.949+0000 7fce88c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cdf00 0x7fce840c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.950+0000 7fce83e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce840ceae0 0x7fce840c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.950+0000 7fce88c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cdf00 0x7fce840c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:36982/0 (socket says 172.21.15.6:36982) 2024-03-22T13:12:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.950+0000 7fce88c00640 1 -- 172.21.15.6:0/2750142727 learned_addr learned my addr 172.21.15.6:0/2750142727 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.955+0000 7fce83e00640 1 -- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cdf00 msgr2=0x7fce840c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.955+0000 7fce83e00640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cdf00 0x7fce840c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.955+0000 7fce83e00640 1 -- 172.21.15.6:0/2750142727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce840cf660 msgr2=0x7fce841e5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.955+0000 7fce83e00640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce840cf660 0x7fce841e5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.955+0000 7fce83e00640 1 -- 172.21.15.6:0/2750142727 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce841f11e0 con 0x7fce840ceae0 2024-03-22T13:12:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.956+0000 7fce88c00640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cdf00 0x7fce840c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T13:12:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.959+0000 7fce83e00640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce840ceae0 0x7fce840c37d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00b410 tx=0x7fce6c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:11.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.964+0000 7fce80c00640 1 -- 172.21.15.6:0/2750142727 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c015020 con 0x7fce840ceae0 2024-03-22T13:12:11.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.964+0000 7fce80c00640 1 -- 172.21.15.6:0/2750142727 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c011830 con 0x7fce840ceae0 2024-03-22T13:12:11.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.964+0000 7fce80c00640 1 -- 172.21.15.6:0/2750142727 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c011c20 con 0x7fce840ceae0 2024-03-22T13:12:11.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.964+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce841f0f00 con 0x7fce840ceae0 2024-03-22T13:12:11.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.964+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce840b3d50 con 0x7fce840ceae0 2024-03-22T13:12:11.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:11.965+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce840b2400 con 0x7fce840ceae0 2024-03-22T13:12:12.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.241+0000 7fce80c00640 1 -- 172.21.15.6:0/2750142727 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fce6c018020 con 0x7fce840ceae0 2024-03-22T13:12:12.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.243+0000 7fce80c00640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce6006bc20 0x7fce6006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.243+0000 7fce88c00640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce6006bc20 0x7fce6006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.245+0000 7fce88c00640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce6006bc20 0x7fce6006e0e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fce74000f80 tx=0x7fce74030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:12.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.246+0000 7fce80c00640 1 -- 172.21.15.6:0/2750142727 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fce6c057300 con 0x7fce840ceae0 2024-03-22T13:12:12.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.265+0000 7fce80c00640 1 -- 172.21.15.6:0/2750142727 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fce840b2400 con 0x7fce840ceae0 2024-03-22T13:12:12.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.480+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fce840757d0 con 0x7fce6006bc20 2024-03-22T13:12:12.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.481+0000 7fce80c00640 1 -- 172.21.15.6:0/2750142727 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fce840757d0 con 0x7fce6006bc20 2024-03-22T13:12:12.484 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.484+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce6006bc20 msgr2=0x7fce6006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.484+0000 7fce8b600640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce6006bc20 0x7fce6006e0e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fce74000f80 tx=0x7fce74030000 comp rx=0 tx=0).stop 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.484+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce840ceae0 msgr2=0x7fce840c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.484+0000 7fce8b600640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce840ceae0 0x7fce840c37d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00b410 tx=0x7fce6c00b8d0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 shutdown_connections 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce840cf660 0x7fce841e5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce840ceae0 0x7fce840c37d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fce6006bc20 0x7fce6006e0e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 --2- 172.21.15.6:0/2750142727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce840cdf00 0x7fce840c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 >> 172.21.15.6:0/2750142727 conn(0x7fce840b16d0 msgr2=0x7fce8406d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:12.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 shutdown_connections 2024-03-22T13:12:12.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.485+0000 7fce8b600640 1 -- 172.21.15.6:0/2750142727 wait complete. 2024-03-22T13:12:12.511 INFO:tasks.ceph:Scrubbing osd.4 2024-03-22T13:12:12.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:12.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.708+0000 7f301b200640 1 Processor -- start 2024-03-22T13:12:12.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.708+0000 7f301b200640 1 -- start start 2024-03-22T13:12:12.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.710+0000 7f301b200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3014154220 0x7f30140c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.710+0000 7f301b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f301415ad50 0x7f301406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.710+0000 7f301b200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301415da40 con 0x7f301406c850 2024-03-22T13:12:12.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.710+0000 7f301b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f301415dce0 con 0x7f301415ad50 2024-03-22T13:12:12.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.710+0000 7f301b200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301415dfc0 con 0x7f3014154220 2024-03-22T13:12:12.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.711+0000 7f3013e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3014154220 0x7f30140c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.711+0000 7f300be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f301415ad50 0x7f301406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.712+0000 7f3019200640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f301406c850 0x7f30140c8df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:43840/0 (socket says 172.21.15.6:43840) 2024-03-22T13:12:12.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.712+0000 7f3019200640 1 -- 172.21.15.6:0/1649766555 learned_addr learned my addr 172.21.15.6:0/1649766555 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.714+0000 7f300be00640 1 -- 172.21.15.6:0/1649766555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3014154220 msgr2=0x7f30140c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.714+0000 7f300be00640 1 --2- 172.21.15.6:0/1649766555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3014154220 0x7f30140c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.715+0000 7f300be00640 1 -- 172.21.15.6:0/1649766555 >> v1:172.21.15.92:6789/0 conn(0x7f301406c850 legacy=0x7f30140c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.715+0000 7f300be00640 1 -- 172.21.15.6:0/1649766555 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301415e2a0 con 0x7f301415ad50 2024-03-22T13:12:12.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.718+0000 7f300be00640 1 --2- 172.21.15.6:0/1649766555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f301415ad50 0x7f301406c310 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0097a0 tx=0x7f2ffc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25472daa332bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:12.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.723+0000 7f3013400640 1 -- 172.21.15.6:0/1649766555 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc035020 con 0x7f301415ad50 2024-03-22T13:12:12.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.723+0000 7f3013400640 1 -- 172.21.15.6:0/1649766555 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ffc031810 con 0x7f301415ad50 2024-03-22T13:12:12.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.723+0000 7f3013400640 1 -- 172.21.15.6:0/1649766555 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc031be0 con 0x7f301415ad50 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 -- 172.21.15.6:0/1649766555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f301415ad50 msgr2=0x7f301406c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 --2- 172.21.15.6:0/1649766555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f301415ad50 0x7f301406c310 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0097a0 tx=0x7f2ffc02b8b0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 -- 172.21.15.6:0/1649766555 shutdown_connections 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 --2- 172.21.15.6:0/1649766555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f301415ad50 0x7f301406c310 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 --2- 172.21.15.6:0/1649766555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3014154220 0x7f30140c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 -- 172.21.15.6:0/1649766555 >> 172.21.15.6:0/1649766555 conn(0x7f30140727e0 msgr2=0x7f3014072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 -- 172.21.15.6:0/1649766555 shutdown_connections 2024-03-22T13:12:12.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.724+0000 7f301b200640 1 -- 172.21.15.6:0/1649766555 wait complete. 2024-03-22T13:12:12.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.726+0000 7f301b200640 1 Processor -- start 2024-03-22T13:12:12.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.726+0000 7f301b200640 1 -- start start 2024-03-22T13:12:12.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.727+0000 7f301b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f301406c850 0x7f301414cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.727+0000 7f301b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3014154220 0x7f301414f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.727+0000 7f301b200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f301415ad50 0x7f30141f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.727+0000 7f301b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301415b190 con 0x7f3014154220 2024-03-22T13:12:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.727+0000 7f301b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301406cfc0 con 0x7f301406c850 2024-03-22T13:12:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.727+0000 7f301b200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3014154600 con 0x7f301415ad50 2024-03-22T13:12:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.728+0000 7f3013e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f301406c850 0x7f301414cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.728+0000 7f3019200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f301415ad50 0x7f30141f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.729+0000 7f3019200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f301415ad50 0x7f30141f0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54432/0 (socket says 172.21.15.6:54432) 2024-03-22T13:12:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.729+0000 7f3019200640 1 -- 172.21.15.6:0/2202687867 learned_addr learned my addr 172.21.15.6:0/2202687867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:12.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.729+0000 7f300be00640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3014154220 0x7f301414f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.734+0000 7f3019200640 1 -- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f301406c850 msgr2=0x7f301414cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.734+0000 7f3019200640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f301406c850 0x7f301414cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.734+0000 7f3019200640 1 -- 172.21.15.6:0/2202687867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3014154220 msgr2=0x7f301414f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.734+0000 7f3019200640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3014154220 0x7f301414f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.734+0000 7f3019200640 1 -- 172.21.15.6:0/2202687867 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301414d2b0 con 0x7f301415ad50 2024-03-22T13:12:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.739+0000 7f3019200640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f301415ad50 0x7f30141f0110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f300c00ca80 tx=0x7f300c00cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.744+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c017070 con 0x7f301415ad50 2024-03-22T13:12:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.744+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c00ac30 con 0x7f301415ad50 2024-03-22T13:12:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.744+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c012600 con 0x7f301415ad50 2024-03-22T13:12:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.744+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301414d590 con 0x7f301415ad50 2024-03-22T13:12:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.744+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30141f3a70 con 0x7f301415ad50 2024-03-22T13:12:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.745+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f30140757d0 con 0x7f301415ad50 2024-03-22T13:12:12.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.779+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f300c01a070 con 0x7f301415ad50 2024-03-22T13:12:12.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.780+0000 7f3010c00640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ff006bc20 0x7f2ff006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.781+0000 7f3013e00640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ff006bc20 0x7f2ff006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.781+0000 7f3013e00640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ff006bc20 0x7f2ff006e0e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3004009b40 tx=0x7f3004009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.783+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f300c057430 con 0x7f301415ad50 2024-03-22T13:12:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.783+0000 7f3010c00640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f2ff00757d0 0x7f2ff0077c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.783+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 --> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f300c013e50 con 0x7f2ff00757d0 2024-03-22T13:12:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.783+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f300c055630 con 0x7f301415ad50 2024-03-22T13:12:12.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.784+0000 7f300be00640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f2ff00757d0 0x7f2ff0077c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:12.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.784+0000 7f300be00640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f2ff00757d0 0x7f2ff0077c70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:12.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.786+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== osd.4 v2:172.21.15.92:6819/2540113998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f300c013e50 con 0x7f2ff00757d0 2024-03-22T13:12:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.805+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 --> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f30140c8550 con 0x7f2ff00757d0 2024-03-22T13:12:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.805+0000 7f3010c00640 1 -- 172.21.15.6:0/2202687867 <== osd.4 v2:172.21.15.92:6819/2540113998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f30140c8550 con 0x7f2ff00757d0 2024-03-22T13:12:12.805 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:12.805 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:12.805 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f2ff00757d0 msgr2=0x7f2ff0077c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6819/2540113998,v1:172.21.15.92:6821/2540113998] conn(0x7f2ff00757d0 0x7f2ff0077c70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ff006bc20 msgr2=0x7f2ff006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f2ff006bc20 0x7f2ff006e0e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3004009b40 tx=0x7f3004009190 comp rx=0 tx=0).stop 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f301415ad50 msgr2=0x7f30141f0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 --2- 172.21.15.6:0/2202687867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f301415ad50 0x7f30141f0110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f300c00ca80 tx=0x7f300c00cf40 comp rx=0 tx=0).stop 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f3019200640 1 -- 172.21.15.6:0/2202687867 reap_dead start 2024-03-22T13:12:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 shutdown_connections 2024-03-22T13:12:12.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 >> 172.21.15.6:0/2202687867 conn(0x7f30140727e0 msgr2=0x7f30140c8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:12.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 shutdown_connections 2024-03-22T13:12:12.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:12.806+0000 7f301b200640 1 -- 172.21.15.6:0/2202687867 wait complete. 2024-03-22T13:12:12.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-22T13:12:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.019+0000 7f4540a00640 1 Processor -- start 2024-03-22T13:12:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.019+0000 7f4540a00640 1 -- start start 2024-03-22T13:12:13.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.020+0000 7f4540a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f453c154520 0x7f453c15dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.020+0000 7f4540a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f453c15a130 0x7f453c0c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.021+0000 7f4540a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c15e320 0x7f453c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.021+0000 7f4540a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f453c15b140 con 0x7f453c15a130 2024-03-22T13:12:13.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.021+0000 7f4540a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f453c154900 con 0x7f453c154520 2024-03-22T13:12:13.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.021+0000 7f4540a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453c071db0 con 0x7f453c15e320 2024-03-22T13:12:13.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.027+0000 7f453be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c15e320 0x7f453c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.028+0000 7f453be00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c15e320 0x7f453c0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54426/0 (socket says 172.21.15.6:54426) 2024-03-22T13:12:13.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.028+0000 7f453be00640 1 -- 172.21.15.6:0/3225849611 learned_addr learned my addr 172.21.15.6:0/3225849611 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:13.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.030+0000 7f453be00640 1 -- 172.21.15.6:0/3225849611 >> v2:172.21.15.92:3300/0 conn(0x7f453c154520 msgr2=0x7f453c15dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:13.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.030+0000 7f453be00640 1 --2- 172.21.15.6:0/3225849611 >> v2:172.21.15.92:3300/0 conn(0x7f453c154520 0x7f453c15dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.030+0000 7f453be00640 1 -- 172.21.15.6:0/3225849611 >> v2:172.21.15.6:3300/0 conn(0x7f453c15a130 msgr2=0x7f453c0c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:13.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.030+0000 7f453be00640 1 --2- 172.21.15.6:0/3225849611 >> v2:172.21.15.6:3300/0 conn(0x7f453c15a130 0x7f453c0c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.030+0000 7f453be00640 1 -- 172.21.15.6:0/3225849611 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c15a5c0 con 0x7f453c15e320 2024-03-22T13:12:13.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.033+0000 7f453be00640 1 --2- 172.21.15.6:0/3225849611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c15e320 0x7f453c0c6c00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f453c06fd30 tx=0x7f453402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6c849ab339b8b4a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.037+0000 7f453a000640 1 -- 172.21.15.6:0/3225849611 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453402ca70 con 0x7f453c15e320 2024-03-22T13:12:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.037+0000 7f453a000640 1 -- 172.21.15.6:0/3225849611 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453402cc10 con 0x7f453c15e320 2024-03-22T13:12:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.037+0000 7f453a000640 1 -- 172.21.15.6:0/3225849611 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45340343f0 con 0x7f453c15e320 2024-03-22T13:12:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 -- 172.21.15.6:0/3225849611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c15e320 msgr2=0x7f453c0c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 --2- 172.21.15.6:0/3225849611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c15e320 0x7f453c0c6c00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f453c06fd30 tx=0x7f453402bed0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 -- 172.21.15.6:0/3225849611 shutdown_connections 2024-03-22T13:12:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 --2- 172.21.15.6:0/3225849611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c15e320 0x7f453c0c6c00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 --2- 172.21.15.6:0/3225849611 >> v2:172.21.15.6:3300/0 conn(0x7f453c15a130 0x7f453c0c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 --2- 172.21.15.6:0/3225849611 >> v2:172.21.15.92:3300/0 conn(0x7f453c154520 0x7f453c15dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 -- 172.21.15.6:0/3225849611 >> 172.21.15.6:0/3225849611 conn(0x7f453c06f0e0 msgr2=0x7f453c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 -- 172.21.15.6:0/3225849611 shutdown_connections 2024-03-22T13:12:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.038+0000 7f4540a00640 1 -- 172.21.15.6:0/3225849611 wait complete. 2024-03-22T13:12:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.041+0000 7f4540a00640 1 Processor -- start 2024-03-22T13:12:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.041+0000 7f4540a00640 1 -- start start 2024-03-22T13:12:13.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f4540a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c154520 0x7f453c149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f4540a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f453c15e320 0x7f453c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f4540a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453c14f2c0 0x7f453c1efe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f4540a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c15ae60 con 0x7f453c15e320 2024-03-22T13:12:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f4540a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c15ab80 con 0x7f453c14f2c0 2024-03-22T13:12:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f4540a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453c15d8c0 con 0x7f453c154520 2024-03-22T13:12:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f453b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c154520 0x7f453c149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.043+0000 7f453b400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c154520 0x7f453c149190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54436/0 (socket says 172.21.15.6:54436) 2024-03-22T13:12:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.044+0000 7f453b400640 1 -- 172.21.15.6:0/822787635 learned_addr learned my addr 172.21.15.6:0/822787635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.044+0000 7f453aa00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f453c15e320 0x7f453c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.048+0000 7f453b400640 1 -- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453c14f2c0 msgr2=0x7f453c1efe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.048+0000 7f453b400640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453c14f2c0 0x7f453c1efe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.048+0000 7f453b400640 1 -- 172.21.15.6:0/822787635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f453c15e320 msgr2=0x7f453c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.048+0000 7f453b400640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f453c15e320 0x7f453c15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.048+0000 7f453b400640 1 -- 172.21.15.6:0/822787635 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c1f0920 con 0x7f453c154520 2024-03-22T13:12:13.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.051+0000 7f453b400640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c154520 0x7f453c149190 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f452c00b4d0 tx=0x7f452c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:13.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.056+0000 7f4533400640 1 -- 172.21.15.6:0/822787635 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c015020 con 0x7f453c154520 2024-03-22T13:12:13.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.056+0000 7f4533400640 1 -- 172.21.15.6:0/822787635 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452c011850 con 0x7f453c154520 2024-03-22T13:12:13.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.056+0000 7f4533400640 1 -- 172.21.15.6:0/822787635 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c011c40 con 0x7f453c154520 2024-03-22T13:12:13.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.056+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c1f0c00 con 0x7f453c154520 2024-03-22T13:12:13.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.056+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453c14eac0 con 0x7f453c154520 2024-03-22T13:12:13.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.057+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453c075650 con 0x7f453c154520 2024-03-22T13:12:13.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.080+0000 7f4533400640 1 -- 172.21.15.6:0/822787635 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f452c018020 con 0x7f453c154520 2024-03-22T13:12:13.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.081+0000 7f4533400640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f451c06bc20 0x7f451c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.082+0000 7f453aa00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f451c06bc20 0x7f451c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.082+0000 7f453aa00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f451c06bc20 0x7f451c06e0e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f453c15d0a0 tx=0x7f4528009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.083+0000 7f4533400640 1 -- 172.21.15.6:0/822787635 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f452c011de0 con 0x7f453c154520 2024-03-22T13:12:13.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.368+0000 7f4533400640 1 -- 172.21.15.6:0/822787635 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f452c01f020 con 0x7f453c154520 2024-03-22T13:12:13.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.574+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f453c06ff00 con 0x7f451c06bc20 2024-03-22T13:12:13.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.575+0000 7f4533400640 1 -- 172.21.15.6:0/822787635 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f453c06ff00 con 0x7f451c06bc20 2024-03-22T13:12:13.575 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2024-03-22T13:12:13.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f451c06bc20 msgr2=0x7f451c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f451c06bc20 0x7f451c06e0e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f453c15d0a0 tx=0x7f4528009110 comp rx=0 tx=0).stop 2024-03-22T13:12:13.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c154520 msgr2=0x7f453c149190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c154520 0x7f453c149190 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f452c00b4d0 tx=0x7f452c00b8d0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 shutdown_connections 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f451c06bc20 0x7f451c06e0e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453c14f2c0 0x7f453c1efe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f453c15e320 0x7f453c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 --2- 172.21.15.6:0/822787635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f453c154520 0x7f453c149190 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 >> 172.21.15.6:0/822787635 conn(0x7f453c06f0e0 msgr2=0x7f453c1585d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.578+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 shutdown_connections 2024-03-22T13:12:13.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.579+0000 7f4540a00640 1 -- 172.21.15.6:0/822787635 wait complete. 2024-03-22T13:12:13.604 INFO:tasks.ceph:Scrubbing osd.5 2024-03-22T13:12:13.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:13.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.801+0000 7f3b87600640 1 Processor -- start 2024-03-22T13:12:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.802+0000 7f3b87600640 1 -- start start 2024-03-22T13:12:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.803+0000 7f3b87600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b8015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.803+0000 7f3b87600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b800cea70 0x7f3b8006c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.803+0000 7f3b87600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b800cc9b0 con 0x7f3b800ce520 2024-03-22T13:12:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.803+0000 7f3b87600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8015aca0 con 0x7f3b800cea70 2024-03-22T13:12:13.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.803+0000 7f3b87600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800ccc90 con 0x7f3b801549a0 2024-03-22T13:12:13.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.804+0000 7f3b84c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b8015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.804+0000 7f3b84c00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b8015b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54442/0 (socket says 172.21.15.6:54442) 2024-03-22T13:12:13.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.805+0000 7f3b84c00640 1 -- 172.21.15.6:0/1676239623 learned_addr learned my addr 172.21.15.6:0/1676239623 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.808+0000 7f3b84c00640 1 -- 172.21.15.6:0/1676239623 >> v2:172.21.15.6:3300/0 conn(0x7f3b800cea70 msgr2=0x7f3b8006c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.808+0000 7f3b84c00640 1 --2- 172.21.15.6:0/1676239623 >> v2:172.21.15.6:3300/0 conn(0x7f3b800cea70 0x7f3b8006c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.808+0000 7f3b84c00640 1 -- 172.21.15.6:0/1676239623 >> v1:172.21.15.6:6789/0 conn(0x7f3b800ce520 legacy=0x7f3b800cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.808+0000 7f3b84c00640 1 -- 172.21.15.6:0/1676239623 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8015a650 con 0x7f3b801549a0 2024-03-22T13:12:13.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.811+0000 7f3b84c00640 1 --2- 172.21.15.6:0/1676239623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b8015b130 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009870 tx=0x7f3b7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3febc0209d773168 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:13.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.815+0000 7f3b7f400640 1 -- 172.21.15.6:0/1676239623 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74034070 con 0x7f3b801549a0 2024-03-22T13:12:13.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.815+0000 7f3b7f400640 1 -- 172.21.15.6:0/1676239623 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b74030840 con 0x7f3b801549a0 2024-03-22T13:12:13.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.815+0000 7f3b7f400640 1 -- 172.21.15.6:0/1676239623 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74030c10 con 0x7f3b801549a0 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 -- 172.21.15.6:0/1676239623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 msgr2=0x7f3b8015b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 --2- 172.21.15.6:0/1676239623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b8015b130 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009870 tx=0x7f3b7402b980 comp rx=0 tx=0).stop 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 -- 172.21.15.6:0/1676239623 shutdown_connections 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 --2- 172.21.15.6:0/1676239623 >> v2:172.21.15.6:3300/0 conn(0x7f3b800cea70 0x7f3b8006c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 --2- 172.21.15.6:0/1676239623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b8015b130 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 -- 172.21.15.6:0/1676239623 >> 172.21.15.6:0/1676239623 conn(0x7f3b800b1700 msgr2=0x7f3b800b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 -- 172.21.15.6:0/1676239623 shutdown_connections 2024-03-22T13:12:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.817+0000 7f3b87600640 1 -- 172.21.15.6:0/1676239623 wait complete. 2024-03-22T13:12:13.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.820+0000 7f3b87600640 1 Processor -- start 2024-03-22T13:12:13.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.820+0000 7f3b87600640 1 -- start start 2024-03-22T13:12:13.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.821+0000 7f3b87600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b800ce520 0x7f3b80153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.821+0000 7f3b87600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b800cea70 0x7f3b80153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.821+0000 7f3b87600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b801e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.821+0000 7f3b87600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800cd430 con 0x7f3b800cea70 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.821+0000 7f3b87600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800ccf70 con 0x7f3b800ce520 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.821+0000 7f3b87600640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800cd5b0 con 0x7f3b801549a0 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.822+0000 7f3b85600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b801e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.823+0000 7f3b85600640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b801e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54444/0 (socket says 172.21.15.6:54444) 2024-03-22T13:12:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.823+0000 7f3b85600640 1 -- 172.21.15.6:0/2365104246 learned_addr learned my addr 172.21.15.6:0/2365104246 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:13.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.827+0000 7f3b84c00640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b800ce520 0x7f3b80153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.828+0000 7f3b85600640 1 -- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b800ce520 msgr2=0x7f3b80153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.828+0000 7f3b85600640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b800ce520 0x7f3b80153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.828+0000 7f3b85600640 1 -- 172.21.15.6:0/2365104246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b800cea70 msgr2=0x7f3b80153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.828+0000 7f3b85600640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b800cea70 0x7f3b80153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.828+0000 7f3b85600640 1 -- 172.21.15.6:0/2365104246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b801595e0 con 0x7f3b801549a0 2024-03-22T13:12:13.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.835+0000 7f3b85600640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b801e4b00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800b990 tx=0x7f3b7800be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.931+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78019070 con 0x7f3b801549a0 2024-03-22T13:12:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.932+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7800cba0 con 0x7f3b801549a0 2024-03-22T13:12:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.932+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b801598c0 con 0x7f3b801549a0 2024-03-22T13:12:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.932+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b80159e80 con 0x7f3b801549a0 2024-03-22T13:12:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.932+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78014440 con 0x7f3b801549a0 2024-03-22T13:12:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.933+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b800757d0 con 0x7f3b801549a0 2024-03-22T13:12:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.965+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3b7801c070 con 0x7f3b801549a0 2024-03-22T13:12:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.970+0000 7f3b7cc00640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b6006bc20 0x7f3b6006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.970+0000 7f3b84c00640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b6006bc20 0x7f3b6006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.970+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f3b7808b260 con 0x7f3b801549a0 2024-03-22T13:12:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.971+0000 7f3b84c00640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b6006bc20 0x7f3b6006e0e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3b74036000 tx=0x7f3b7403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.971+0000 7f3b7cc00640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7f3b60075840 0x7f3b60077ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.971+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 --> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b600783b0 con 0x7f3b60075840 2024-03-22T13:12:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.971+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b780555d0 con 0x7f3b801549a0 2024-03-22T13:12:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.971+0000 7f3b7fe00640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7f3b60075840 0x7f3b60077ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:13.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.972+0000 7f3b7fe00640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7f3b60075840 0x7f3b60077ce0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:13.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.973+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== osd.5 v2:172.21.15.92:6810/1588636001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3b600783b0 con 0x7f3b60075840 2024-03-22T13:12:13.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.991+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 --> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3b800b5b50 con 0x7f3b60075840 2024-03-22T13:12:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.991+0000 7f3b7cc00640 1 -- 172.21.15.6:0/2365104246 <== osd.5 v2:172.21.15.92:6810/1588636001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3b800b5b50 con 0x7f3b60075840 2024-03-22T13:12:13.991 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:13.991 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:13.991 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.992+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7f3b60075840 msgr2=0x7f3b60077ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.992+0000 7f3b87600640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6810/1588636001,v1:172.21.15.92:6811/1588636001] conn(0x7f3b60075840 0x7f3b60077ce0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.992+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b6006bc20 msgr2=0x7f3b6006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.992+0000 7f3b87600640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f3b6006bc20 0x7f3b6006e0e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3b74036000 tx=0x7f3b7403b040 comp rx=0 tx=0).stop 2024-03-22T13:12:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.992+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 msgr2=0x7f3b801e4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.992+0000 7f3b87600640 1 --2- 172.21.15.6:0/2365104246 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b801549a0 0x7f3b801e4b00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800b990 tx=0x7f3b7800be50 comp rx=0 tx=0).stop 2024-03-22T13:12:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.992+0000 7f3b85600640 1 -- 172.21.15.6:0/2365104246 reap_dead start 2024-03-22T13:12:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.993+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 shutdown_connections 2024-03-22T13:12:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.993+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 >> 172.21.15.6:0/2365104246 conn(0x7f3b800b1700 msgr2=0x7f3b8006d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.993+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 shutdown_connections 2024-03-22T13:12:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:13.993+0000 7f3b87600640 1 -- 172.21.15.6:0/2365104246 wait complete. 2024-03-22T13:12:14.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-22T13:12:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.203+0000 7fb934200640 1 Processor -- start 2024-03-22T13:12:14.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.203+0000 7fb934200640 1 -- start start 2024-03-22T13:12:14.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.205+0000 7fb934200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c1548b0 0x7fb92c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.206+0000 7fb934200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb92c0cc700 0x7fb92c0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.206+0000 7fb934200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92c0c5470 con 0x7fb92c067550 2024-03-22T13:12:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.206+0000 7fb934200640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0cd070 con 0x7fb92c0cc700 2024-03-22T13:12:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.206+0000 7fb934200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0c4ef0 con 0x7fb92c1548b0 2024-03-22T13:12:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.206+0000 7fb931800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c1548b0 0x7fb92c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.207+0000 7fb932200640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb92c0cc700 0x7fb92c0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.207+0000 7fb931800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c1548b0 0x7fb92c15b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54446/0 (socket says 172.21.15.6:54446) 2024-03-22T13:12:14.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.207+0000 7fb931800640 1 -- 172.21.15.6:0/1932340478 learned_addr learned my addr 172.21.15.6:0/1932340478 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.210+0000 7fb931800640 1 -- 172.21.15.6:0/1932340478 >> v2:172.21.15.92:3300/0 conn(0x7fb92c0cc700 msgr2=0x7fb92c0ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.210+0000 7fb931800640 1 --2- 172.21.15.6:0/1932340478 >> v2:172.21.15.92:3300/0 conn(0x7fb92c0cc700 0x7fb92c0ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.210+0000 7fb931800640 1 -- 172.21.15.6:0/1932340478 >> v1:172.21.15.92:6789/0 conn(0x7fb92c067550 legacy=0x7fb92c0cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.210+0000 7fb931800640 1 -- 172.21.15.6:0/1932340478 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0cba50 con 0x7fb92c1548b0 2024-03-22T13:12:14.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.213+0000 7fb931800640 1 --2- 172.21.15.6:0/1932340478 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c1548b0 0x7fb92c15b100 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb920009870 tx=0x7fb92002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99c7b9fda7ee7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:14.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.220+0000 7fb92be00640 1 -- 172.21.15.6:0/1932340478 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9200317c0 con 0x7fb92c1548b0 2024-03-22T13:12:14.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.220+0000 7fb92be00640 1 -- 172.21.15.6:0/1932340478 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb920037070 con 0x7fb92c1548b0 2024-03-22T13:12:14.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.220+0000 7fb92be00640 1 -- 172.21.15.6:0/1932340478 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920031b80 con 0x7fb92c1548b0 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 -- 172.21.15.6:0/1932340478 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c1548b0 msgr2=0x7fb92c15b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 --2- 172.21.15.6:0/1932340478 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c1548b0 0x7fb92c15b100 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb920009870 tx=0x7fb92002b860 comp rx=0 tx=0).stop 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 -- 172.21.15.6:0/1932340478 shutdown_connections 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 --2- 172.21.15.6:0/1932340478 >> v2:172.21.15.92:3300/0 conn(0x7fb92c0cc700 0x7fb92c0ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 --2- 172.21.15.6:0/1932340478 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c1548b0 0x7fb92c15b100 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 -- 172.21.15.6:0/1932340478 >> 172.21.15.6:0/1932340478 conn(0x7fb92c06cbc0 msgr2=0x7fb92c066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 -- 172.21.15.6:0/1932340478 shutdown_connections 2024-03-22T13:12:14.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.221+0000 7fb934200640 1 -- 172.21.15.6:0/1932340478 wait complete. 2024-03-22T13:12:14.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.224+0000 7fb934200640 1 Processor -- start 2024-03-22T13:12:14.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.224+0000 7fb934200640 1 -- start start 2024-03-22T13:12:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.225+0000 7fb934200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c067550 0x7fb92c0c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.225+0000 7fb934200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c0cc700 0x7fb92c0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.225+0000 7fb934200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb92c1548b0 0x7fb92c14b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.225+0000 7fb934200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0ca860 con 0x7fb92c0cc700 2024-03-22T13:12:14.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.225+0000 7fb934200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c15b640 con 0x7fb92c1548b0 2024-03-22T13:12:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.225+0000 7fb934200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0cab40 con 0x7fb92c067550 2024-03-22T13:12:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.226+0000 7fb931800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c067550 0x7fb92c0c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.226+0000 7fb932200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb92c1548b0 0x7fb92c14b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.226+0000 7fb931800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c067550 0x7fb92c0c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54452/0 (socket says 172.21.15.6:54452) 2024-03-22T13:12:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.226+0000 7fb931800640 1 -- 172.21.15.6:0/1934794676 learned_addr learned my addr 172.21.15.6:0/1934794676 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:14.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.227+0000 7fb930e00640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c0cc700 0x7fb92c0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.231+0000 7fb931800640 1 -- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb92c1548b0 msgr2=0x7fb92c14b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.231+0000 7fb931800640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb92c1548b0 0x7fb92c14b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.231+0000 7fb931800640 1 -- 172.21.15.6:0/1934794676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c0cc700 msgr2=0x7fb92c0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.231+0000 7fb931800640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c0cc700 0x7fb92c0c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.231+0000 7fb931800640 1 -- 172.21.15.6:0/1934794676 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c15b070 con 0x7fb92c067550 2024-03-22T13:12:14.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.235+0000 7fb931800640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c067550 0x7fb92c0c93e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb9200090b0 tx=0x7fb9200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.239+0000 7fb929600640 1 -- 172.21.15.6:0/1934794676 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920032650 con 0x7fb92c067550 2024-03-22T13:12:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.239+0000 7fb929600640 1 -- 172.21.15.6:0/1934794676 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9200327f0 con 0x7fb92c067550 2024-03-22T13:12:14.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.239+0000 7fb929600640 1 -- 172.21.15.6:0/1934794676 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920032be0 con 0x7fb92c067550 2024-03-22T13:12:14.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.239+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c15b330 con 0x7fb92c067550 2024-03-22T13:12:14.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.240+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c14d6e0 con 0x7fb92c067550 2024-03-22T13:12:14.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.241+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c06b740 con 0x7fb92c067550 2024-03-22T13:12:14.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.516+0000 7fb929600640 1 -- 172.21.15.6:0/1934794676 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb92004c060 con 0x7fb92c067550 2024-03-22T13:12:14.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.517+0000 7fb929600640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb90c06bc20 0x7fb90c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.518+0000 7fb930e00640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb90c06bc20 0x7fb90c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.519+0000 7fb930e00640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb90c06bc20 0x7fb90c06e0e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb92c14e440 tx=0x7fb918009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:14.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.522+0000 7fb929600640 1 -- 172.21.15.6:0/1934794676 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fb920078ae0 con 0x7fb92c067550 2024-03-22T13:12:14.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.541+0000 7fb929600640 1 -- 172.21.15.6:0/1934794676 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb920040070 con 0x7fb92c067550 2024-03-22T13:12:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.753+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fb92c0757d0 con 0x7fb90c06bc20 2024-03-22T13:12:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.753+0000 7fb929600640 1 -- 172.21.15.6:0/1934794676 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb92c0757d0 con 0x7fb90c06bc20 2024-03-22T13:12:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2024-03-22T13:12:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb90c06bc20 msgr2=0x7fb90c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb90c06bc20 0x7fb90c06e0e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb92c14e440 tx=0x7fb918009040 comp rx=0 tx=0).stop 2024-03-22T13:12:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c067550 msgr2=0x7fb92c0c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c067550 0x7fb92c0c93e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb9200090b0 tx=0x7fb9200046c0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 shutdown_connections 2024-03-22T13:12:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7fb90c06bc20 0x7fb90c06e0e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb92c1548b0 0x7fb92c14b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c0cc700 0x7fb92c0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 --2- 172.21.15.6:0/1934794676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb92c067550 0x7fb92c0c93e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 >> 172.21.15.6:0/1934794676 conn(0x7fb92c06cbc0 msgr2=0x7fb92c14fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 shutdown_connections 2024-03-22T13:12:14.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.757+0000 7fb934200640 1 -- 172.21.15.6:0/1934794676 wait complete. 2024-03-22T13:12:14.774 INFO:tasks.ceph:Scrubbing osd.6 2024-03-22T13:12:14.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.976+0000 7f50ab200640 1 Processor -- start 2024-03-22T13:12:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.976+0000 7f50ab200640 1 -- start start 2024-03-22T13:12:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.977+0000 7f50ab200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a415a2d0 0x7f50a40ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.977+0000 7f50ab200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a40cdf80 con 0x7f50a40cd8d0 2024-03-22T13:12:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.977+0000 7f50ab200640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a40cee40 con 0x7f50a415aeb0 2024-03-22T13:12:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.977+0000 7f50ab200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a40cdd60 con 0x7f50a415a2d0 2024-03-22T13:12:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.978+0000 7f50aa800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f50a40cd8d0 0x7f50a406d2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41228/0 (socket says 172.21.15.6:41228) 2024-03-22T13:12:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.978+0000 7f50aa800640 1 -- 172.21.15.6:0/2723050112 learned_addr learned my addr 172.21.15.6:0/2723050112 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.981+0000 7f50a9400640 1 --2- 172.21.15.6:0/2723050112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a415a2d0 0x7f50a40ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.981+0000 7f50a8a00640 1 -- 172.21.15.6:0/2723050112 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690697020 0 0) 0x7f50a40cdf80 con 0x7f50a40cd8d0 2024-03-22T13:12:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.981+0000 7f50a8a00640 1 -- 172.21.15.6:0/2723050112 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508c003520 con 0x7f50a40cd8d0 2024-03-22T13:12:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.984+0000 7f50a9400640 1 -- 172.21.15.6:0/2723050112 >> v1:172.21.15.92:6789/0 conn(0x7f50a415aeb0 legacy=0x7f50a40cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.984+0000 7f50a9400640 1 -- 172.21.15.6:0/2723050112 >> v1:172.21.15.6:6789/0 conn(0x7f50a40cd8d0 legacy=0x7f50a406d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.984+0000 7f50a9400640 1 -- 172.21.15.6:0/2723050112 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a40cf120 con 0x7f50a415a2d0 2024-03-22T13:12:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.988+0000 7f50a9400640 1 --2- 172.21.15.6:0/2723050112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a415a2d0 0x7f50a40ce240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f50940097d0 tx=0x7f509402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6290800def4d99bf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.990+0000 7f50a8a00640 1 -- 172.21.15.6:0/2723050112 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094035020 con 0x7f50a415a2d0 2024-03-22T13:12:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.992+0000 7f50a8a00640 1 -- 172.21.15.6:0/2723050112 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5094031890 con 0x7f50a415a2d0 2024-03-22T13:12:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.992+0000 7f50a8a00640 1 -- 172.21.15.6:0/2723050112 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094031c40 con 0x7f50a415a2d0 2024-03-22T13:12:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.993+0000 7f50ab200640 1 -- 172.21.15.6:0/2723050112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a415a2d0 msgr2=0x7f50a40ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.993+0000 7f50ab200640 1 --2- 172.21.15.6:0/2723050112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a415a2d0 0x7f50a40ce240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f50940097d0 tx=0x7f509402b930 comp rx=0 tx=0).stop 2024-03-22T13:12:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.994+0000 7f50ab200640 1 -- 172.21.15.6:0/2723050112 shutdown_connections 2024-03-22T13:12:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.994+0000 7f50ab200640 1 --2- 172.21.15.6:0/2723050112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a415a2d0 0x7f50a40ce240 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.994+0000 7f50ab200640 1 -- 172.21.15.6:0/2723050112 >> 172.21.15.6:0/2723050112 conn(0x7f50a406c8d0 msgr2=0x7f50a4066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.994+0000 7f50ab200640 1 -- 172.21.15.6:0/2723050112 shutdown_connections 2024-03-22T13:12:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.994+0000 7f50ab200640 1 -- 172.21.15.6:0/2723050112 wait complete. 2024-03-22T13:12:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.996+0000 7f50ab200640 1 Processor -- start 2024-03-22T13:12:14.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.997+0000 7f50ab200640 1 -- start start 2024-03-22T13:12:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.998+0000 7f50ab200640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a40cd8d0 0x7f50a4150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.998+0000 7f50ab200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50a415a2d0 0x7f50a406d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.998+0000 7f50ab200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50a415aeb0 0x7f50a41efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.998+0000 7f50ab200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a4154a90 con 0x7f50a415aeb0 2024-03-22T13:12:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.998+0000 7f50ab200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a415b290 con 0x7f50a415a2d0 2024-03-22T13:12:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.998+0000 7f50ab200640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a40cfa60 con 0x7f50a40cd8d0 2024-03-22T13:12:14.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.999+0000 7f50a9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a40cd8d0 0x7f50a4150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.999+0000 7f50aa800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50a415aeb0 0x7f50a41efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:14.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.999+0000 7f50a9e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a40cd8d0 0x7f50a4150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54464/0 (socket says 172.21.15.6:54464) 2024-03-22T13:12:14.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:14.999+0000 7f50a9e00640 1 -- 172.21.15.6:0/1926081279 learned_addr learned my addr 172.21.15.6:0/1926081279 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:15.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.004+0000 7f50a9e00640 1 -- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50a415a2d0 msgr2=0x7f50a406d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.004+0000 7f50a9e00640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50a415a2d0 0x7f50a406d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.004+0000 7f50a9e00640 1 -- 172.21.15.6:0/1926081279 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50a415aeb0 msgr2=0x7f50a41efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.004+0000 7f50a9e00640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50a415aeb0 0x7f50a41efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.004+0000 7f50a9e00640 1 -- 172.21.15.6:0/1926081279 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a41f3210 con 0x7f50a40cd8d0 2024-03-22T13:12:15.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.008+0000 7f50a9e00640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a40cd8d0 0x7f50a4150900 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f509800b410 tx=0x7f509800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.013+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098015020 con 0x7f50a40cd8d0 2024-03-22T13:12:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.013+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5098011830 con 0x7f50a40cd8d0 2024-03-22T13:12:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.013+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098011c20 con 0x7f50a40cd8d0 2024-03-22T13:12:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.013+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a415e5c0 con 0x7f50a40cd8d0 2024-03-22T13:12:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.013+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a41f3760 con 0x7f50a40cd8d0 2024-03-22T13:12:15.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.015+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50a4075650 con 0x7f50a40cd8d0 2024-03-22T13:12:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.045+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5098018020 con 0x7f50a40cd8d0 2024-03-22T13:12:15.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.046+0000 7f509e000640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f508806bc20 0x7f508806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.046+0000 7f50a9400640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f508806bc20 0x7f508806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.047+0000 7f50a9400640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f508806bc20 0x7f508806e0e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5094036000 tx=0x7f5094030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.051+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5098057400 con 0x7f50a40cd8d0 2024-03-22T13:12:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.051+0000 7f509e000640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7f50880757d0 0x7f5088077c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.051+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 --> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5098006730 con 0x7f50880757d0 2024-03-22T13:12:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.051+0000 7f50aa800640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7f50880757d0 0x7f5088077c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.052+0000 7f50aa800640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7f50880757d0 0x7f5088077c70 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.054+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 <== osd.6 v2:172.21.15.92:6802/3646124871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5098006730 con 0x7f50880757d0 2024-03-22T13:12:15.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.072+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 --> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f50a40cf6b0 con 0x7f50880757d0 2024-03-22T13:12:15.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.072+0000 7f509e000640 1 -- 172.21.15.6:0/1926081279 <== osd.6 v2:172.21.15.92:6802/3646124871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f50a40cf6b0 con 0x7f50880757d0 2024-03-22T13:12:15.073 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:15.073 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:15.073 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:15.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.073+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7f50880757d0 msgr2=0x7f5088077c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.073+0000 7f50ab200640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6802/3646124871,v1:172.21.15.92:6803/3646124871] conn(0x7f50880757d0 0x7f5088077c70 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.073+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f508806bc20 msgr2=0x7f508806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.073+0000 7f50ab200640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f508806bc20 0x7f508806e0e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5094036000 tx=0x7f5094030000 comp rx=0 tx=0).stop 2024-03-22T13:12:15.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.073+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a40cd8d0 msgr2=0x7f50a4150900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.073+0000 7f50ab200640 1 --2- 172.21.15.6:0/1926081279 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50a40cd8d0 0x7f50a4150900 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f509800b410 tx=0x7f509800b8d0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.073+0000 7f50aa800640 1 -- 172.21.15.6:0/1926081279 reap_dead start 2024-03-22T13:12:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.074+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 shutdown_connections 2024-03-22T13:12:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.074+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 >> 172.21.15.6:0/1926081279 conn(0x7f50a406c8d0 msgr2=0x7f50a4066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.074+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 shutdown_connections 2024-03-22T13:12:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.074+0000 7f50ab200640 1 -- 172.21.15.6:0/1926081279 wait complete. 2024-03-22T13:12:15.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-22T13:12:15.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.286+0000 7f0ae1800640 1 Processor -- start 2024-03-22T13:12:15.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.286+0000 7f0ae1800640 1 -- start start 2024-03-22T13:12:15.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.288+0000 7f0ae1800640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0adc0c4b90 0x7f0adc0cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.288+0000 7f0ae1800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0adc0c5570 0x7f0adc0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.288+0000 7f0ae1800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0adc0c5270 con 0x7f0adc1547f0 2024-03-22T13:12:15.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.288+0000 7f0ae1800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0adc0c5020 con 0x7f0adc0c5570 2024-03-22T13:12:15.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.288+0000 7f0ae1800640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0adc15e380 con 0x7f0adc0c4b90 2024-03-22T13:12:15.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.288+0000 7f0ada000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0adc0c4b90 0x7f0adc0cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.288+0000 7f0adb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0adc0c5570 0x7f0adc0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.289+0000 7f0ada000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0adc0c4b90 0x7f0adc0cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.6:37618/0 (socket says 172.21.15.6:37618) 2024-03-22T13:12:15.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.289+0000 7f0ada000640 1 -- 172.21.15.6:0/3572627456 learned_addr learned my addr 172.21.15.6:0/3572627456 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.291+0000 7f0ada000640 1 -- 172.21.15.6:0/3572627456 >> v2:172.21.15.6:3300/0 conn(0x7f0adc0c5570 msgr2=0x7f0adc0ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.291+0000 7f0ada000640 1 --2- 172.21.15.6:0/3572627456 >> v2:172.21.15.6:3300/0 conn(0x7f0adc0c5570 0x7f0adc0ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.291+0000 7f0ada000640 1 -- 172.21.15.6:0/3572627456 >> v1:172.21.15.92:6789/0 conn(0x7f0adc1547f0 legacy=0x7f0adc0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.292+0000 7f0ada000640 1 -- 172.21.15.6:0/3572627456 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc15e660 con 0x7f0adc0c4b90 2024-03-22T13:12:15.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.296+0000 7f0ada000640 1 --2- 172.21.15.6:0/3572627456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c4b90 0x7f0adc0cc620 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0adc06b210 tx=0x7f0ac802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=180caf9cf18874b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:15.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.300+0000 7f0ad9600640 1 -- 172.21.15.6:0/3572627456 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8035020 con 0x7f0adc0c4b90 2024-03-22T13:12:15.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.302+0000 7f0ad9600640 1 -- 172.21.15.6:0/3572627456 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac8031870 con 0x7f0adc0c4b90 2024-03-22T13:12:15.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.302+0000 7f0ad9600640 1 -- 172.21.15.6:0/3572627456 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8031c40 con 0x7f0adc0c4b90 2024-03-22T13:12:15.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.303+0000 7f0ae1800640 1 -- 172.21.15.6:0/3572627456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c4b90 msgr2=0x7f0adc0cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.303+0000 7f0ae1800640 1 --2- 172.21.15.6:0/3572627456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c4b90 0x7f0adc0cc620 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0adc06b210 tx=0x7f0ac802b910 comp rx=0 tx=0).stop 2024-03-22T13:12:15.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.303+0000 7f0ae1800640 1 -- 172.21.15.6:0/3572627456 shutdown_connections 2024-03-22T13:12:15.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.303+0000 7f0ae1800640 1 --2- 172.21.15.6:0/3572627456 >> v2:172.21.15.6:3300/0 conn(0x7f0adc0c5570 0x7f0adc0ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.303+0000 7f0ae1800640 1 --2- 172.21.15.6:0/3572627456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c4b90 0x7f0adc0cc620 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.303+0000 7f0ae1800640 1 -- 172.21.15.6:0/3572627456 >> 172.21.15.6:0/3572627456 conn(0x7f0adc066e40 msgr2=0x7f0adc067250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:15.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.304+0000 7f0ae1800640 1 -- 172.21.15.6:0/3572627456 shutdown_connections 2024-03-22T13:12:15.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.304+0000 7f0ae1800640 1 -- 172.21.15.6:0/3572627456 wait complete. 2024-03-22T13:12:15.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.306+0000 7f0ae1800640 1 Processor -- start 2024-03-22T13:12:15.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.306+0000 7f0ae1800640 1 -- start start 2024-03-22T13:12:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.308+0000 7f0ae1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0adc0c4b90 0x7f0adc1f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.308+0000 7f0ae1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c5570 0x7f0adc1f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.308+0000 7f0ae1800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0adc1547f0 0x7f0adc1e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.308+0000 7f0ae1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc0ca5d0 con 0x7f0adc0c4b90 2024-03-22T13:12:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.308+0000 7f0ae1800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc0c5f30 con 0x7f0adc0c5570 2024-03-22T13:12:15.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.308+0000 7f0ae1800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc0ca750 con 0x7f0adc1547f0 2024-03-22T13:12:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.309+0000 7f0ada000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c5570 0x7f0adc1f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.309+0000 7f0adb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0adc1547f0 0x7f0adc1e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.309+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0adc0c4b90 0x7f0adc1f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.309+0000 7f0adb400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0adc1547f0 0x7f0adc1e5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54470/0 (socket says 172.21.15.6:54470) 2024-03-22T13:12:15.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.309+0000 7f0adb400640 1 -- 172.21.15.6:0/848744353 learned_addr learned my addr 172.21.15.6:0/848744353 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.313+0000 7f0adb400640 1 -- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c5570 msgr2=0x7f0adc1f8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.313+0000 7f0adb400640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c5570 0x7f0adc1f8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.313+0000 7f0adb400640 1 -- 172.21.15.6:0/848744353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0adc0c4b90 msgr2=0x7f0adc1f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.313+0000 7f0adb400640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0adc0c4b90 0x7f0adc1f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.313+0000 7f0adb400640 1 -- 172.21.15.6:0/848744353 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc0b4540 con 0x7f0adc1547f0 2024-03-22T13:12:15.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.315+0000 7f0adb400640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0adc1547f0 0x7f0adc1e5cf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400efc0 tx=0x7f0ad400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.320+0000 7f0ad2a00640 1 -- 172.21.15.6:0/848744353 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4018020 con 0x7f0adc1547f0 2024-03-22T13:12:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.320+0000 7f0ad2a00640 1 -- 172.21.15.6:0/848744353 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad4014480 con 0x7f0adc1547f0 2024-03-22T13:12:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.320+0000 7f0ad2a00640 1 -- 172.21.15.6:0/848744353 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4014870 con 0x7f0adc1547f0 2024-03-22T13:12:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.320+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0adc0b4820 con 0x7f0adc1547f0 2024-03-22T13:12:15.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.320+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0adc1f82a0 con 0x7f0adc1547f0 2024-03-22T13:12:15.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.322+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0adc0757d0 con 0x7f0adc1547f0 2024-03-22T13:12:15.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.354+0000 7f0ad2a00640 1 -- 172.21.15.6:0/848744353 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0ad401b020 con 0x7f0adc1547f0 2024-03-22T13:12:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.355+0000 7f0ad2a00640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0abc06bc20 0x7f0abc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:15.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.356+0000 7f0adaa00640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0abc06bc20 0x7f0abc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:15.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.356+0000 7f0adaa00640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0abc06bc20 0x7f0abc06e0e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0acc001780 tx=0x7f0acc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:15.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.599+0000 7f0ad2a00640 1 -- 172.21.15.6:0/848744353 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f0ad40583c0 con 0x7f0adc1547f0 2024-03-22T13:12:15.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.618+0000 7f0ad2a00640 1 -- 172.21.15.6:0/848744353 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0ad4022020 con 0x7f0adc1547f0 2024-03-22T13:12:15.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.830+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f0adc0cbf30 con 0x7f0abc06bc20 2024-03-22T13:12:15.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.831+0000 7f0ad2a00640 1 -- 172.21.15.6:0/848744353 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0adc0cbf30 con 0x7f0abc06bc20 2024-03-22T13:12:15.830 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 6 to deep-scrub 2024-03-22T13:12:15.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0abc06bc20 msgr2=0x7f0abc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0abc06bc20 0x7f0abc06e0e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0acc001780 tx=0x7f0acc009040 comp rx=0 tx=0).stop 2024-03-22T13:12:15.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0adc1547f0 msgr2=0x7f0adc1e5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:15.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0adc1547f0 0x7f0adc1e5cf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400efc0 tx=0x7f0ad400c490 comp rx=0 tx=0).stop 2024-03-22T13:12:15.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 shutdown_connections 2024-03-22T13:12:15.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0adc1547f0 0x7f0adc1e5cf0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f0abc06bc20 0x7f0abc06e0e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0adc0c5570 0x7f0adc1f8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 --2- 172.21.15.6:0/848744353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0adc0c4b90 0x7f0adc1f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:15.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 >> 172.21.15.6:0/848744353 conn(0x7f0adc066e40 msgr2=0x7f0adc067220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:15.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 shutdown_connections 2024-03-22T13:12:15.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:15.834+0000 7f0ae1800640 1 -- 172.21.15.6:0/848744353 wait complete. 2024-03-22T13:12:15.860 INFO:tasks.ceph:Scrubbing osd.7 2024-03-22T13:12:15.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-22T13:12:16.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.062+0000 7f4d88a00640 1 Processor -- start 2024-03-22T13:12:16.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.062+0000 7f4d88a00640 1 -- start start 2024-03-22T13:12:16.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.064+0000 7f4d88a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d8415c220 0x7f4d841580f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.064+0000 7f4d88a00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4d84158630 0x7f4d840c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.064+0000 7f4d88a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d840cb2b0 con 0x7f4d84152480 2024-03-22T13:12:16.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.064+0000 7f4d88a00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d840c9940 con 0x7f4d84158630 2024-03-22T13:12:16.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.064+0000 7f4d88a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8415c660 con 0x7f4d8415c220 2024-03-22T13:12:16.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.070+0000 7f4d81400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d8415c220 0x7f4d841580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.071+0000 7f4d81400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d8415c220 0x7f4d841580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54480/0 (socket says 172.21.15.6:54480) 2024-03-22T13:12:16.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.071+0000 7f4d81400640 1 -- 172.21.15.6:0/2462206656 learned_addr learned my addr 172.21.15.6:0/2462206656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.076+0000 7f4d81400640 1 -- 172.21.15.6:0/2462206656 >> v2:172.21.15.92:3300/0 conn(0x7f4d84158630 msgr2=0x7f4d840c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.076+0000 7f4d81400640 1 --2- 172.21.15.6:0/2462206656 >> v2:172.21.15.92:3300/0 conn(0x7f4d84158630 0x7f4d840c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.076+0000 7f4d81400640 1 -- 172.21.15.6:0/2462206656 >> v1:172.21.15.6:6789/0 conn(0x7f4d84152480 legacy=0x7f4d840caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.076+0000 7f4d81400640 1 -- 172.21.15.6:0/2462206656 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d840c2e50 con 0x7f4d8415c220 2024-03-22T13:12:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.080+0000 7f4d81400640 1 --2- 172.21.15.6:0/2462206656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d8415c220 0x7f4d841580f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4d700098a0 tx=0x7f4d7002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d108bbc0649d2e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.182+0000 7f4d80a00640 1 -- 172.21.15.6:0/2462206656 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70035020 con 0x7f4d8415c220 2024-03-22T13:12:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.183+0000 7f4d80a00640 1 -- 172.21.15.6:0/2462206656 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70031870 con 0x7f4d8415c220 2024-03-22T13:12:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.183+0000 7f4d80a00640 1 -- 172.21.15.6:0/2462206656 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70031c40 con 0x7f4d8415c220 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 -- 172.21.15.6:0/2462206656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d8415c220 msgr2=0x7f4d841580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 --2- 172.21.15.6:0/2462206656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d8415c220 0x7f4d841580f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4d700098a0 tx=0x7f4d7002b910 comp rx=0 tx=0).stop 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 -- 172.21.15.6:0/2462206656 shutdown_connections 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 --2- 172.21.15.6:0/2462206656 >> v2:172.21.15.92:3300/0 conn(0x7f4d84158630 0x7f4d840c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 --2- 172.21.15.6:0/2462206656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d8415c220 0x7f4d841580f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 -- 172.21.15.6:0/2462206656 >> 172.21.15.6:0/2462206656 conn(0x7f4d840714a0 msgr2=0x7f4d84071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 -- 172.21.15.6:0/2462206656 shutdown_connections 2024-03-22T13:12:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.184+0000 7f4d88a00640 1 -- 172.21.15.6:0/2462206656 wait complete. 2024-03-22T13:12:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.188+0000 7f4d88a00640 1 Processor -- start 2024-03-22T13:12:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.188+0000 7f4d88a00640 1 -- start start 2024-03-22T13:12:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.189+0000 7f4d88a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d84152480 0x7f4d840cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.189+0000 7f4d88a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d84158630 0x7f4d840cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.189+0000 7f4d88a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8415c220 0x7f4d840cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.189+0000 7f4d88a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d84158df0 con 0x7f4d84152480 2024-03-22T13:12:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.189+0000 7f4d88a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d84152860 con 0x7f4d8415c220 2024-03-22T13:12:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.189+0000 7f4d88a00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840ca0c0 con 0x7f4d84158630 2024-03-22T13:12:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.189+0000 7f4d81400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d84158630 0x7f4d840cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.190+0000 7f4d81400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d84158630 0x7f4d840cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54482/0 (socket says 172.21.15.6:54482) 2024-03-22T13:12:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.190+0000 7f4d81400640 1 -- 172.21.15.6:0/2327879605 learned_addr learned my addr 172.21.15.6:0/2327879605 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.190+0000 7f4d82800640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8415c220 0x7f4d840cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.195+0000 7f4d81400640 1 -- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8415c220 msgr2=0x7f4d840cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.195+0000 7f4d81400640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8415c220 0x7f4d840cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.195+0000 7f4d81400640 1 -- 172.21.15.6:0/2327879605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d84152480 msgr2=0x7f4d840cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.195+0000 7f4d81400640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d84152480 0x7f4d840cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.195+0000 7f4d81400640 1 -- 172.21.15.6:0/2327879605 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d840cca50 con 0x7f4d84158630 2024-03-22T13:12:16.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.199+0000 7f4d81400640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d84158630 0x7f4d840cd1f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4d700090b0 tx=0x7f4d700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.203+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d700443f0 con 0x7f4d84158630 2024-03-22T13:12:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.203+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70044590 con 0x7f4d84158630 2024-03-22T13:12:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.203+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70044980 con 0x7f4d84158630 2024-03-22T13:12:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.203+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8414d350 con 0x7f4d84158630 2024-03-22T13:12:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.203+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d841f02c0 con 0x7f4d84158630 2024-03-22T13:12:16.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.204+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d8414d630 con 0x7f4d84158630 2024-03-22T13:12:16.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.227+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d7004b060 con 0x7f4d84158630 2024-03-22T13:12:16.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.228+0000 7f4d7a000640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d6406bc50 0x7f4d6406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.229+0000 7f4d81e00640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d6406bc50 0x7f4d6406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.229+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f4d70079250 con 0x7f4d84158630 2024-03-22T13:12:16.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.229+0000 7f4d7a000640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f4d64075880 0x7f4d64077d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.229+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 --> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d70033e90 con 0x7f4d64075880 2024-03-22T13:12:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.230+0000 7f4d82800640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f4d64075880 0x7f4d64077d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.230+0000 7f4d81e00640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d6406bc50 0x7f4d6406e110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4d74000f60 tx=0x7f4d740079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.230+0000 7f4d82800640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f4d64075880 0x7f4d64077d20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.232+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 <== osd.7 v2:172.21.15.92:6818/1078860650 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4d70033e90 con 0x7f4d64075880 2024-03-22T13:12:16.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.262+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 --> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4d840757d0 con 0x7f4d64075880 2024-03-22T13:12:16.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.262+0000 7f4d7a000640 1 -- 172.21.15.6:0/2327879605 <== osd.7 v2:172.21.15.92:6818/1078860650 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4d840757d0 con 0x7f4d64075880 2024-03-22T13:12:16.262 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-22T13:12:16.263 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-22T13:12:16.263 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f4d64075880 msgr2=0x7f4d64077d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6818/1078860650,v1:172.21.15.92:6820/1078860650] conn(0x7f4d64075880 0x7f4d64077d20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d6406bc50 msgr2=0x7f4d6406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f4d6406bc50 0x7f4d6406e110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4d74000f60 tx=0x7f4d740079d0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d84158630 msgr2=0x7f4d840cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 --2- 172.21.15.6:0/2327879605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d84158630 0x7f4d840cd1f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4d700090b0 tx=0x7f4d700046c0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d82800640 1 -- 172.21.15.6:0/2327879605 reap_dead start 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 shutdown_connections 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.263+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 >> 172.21.15.6:0/2327879605 conn(0x7f4d840714a0 msgr2=0x7f4d840caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.264+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 shutdown_connections 2024-03-22T13:12:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.264+0000 7f4d88a00640 1 -- 172.21.15.6:0/2327879605 wait complete. 2024-03-22T13:12:16.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-22T13:12:16.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.467+0000 7f5cdde00640 1 Processor -- start 2024-03-22T13:12:16.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.467+0000 7f5cdde00640 1 -- start start 2024-03-22T13:12:16.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.468+0000 7f5cdde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cc7e0 0x7f5cd80ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.468+0000 7f5cdde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5cd80cef10 0x7f5cd8164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.468+0000 7f5cdde00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd80cf3a0 con 0x7f5cd80cc400 2024-03-22T13:12:16.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.469+0000 7f5cdde00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd806dd30 con 0x7f5cd80cef10 2024-03-22T13:12:16.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.469+0000 7f5cdde00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd815a670 con 0x7f5cd80cc7e0 2024-03-22T13:12:16.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.469+0000 7f5cd7e00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cc7e0 0x7f5cd80ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.469+0000 7f5cdca00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5cd80cc400 0x7f5cd80cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:49696/0 (socket says 172.21.15.6:49696) 2024-03-22T13:12:16.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.469+0000 7f5cdca00640 1 -- 172.21.15.6:0/3809196020 learned_addr learned my addr 172.21.15.6:0/3809196020 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:16.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.472+0000 7f5cd7400640 1 -- 172.21.15.6:0/3809196020 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887796377 0 0) 0x7f5cd80cf3a0 con 0x7f5cd80cc400 2024-03-22T13:12:16.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.472+0000 7f5cd7400640 1 -- 172.21.15.6:0/3809196020 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc0003520 con 0x7f5cd80cc400 2024-03-22T13:12:16.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.473+0000 7f5cd7e00640 1 -- 172.21.15.6:0/3809196020 >> v2:172.21.15.6:3300/0 conn(0x7f5cd80cef10 msgr2=0x7f5cd8164c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:16.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.473+0000 7f5cd7e00640 1 --2- 172.21.15.6:0/3809196020 >> v2:172.21.15.6:3300/0 conn(0x7f5cd80cef10 0x7f5cd8164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.473+0000 7f5cd7e00640 1 -- 172.21.15.6:0/3809196020 >> v1:172.21.15.92:6789/0 conn(0x7f5cd80cc400 legacy=0x7f5cd80cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.473+0000 7f5cd7e00640 1 -- 172.21.15.6:0/3809196020 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd815a930 con 0x7f5cd80cc7e0 2024-03-22T13:12:16.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.477+0000 7f5cd7e00640 1 --2- 172.21.15.6:0/3809196020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cc7e0 0x7f5cd80ce9d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40097a0 tx=0x7f5cc402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca18c9829a4b6703 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:16.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.482+0000 7f5cd7400640 1 -- 172.21.15.6:0/3809196020 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4031780 con 0x7f5cd80cc7e0 2024-03-22T13:12:16.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.482+0000 7f5cd7400640 1 -- 172.21.15.6:0/3809196020 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc4037070 con 0x7f5cd80cc7e0 2024-03-22T13:12:16.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.482+0000 7f5cd7400640 1 -- 172.21.15.6:0/3809196020 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4031b40 con 0x7f5cd80cc7e0 2024-03-22T13:12:16.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 -- 172.21.15.6:0/3809196020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cc7e0 msgr2=0x7f5cd80ce9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3809196020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cc7e0 0x7f5cd80ce9d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40097a0 tx=0x7f5cc402b820 comp rx=0 tx=0).stop 2024-03-22T13:12:16.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 -- 172.21.15.6:0/3809196020 shutdown_connections 2024-03-22T13:12:16.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3809196020 >> v2:172.21.15.6:3300/0 conn(0x7f5cd80cef10 0x7f5cd8164c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3809196020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cc7e0 0x7f5cd80ce9d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 -- 172.21.15.6:0/3809196020 >> 172.21.15.6:0/3809196020 conn(0x7f5cd80cf9f0 msgr2=0x7f5cd80cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:16.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 -- 172.21.15.6:0/3809196020 shutdown_connections 2024-03-22T13:12:16.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.483+0000 7f5cdde00640 1 -- 172.21.15.6:0/3809196020 wait complete. 2024-03-22T13:12:16.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.485+0000 7f5cdde00640 1 Processor -- start 2024-03-22T13:12:16.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.485+0000 7f5cdde00640 1 -- start start 2024-03-22T13:12:16.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.486+0000 7f5cdde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cd80cc400 0x7f5cd81f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.486+0000 7f5cdde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cd80cc7e0 0x7f5cd80c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.486+0000 7f5cdde00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cef10 0x7f5cd80b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.486+0000 7f5cdde00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd815b1d0 con 0x7f5cd80cc7e0 2024-03-22T13:12:16.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.486+0000 7f5cdde00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd815aef0 con 0x7f5cd80cc400 2024-03-22T13:12:16.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.486+0000 7f5cdde00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80ce1d0 con 0x7f5cd80cef10 2024-03-22T13:12:16.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.487+0000 7f5cd7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cd80cc7e0 0x7f5cd80c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.487+0000 7f5cdd400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cef10 0x7f5cd80b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.487+0000 7f5cdca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cd80cc400 0x7f5cd81f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.487+0000 7f5cd7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cd80cc7e0 0x7f5cd80c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48592/0 (socket says 172.21.15.6:48592) 2024-03-22T13:12:16.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.487+0000 7f5cd7e00640 1 -- 172.21.15.6:0/3569243245 learned_addr learned my addr 172.21.15.6:0/3569243245 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:16.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.487+0000 7f5cdd400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cef10 0x7f5cd80b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54492/0 (socket says 172.21.15.6:54492) 2024-03-22T13:12:16.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.490+0000 7f5cdca00640 1 -- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cef10 msgr2=0x7f5cd80b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.490+0000 7f5cdca00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cef10 0x7f5cd80b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.490+0000 7f5cdca00640 1 -- 172.21.15.6:0/3569243245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cd80cc7e0 msgr2=0x7f5cd80c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:16.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.490+0000 7f5cdca00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cd80cc7e0 0x7f5cd80c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:16.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.490+0000 7f5cdca00640 1 -- 172.21.15.6:0/3569243245 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd81e3780 con 0x7f5cd80cc400 2024-03-22T13:12:16.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.493+0000 7f5cdca00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cd80cc400 0x7f5cd81f0a60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c400 tx=0x7f5ccc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:16.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.497+0000 7f5cd4c00640 1 -- 172.21.15.6:0/3569243245 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc017070 con 0x7f5cd80cc400 2024-03-22T13:12:16.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.497+0000 7f5cd4c00640 1 -- 172.21.15.6:0/3569243245 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc005ce0 con 0x7f5cd80cc400 2024-03-22T13:12:16.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.497+0000 7f5cd4c00640 1 -- 172.21.15.6:0/3569243245 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0072f0 con 0x7f5cd80cc400 2024-03-22T13:12:16.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.497+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd80af6a0 con 0x7f5cd80cc400 2024-03-22T13:12:16.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.497+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd80afc60 con 0x7f5cd80cc400 2024-03-22T13:12:16.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.499+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd80af430 con 0x7f5cd80cc400 2024-03-22T13:12:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.524+0000 7f5cd4c00640 1 -- 172.21.15.6:0/3569243245 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5ccc01a070 con 0x7f5cd80cc400 2024-03-22T13:12:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.525+0000 7f5cd4c00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5cb806bc20 0x7f5cb806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.525+0000 7f5cd7e00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5cb806bc20 0x7f5cb806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:16.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.526+0000 7f5cd7e00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5cb806bc20 0x7f5cb806e0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4000c00 tx=0x7f5cc403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:16.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.785+0000 7f5cd4c00640 1 -- 172.21.15.6:0/3569243245 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5ccc056760 con 0x7f5cd80cc400 2024-03-22T13:12:16.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:16.796+0000 7f5cd4c00640 1 -- 172.21.15.6:0/3569243245 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5ccc021020 con 0x7f5cd80cc400 2024-03-22T13:12:17.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.004+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cd8075650 con 0x7f5cb806bc20 2024-03-22T13:12:17.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.005+0000 7f5cd4c00640 1 -- 172.21.15.6:0/3569243245 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5cd8075650 con 0x7f5cb806bc20 2024-03-22T13:12:17.005 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 7 to deep-scrub 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.008+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5cb806bc20 msgr2=0x7f5cb806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.008+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5cb806bc20 0x7f5cb806e0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4000c00 tx=0x7f5cc403d040 comp rx=0 tx=0).stop 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.008+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cd80cc400 msgr2=0x7f5cd81f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.008+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cd80cc400 0x7f5cd81f0a60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00c400 tx=0x7f5ccc00c8c0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 shutdown_connections 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5cd80cef10 0x7f5cd80b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cd80cc7e0 0x7f5cd80c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5cb806bc20 0x7f5cb806e0e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 --2- 172.21.15.6:0/3569243245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5cd80cc400 0x7f5cd81f0a60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 >> 172.21.15.6:0/3569243245 conn(0x7f5cd80cf9f0 msgr2=0x7f5cd8159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 shutdown_connections 2024-03-22T13:12:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.009+0000 7f5cdde00640 1 -- 172.21.15.6:0/3569243245 wait complete. 2024-03-22T13:12:17.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T13:12:17.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.203+0000 7f5b8fa00640 1 Processor -- start 2024-03-22T13:12:17.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.203+0000 7f5b8fa00640 1 -- start start 2024-03-22T13:12:17.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.204+0000 7f5b8fa00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5b8814d880 0x7f5b88072f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:17.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.205+0000 7f5b8fa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5b88073470 0x7f5b880c7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:17.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.205+0000 7f5b8fa00640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b88072750 con 0x7f5b880b2e10 2024-03-22T13:12:17.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.205+0000 7f5b8fa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b88072aa0 con 0x7f5b88073470 2024-03-22T13:12:17.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.205+0000 7f5b8fa00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b881574e0 con 0x7f5b8814d880 2024-03-22T13:12:17.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.205+0000 7f5b8d000640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5b8814d880 0x7f5b88072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:17.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.206+0000 7f5b8da00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5b880b2e10 0x7f5b8815a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:49704/0 (socket says 172.21.15.6:49704) 2024-03-22T13:12:17.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.206+0000 7f5b8da00640 1 -- 172.21.15.6:0/562961984 learned_addr learned my addr 172.21.15.6:0/562961984 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:17.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.210+0000 7f5b87e00640 1 --2- 172.21.15.6:0/562961984 >> v2:172.21.15.6:3300/0 conn(0x7f5b88073470 0x7f5b880c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.212+0000 7f5b8d000640 1 -- 172.21.15.6:0/562961984 >> v2:172.21.15.6:3300/0 conn(0x7f5b88073470 msgr2=0x7f5b880c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.212+0000 7f5b8d000640 1 --2- 172.21.15.6:0/562961984 >> v2:172.21.15.6:3300/0 conn(0x7f5b88073470 0x7f5b880c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.212+0000 7f5b8d000640 1 -- 172.21.15.6:0/562961984 >> v1:172.21.15.92:6789/0 conn(0x7f5b880b2e10 legacy=0x7f5b8815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.212+0000 7f5b8d000640 1 -- 172.21.15.6:0/562961984 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b881577c0 con 0x7f5b8814d880 2024-03-22T13:12:17.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.216+0000 7f5b8d000640 1 --2- 172.21.15.6:0/562961984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 0x7f5b88072f30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5b88072c20 tx=0x7f5b7c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc2584a255baf0c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:17.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.223+0000 7f5b87400640 1 -- 172.21.15.6:0/562961984 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c034070 con 0x7f5b8814d880 2024-03-22T13:12:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.226+0000 7f5b87400640 1 -- 172.21.15.6:0/562961984 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c030610 con 0x7f5b8814d880 2024-03-22T13:12:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.226+0000 7f5b87400640 1 -- 172.21.15.6:0/562961984 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0309e0 con 0x7f5b8814d880 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 -- 172.21.15.6:0/562961984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 msgr2=0x7f5b88072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/562961984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 0x7f5b88072f30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5b88072c20 tx=0x7f5b7c02b960 comp rx=0 tx=0).stop 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 -- 172.21.15.6:0/562961984 shutdown_connections 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/562961984 >> v2:172.21.15.6:3300/0 conn(0x7f5b88073470 0x7f5b880c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/562961984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 0x7f5b88072f30 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 -- 172.21.15.6:0/562961984 >> 172.21.15.6:0/562961984 conn(0x7f5b880ccb50 msgr2=0x7f5b880ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 -- 172.21.15.6:0/562961984 shutdown_connections 2024-03-22T13:12:17.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.227+0000 7f5b8fa00640 1 -- 172.21.15.6:0/562961984 wait complete. 2024-03-22T13:12:17.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.230+0000 7f5b8fa00640 1 Processor -- start 2024-03-22T13:12:17.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.230+0000 7f5b8fa00640 1 -- start start 2024-03-22T13:12:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.231+0000 7f5b8fa00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b88073470 0x7f5b88150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.231+0000 7f5b8fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b880b2e10 0x7f5b88153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.231+0000 7f5b8fa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 0x7f5b88154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.231+0000 7f5b8fa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8806fe30 con 0x7f5b880b2e10 2024-03-22T13:12:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.231+0000 7f5b8fa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88157d80 con 0x7f5b8814d880 2024-03-22T13:12:17.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.231+0000 7f5b8fa00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880c2970 con 0x7f5b88073470 2024-03-22T13:12:17.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.232+0000 7f5b8d000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b88073470 0x7f5b88150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:17.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.232+0000 7f5b8d000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b88073470 0x7f5b88150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:54502/0 (socket says 172.21.15.6:54502) 2024-03-22T13:12:17.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.232+0000 7f5b8d000640 1 -- 172.21.15.6:0/3324884119 learned_addr learned my addr 172.21.15.6:0/3324884119 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:17.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.237+0000 7f5b8d000640 1 -- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 msgr2=0x7f5b88154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.237+0000 7f5b8d000640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 0x7f5b88154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.237+0000 7f5b8d000640 1 -- 172.21.15.6:0/3324884119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b880b2e10 msgr2=0x7f5b88153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.237+0000 7f5b8d000640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b880b2e10 0x7f5b88153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.237+0000 7f5b8d000640 1 -- 172.21.15.6:0/3324884119 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b881546c0 con 0x7f5b88073470 2024-03-22T13:12:17.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.240+0000 7f5b8d000640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b88073470 0x7f5b88150460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:17.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.256+0000 7f5b84c00640 1 -- 172.21.15.6:0/3324884119 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c034040 con 0x7f5b88073470 2024-03-22T13:12:17.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.256+0000 7f5b84c00640 1 -- 172.21.15.6:0/3324884119 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c03aaa0 con 0x7f5b88073470 2024-03-22T13:12:17.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.256+0000 7f5b84c00640 1 -- 172.21.15.6:0/3324884119 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0443f0 con 0x7f5b88073470 2024-03-22T13:12:17.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.256+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b881ea8d0 con 0x7f5b88073470 2024-03-22T13:12:17.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.256+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881eae90 con 0x7f5b88073470 2024-03-22T13:12:17.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.257+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b880756c0 con 0x7f5b88073470 2024-03-22T13:12:17.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.643+0000 7f5b84c00640 1 -- 172.21.15.6:0/3324884119 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b7c037070 con 0x7f5b88073470 2024-03-22T13:12:17.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.645+0000 7f5b84c00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b6c06bc20 0x7f5b6c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:17.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.645+0000 7f5b84c00640 1 -- 172.21.15.6:0/3324884119 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f5b7c0ad380 con 0x7f5b88073470 2024-03-22T13:12:17.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.646+0000 7f5b87e00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b6c06bc20 0x7f5b6c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:17.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.647+0000 7f5b87e00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b6c06bc20 0x7f5b6c06e0e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b78006a90 tx=0x7f5b78006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.661+0000 7f5b84c00640 1 -- 172.21.15.6:0/3324884119 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b7c03f070 con 0x7f5b88073470 2024-03-22T13:12:17.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.874+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b88149480 con 0x7f5b6c06bc20 2024-03-22T13:12:17.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.883+0000 7f5b84c00640 1 -- 172.21.15.6:0/3324884119 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+330324 (secure 0 0 0) 0x7f5b88149480 con 0x7f5b6c06bc20 2024-03-22T13:12:17.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:12:17.890 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T13:12:17.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b6c06bc20 msgr2=0x7f5b6c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b6c06bc20 0x7f5b6c06e0e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b78006a90 tx=0x7f5b78006a20 comp rx=0 tx=0).stop 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b88073470 msgr2=0x7f5b88150460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b88073470 0x7f5b88150460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c0046c0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 shutdown_connections 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f5b6c06bc20 0x7f5b6c06e0e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b8814d880 0x7f5b88154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b880b2e10 0x7f5b88153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 --2- 172.21.15.6:0/3324884119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5b88073470 0x7f5b88150460 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 >> 172.21.15.6:0/3324884119 conn(0x7f5b880ccb50 msgr2=0x7f5b8814a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 shutdown_connections 2024-03-22T13:12:17.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:17.890+0000 7f5b8fa00640 1 -- 172.21.15.6:0/3324884119 wait complete. 2024-03-22T13:12:17.907 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-03-22T13:12:15.915383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":333,"ondisk_log_size":333,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219112,"kb_used_data":2976,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755608,"statfs":{"total":773094113280,"available":772869742592,"internally_reserved":0,"allocated":3047424,"data_stored":2312816,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003094"},"pg_stats":[{"pgid":"2.1f","version":"31'19","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:10.518751+0000","last_change":"2024-03-22T13:12:10.518800+0000","last_active":"2024-03-22T13:12:10.518751+0000","last_peered":"2024-03-22T13:12:10.518751+0000","last_clean":"2024-03-22T13:12:10.518751+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T13:12:10.518751+0000","last_undegraded":"2024-03-22T13:12:10.518751+0000","last_fullsized":"2024-03-22T13:12:10.518751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-03-22T13:12:10.518656+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-03-22T13:12:10.518656+0000","last_clean_scrub_stamp":"2024-03-22T13:12:10.518656+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:21:36.449884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:29.373147+0000","last_change":"2024-03-22T12:56:30.269337+0000","last_active":"2024-03-22T13:10:29.373147+0000","last_peered":"2024-03-22T13:10:29.373147+0000","last_clean":"2024-03-22T13:10:29.373147+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T13:10:29.373147+0000","last_undegraded":"2024-03-22T13:10:29.373147+0000","last_fullsized":"2024-03-22T13:10:29.373147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:09.378738+0000","last_change":"2024-03-22T12:56:37.236090+0000","last_active":"2024-03-22T13:11:09.378738+0000","last_peered":"2024-03-22T13:11:09.378738+0000","last_clean":"2024-03-22T13:11:09.378738+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T13:11:09.378738+0000","last_undegraded":"2024-03-22T13:11:09.378738+0000","last_fullsized":"2024-03-22T13:11:09.378738+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"30'2","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:29.373198+0000","last_change":"2024-03-22T12:56:28.782489+0000","last_active":"2024-03-22T13:10:29.373198+0000","last_peered":"2024-03-22T13:10:29.373198+0000","last_clean":"2024-03-22T13:10:29.373198+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T13:10:29.373198+0000","last_undegraded":"2024-03-22T13:10:29.373198+0000","last_fullsized":"2024-03-22T13:10:29.373198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:04.448847+0000","last_change":"2024-03-22T12:56:30.316689+0000","last_active":"2024-03-22T13:11:04.448847+0000","last_peered":"2024-03-22T13:11:04.448847+0000","last_clean":"2024-03-22T13:11:04.448847+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T13:11:04.448847+0000","last_undegraded":"2024-03-22T13:11:04.448847+0000","last_fullsized":"2024-03-22T13:11:04.448847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:09.486456+0000","last_change":"2024-03-22T12:56:37.050470+0000","last_active":"2024-03-22T13:11:09.486456+0000","last_peered":"2024-03-22T13:11:09.486456+0000","last_clean":"2024-03-22T13:11:09.486456+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T13:11:09.486456+0000","last_undegraded":"2024-03-22T13:11:09.486456+0000","last_fullsized":"2024-03-22T13:11:09.486456+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'12","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093797+0000","last_change":"2024-03-22T12:56:28.753096+0000","last_active":"2024-03-22T13:08:58.093797+0000","last_peered":"2024-03-22T13:08:58.093797+0000","last_clean":"2024-03-22T13:08:58.093797+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T13:08:58.093797+0000","last_undegraded":"2024-03-22T13:08:58.093797+0000","last_fullsized":"2024-03-22T13:08:58.093797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:09:44.640399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452477+0000","last_change":"2024-03-22T12:56:30.309879+0000","last_active":"2024-03-22T13:08:59.452477+0000","last_peered":"2024-03-22T13:08:59.452477+0000","last_clean":"2024-03-22T13:08:59.452477+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T13:08:59.452477+0000","last_undegraded":"2024-03-22T13:08:59.452477+0000","last_fullsized":"2024-03-22T13:08:59.452477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:22:02.573691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.408240+0000","last_change":"2024-03-22T12:56:37.046209+0000","last_active":"2024-03-22T13:11:39.408240+0000","last_peered":"2024-03-22T13:11:39.408240+0000","last_clean":"2024-03-22T13:11:39.408240+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T13:11:39.408240+0000","last_undegraded":"2024-03-22T13:11:39.408240+0000","last_fullsized":"2024-03-22T13:11:39.408240+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093761+0000","last_change":"2024-03-22T12:56:28.761429+0000","last_active":"2024-03-22T13:08:58.093761+0000","last_peered":"2024-03-22T13:08:58.093761+0000","last_clean":"2024-03-22T13:08:58.093761+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T13:08:58.093761+0000","last_undegraded":"2024-03-22T13:08:58.093761+0000","last_fullsized":"2024-03-22T13:08:58.093761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452502+0000","last_change":"2024-03-22T12:56:30.308356+0000","last_active":"2024-03-22T13:08:59.452502+0000","last_peered":"2024-03-22T13:08:59.452502+0000","last_clean":"2024-03-22T13:08:59.452502+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T13:08:59.452502+0000","last_undegraded":"2024-03-22T13:08:59.452502+0000","last_fullsized":"2024-03-22T13:08:59.452502+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:50:21.371278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.408221+0000","last_change":"2024-03-22T12:56:37.043714+0000","last_active":"2024-03-22T13:11:39.408221+0000","last_peered":"2024-03-22T13:11:39.408221+0000","last_clean":"2024-03-22T13:11:39.408221+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T13:11:39.408221+0000","last_undegraded":"2024-03-22T13:11:39.408221+0000","last_fullsized":"2024-03-22T13:11:39.408221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049612+0000","last_change":"2024-03-22T12:56:30.305661+0000","last_active":"2024-03-22T13:09:33.049612+0000","last_peered":"2024-03-22T13:09:33.049612+0000","last_clean":"2024-03-22T13:09:33.049612+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T13:09:33.049612+0000","last_undegraded":"2024-03-22T13:09:33.049612+0000","last_fullsized":"2024-03-22T13:09:33.049612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:48:12.716797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:58.099614+0000","last_change":"2024-03-22T12:56:28.754526+0000","last_active":"2024-03-22T13:09:58.099614+0000","last_peered":"2024-03-22T13:09:58.099614+0000","last_clean":"2024-03-22T13:09:58.099614+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T13:09:58.099614+0000","last_undegraded":"2024-03-22T13:09:58.099614+0000","last_fullsized":"2024-03-22T13:09:58.099614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:44:45.451977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:09.481154+0000","last_change":"2024-03-22T12:56:37.049682+0000","last_active":"2024-03-22T13:10:09.481154+0000","last_peered":"2024-03-22T13:10:09.481154+0000","last_clean":"2024-03-22T13:10:09.481154+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T13:10:09.481154+0000","last_undegraded":"2024-03-22T13:10:09.481154+0000","last_fullsized":"2024-03-22T13:10:09.481154+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.482774+0000","last_change":"2024-03-22T12:56:30.307234+0000","last_active":"2024-03-22T13:12:12.482774+0000","last_peered":"2024-03-22T13:12:12.482774+0000","last_clean":"2024-03-22T13:12:12.482774+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T13:12:12.482774+0000","last_undegraded":"2024-03-22T13:12:12.482774+0000","last_fullsized":"2024-03-22T13:12:12.482774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"31'4","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.560555+0000","last_change":"2024-03-22T12:56:28.772704+0000","last_active":"2024-03-22T13:11:39.560555+0000","last_peered":"2024-03-22T13:11:39.560555+0000","last_clean":"2024-03-22T13:11:39.560555+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T13:11:39.560555+0000","last_undegraded":"2024-03-22T13:11:39.560555+0000","last_fullsized":"2024-03-22T13:11:39.560555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:09.481121+0000","last_change":"2024-03-22T12:56:37.046134+0000","last_active":"2024-03-22T13:10:09.481121+0000","last_peered":"2024-03-22T13:10:09.481121+0000","last_clean":"2024-03-22T13:10:09.481121+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T13:10:09.481121+0000","last_undegraded":"2024-03-22T13:10:09.481121+0000","last_fullsized":"2024-03-22T13:10:09.481121+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.382511+0000","last_change":"2024-03-22T12:56:30.270214+0000","last_active":"2024-03-22T13:11:39.382511+0000","last_peered":"2024-03-22T13:11:39.382511+0000","last_clean":"2024-03-22T13:11:39.382511+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T13:11:39.382511+0000","last_undegraded":"2024-03-22T13:11:39.382511+0000","last_fullsized":"2024-03-22T13:11:39.382511+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"31'11","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:11.895621+0000","last_change":"2024-03-22T13:12:11.895680+0000","last_active":"2024-03-22T13:12:11.895621+0000","last_peered":"2024-03-22T13:12:11.895621+0000","last_clean":"2024-03-22T13:12:11.895621+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T13:12:11.895621+0000","last_undegraded":"2024-03-22T13:12:11.895621+0000","last_fullsized":"2024-03-22T13:12:11.895621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-03-22T13:12:11.895497+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-03-22T13:12:11.895497+0000","last_clean_scrub_stamp":"2024-03-22T13:12:11.895497+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:07:32.627951+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:09.548597+0000","last_change":"2024-03-22T12:56:37.238934+0000","last_active":"2024-03-22T13:10:09.548597+0000","last_peered":"2024-03-22T13:10:09.548597+0000","last_clean":"2024-03-22T13:10:09.548597+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T13:10:09.548597+0000","last_undegraded":"2024-03-22T13:10:09.548597+0000","last_fullsized":"2024-03-22T13:10:09.548597+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.408194+0000","last_change":"2024-03-22T12:56:30.318910+0000","last_active":"2024-03-22T13:11:39.408194+0000","last_peered":"2024-03-22T13:11:39.408194+0000","last_clean":"2024-03-22T13:11:39.408194+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T13:11:39.408194+0000","last_undegraded":"2024-03-22T13:11:39.408194+0000","last_fullsized":"2024-03-22T13:11:39.408194+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.096649+0000","last_change":"2024-03-22T12:56:28.767347+0000","last_active":"2024-03-22T13:09:28.096649+0000","last_peered":"2024-03-22T13:09:28.096649+0000","last_clean":"2024-03-22T13:09:28.096649+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T13:09:28.096649+0000","last_undegraded":"2024-03-22T13:09:28.096649+0000","last_fullsized":"2024-03-22T13:09:28.096649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:09.481170+0000","last_change":"2024-03-22T12:56:37.046021+0000","last_active":"2024-03-22T13:10:09.481170+0000","last_peered":"2024-03-22T13:10:09.481170+0000","last_clean":"2024-03-22T13:10:09.481170+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T13:10:09.481170+0000","last_undegraded":"2024-03-22T13:10:09.481170+0000","last_fullsized":"2024-03-22T13:10:09.481170+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.337452+0000","last_change":"2024-03-22T12:56:30.311256+0000","last_active":"2024-03-22T13:11:39.337452+0000","last_peered":"2024-03-22T13:11:39.337452+0000","last_clean":"2024-03-22T13:11:39.337452+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T13:11:39.337452+0000","last_undegraded":"2024-03-22T13:11:39.337452+0000","last_fullsized":"2024-03-22T13:11:39.337452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'7","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:28.049231+0000","last_change":"2024-03-22T12:56:28.754660+0000","last_active":"2024-03-22T13:09:28.049231+0000","last_peered":"2024-03-22T13:09:28.049231+0000","last_clean":"2024-03-22T13:09:28.049231+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T13:09:28.049231+0000","last_undegraded":"2024-03-22T13:09:28.049231+0000","last_fullsized":"2024-03-22T13:09:28.049231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:16:49.192884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:14.405903+0000","last_change":"2024-03-22T12:56:37.237969+0000","last_active":"2024-03-22T13:11:14.405903+0000","last_peered":"2024-03-22T13:11:14.405903+0000","last_clean":"2024-03-22T13:11:14.405903+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T13:11:14.405903+0000","last_undegraded":"2024-03-22T13:11:14.405903+0000","last_fullsized":"2024-03-22T13:11:14.405903+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.382607+0000","last_change":"2024-03-22T12:56:30.270841+0000","last_active":"2024-03-22T13:11:39.382607+0000","last_peered":"2024-03-22T13:11:39.382607+0000","last_clean":"2024-03-22T13:11:39.382607+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T13:11:39.382607+0000","last_undegraded":"2024-03-22T13:11:39.382607+0000","last_fullsized":"2024-03-22T13:11:39.382607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"31'11","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.919291+0000","last_change":"2024-03-22T13:12:12.919321+0000","last_active":"2024-03-22T13:12:12.919291+0000","last_peered":"2024-03-22T13:12:12.919291+0000","last_clean":"2024-03-22T13:12:12.919291+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T13:12:12.919291+0000","last_undegraded":"2024-03-22T13:12:12.919291+0000","last_fullsized":"2024-03-22T13:12:12.919291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-03-22T13:12:12.919229+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-03-22T13:12:12.919229+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.919229+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:09:23.443359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:39.374432+0000","last_change":"2024-03-22T12:56:37.233743+0000","last_active":"2024-03-22T13:10:39.374432+0000","last_peered":"2024-03-22T13:10:39.374432+0000","last_clean":"2024-03-22T13:10:39.374432+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T13:10:39.374432+0000","last_undegraded":"2024-03-22T13:10:39.374432+0000","last_fullsized":"2024-03-22T13:10:39.374432+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:04.555682+0000","last_change":"2024-03-22T12:56:30.318028+0000","last_active":"2024-03-22T13:11:04.555682+0000","last_peered":"2024-03-22T13:11:04.555682+0000","last_clean":"2024-03-22T13:11:04.555682+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T13:11:04.555682+0000","last_undegraded":"2024-03-22T13:11:04.555682+0000","last_fullsized":"2024-03-22T13:11:04.555682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.093813+0000","last_change":"2024-03-22T12:56:28.777248+0000","last_active":"2024-03-22T13:08:58.093813+0000","last_peered":"2024-03-22T13:08:58.093813+0000","last_clean":"2024-03-22T13:08:58.093813+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T13:08:58.093813+0000","last_undegraded":"2024-03-22T13:08:58.093813+0000","last_fullsized":"2024-03-22T13:08:58.093813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:23.338197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:14.334636+0000","last_change":"2024-03-22T12:56:37.237782+0000","last_active":"2024-03-22T13:11:14.334636+0000","last_peered":"2024-03-22T13:11:14.334636+0000","last_clean":"2024-03-22T13:11:14.334636+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T13:11:14.334636+0000","last_undegraded":"2024-03-22T13:11:14.334636+0000","last_fullsized":"2024-03-22T13:11:14.334636+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452512+0000","last_change":"2024-03-22T12:56:30.315399+0000","last_active":"2024-03-22T13:08:59.452512+0000","last_peered":"2024-03-22T13:08:59.452512+0000","last_clean":"2024-03-22T13:08:59.452512+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T13:08:59.452512+0000","last_undegraded":"2024-03-22T13:08:59.452512+0000","last_fullsized":"2024-03-22T13:08:59.452512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:54:12.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"31'2","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:04.555714+0000","last_change":"2024-03-22T12:56:28.780424+0000","last_active":"2024-03-22T13:11:04.555714+0000","last_peered":"2024-03-22T13:11:04.555714+0000","last_clean":"2024-03-22T13:11:04.555714+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T13:11:04.555714+0000","last_undegraded":"2024-03-22T13:11:04.555714+0000","last_fullsized":"2024-03-22T13:11:04.555714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:43.060678+0000","last_change":"2024-03-22T12:57:05.043991+0000","last_active":"2024-03-22T13:11:43.060678+0000","last_peered":"2024-03-22T13:11:43.060678+0000","last_clean":"2024-03-22T13:11:43.060678+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T13:11:43.060678+0000","last_undegraded":"2024-03-22T13:11:43.060678+0000","last_fullsized":"2024-03-22T13:11:43.060678+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:04.486044+0000","last_change":"2024-03-22T12:56:30.319309+0000","last_active":"2024-03-22T13:11:04.486044+0000","last_peered":"2024-03-22T13:11:04.486044+0000","last_clean":"2024-03-22T13:11:04.486044+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T13:11:04.486044+0000","last_undegraded":"2024-03-22T13:11:04.486044+0000","last_fullsized":"2024-03-22T13:11:04.486044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452447+0000","last_change":"2024-03-22T12:56:28.774220+0000","last_active":"2024-03-22T13:08:59.452447+0000","last_peered":"2024-03-22T13:08:59.452447+0000","last_clean":"2024-03-22T13:08:59.452447+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T13:08:59.452447+0000","last_undegraded":"2024-03-22T13:08:59.452447+0000","last_fullsized":"2024-03-22T13:08:59.452447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:17:22.569765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:14.557072+0000","last_change":"2024-03-22T12:56:37.057591+0000","last_active":"2024-03-22T13:11:14.557072+0000","last_peered":"2024-03-22T13:11:14.557072+0000","last_clean":"2024-03-22T13:11:14.557072+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T13:11:14.557072+0000","last_undegraded":"2024-03-22T13:11:14.557072+0000","last_fullsized":"2024-03-22T13:11:14.557072+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:29.551249+0000","last_change":"2024-03-22T12:56:30.315409+0000","last_active":"2024-03-22T13:10:29.551249+0000","last_peered":"2024-03-22T13:10:29.551249+0000","last_clean":"2024-03-22T13:10:29.551249+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T13:10:29.551249+0000","last_undegraded":"2024-03-22T13:10:29.551249+0000","last_fullsized":"2024-03-22T13:10:29.551249+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"31'8","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.270084+0000","last_change":"2024-03-22T13:12:12.270131+0000","last_active":"2024-03-22T13:12:12.270084+0000","last_peered":"2024-03-22T13:12:12.270084+0000","last_clean":"2024-03-22T13:12:12.270084+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T13:12:12.270084+0000","last_undegraded":"2024-03-22T13:12:12.270084+0000","last_fullsized":"2024-03-22T13:12:12.270084+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-03-22T13:12:12.269985+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-03-22T13:12:12.269985+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.269985+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:28.836151+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:08.047290+0000","last_change":"2024-03-22T12:57:04.949601+0000","last_active":"2024-03-22T13:10:08.047290+0000","last_peered":"2024-03-22T13:10:08.047290+0000","last_clean":"2024-03-22T13:10:08.047290+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T13:10:08.047290+0000","last_undegraded":"2024-03-22T13:10:08.047290+0000","last_fullsized":"2024-03-22T13:10:08.047290+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:04.448769+0000","last_change":"2024-03-22T12:56:30.313951+0000","last_active":"2024-03-22T13:11:04.448769+0000","last_peered":"2024-03-22T13:11:04.448769+0000","last_clean":"2024-03-22T13:11:04.448769+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T13:11:04.448769+0000","last_undegraded":"2024-03-22T13:11:04.448769+0000","last_fullsized":"2024-03-22T13:11:04.448769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:28.090677+0000","last_change":"2024-03-22T12:56:28.769417+0000","last_active":"2024-03-22T13:08:28.090677+0000","last_peered":"2024-03-22T13:08:28.090677+0000","last_clean":"2024-03-22T13:08:28.090677+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T13:08:28.090677+0000","last_undegraded":"2024-03-22T13:08:28.090677+0000","last_fullsized":"2024-03-22T13:08:28.090677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:49.338982+0000","last_change":"2024-03-22T12:56:37.235565+0000","last_active":"2024-03-22T13:11:49.338982+0000","last_peered":"2024-03-22T13:11:49.338982+0000","last_clean":"2024-03-22T13:11:49.338982+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T13:11:49.338982+0000","last_undegraded":"2024-03-22T13:11:49.338982+0000","last_fullsized":"2024-03-22T13:11:49.338982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:04.378220+0000","last_change":"2024-03-22T12:56:30.309106+0000","last_active":"2024-03-22T13:11:04.378220+0000","last_peered":"2024-03-22T13:11:04.378220+0000","last_clean":"2024-03-22T13:11:04.378220+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T13:11:04.378220+0000","last_undegraded":"2024-03-22T13:11:04.378220+0000","last_fullsized":"2024-03-22T13:11:04.378220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452461+0000","last_change":"2024-03-22T12:56:28.765742+0000","last_active":"2024-03-22T13:08:59.452461+0000","last_peered":"2024-03-22T13:08:59.452461+0000","last_clean":"2024-03-22T13:08:59.452461+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T13:08:59.452461+0000","last_undegraded":"2024-03-22T13:08:59.452461+0000","last_fullsized":"2024-03-22T13:08:59.452461+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:17.315376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":194,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:18.089652+0000","last_change":"2024-03-22T12:56:18.199667+0000","last_active":"2024-03-22T13:08:18.089652+0000","last_peered":"2024-03-22T13:08:18.089652+0000","last_clean":"2024-03-22T13:08:18.089652+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T13:08:18.089652+0000","last_undegraded":"2024-03-22T13:08:18.089652+0000","last_fullsized":"2024-03-22T13:08:18.089652+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:14.706607+0000","last_clean_scrub_stamp":"2024-03-22T12:56:14.706607+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:17:19.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047239+0000","last_change":"2024-03-22T12:56:37.235352+0000","last_active":"2024-03-22T13:09:08.047239+0000","last_peered":"2024-03-22T13:09:08.047239+0000","last_clean":"2024-03-22T13:09:08.047239+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T13:09:08.047239+0000","last_undegraded":"2024-03-22T13:09:08.047239+0000","last_fullsized":"2024-03-22T13:09:08.047239+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:35.216780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086769+0000","last_change":"2024-03-22T12:56:30.307126+0000","last_active":"2024-03-22T13:07:48.086769+0000","last_peered":"2024-03-22T13:07:48.086769+0000","last_clean":"2024-03-22T13:07:48.086769+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T13:07:48.086769+0000","last_undegraded":"2024-03-22T13:07:48.086769+0000","last_fullsized":"2024-03-22T13:07:48.086769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:23:28.198263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"31'6","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:11.421798+0000","last_change":"2024-03-22T12:56:28.764777+0000","last_active":"2024-03-22T13:12:11.421798+0000","last_peered":"2024-03-22T13:12:11.421798+0000","last_clean":"2024-03-22T13:12:11.421798+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T13:12:11.421798+0000","last_undegraded":"2024-03-22T13:12:11.421798+0000","last_fullsized":"2024-03-22T13:12:11.421798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:38.050005+0000","last_change":"2024-03-22T12:56:37.233888+0000","last_active":"2024-03-22T13:09:38.050005+0000","last_peered":"2024-03-22T13:09:38.050005+0000","last_clean":"2024-03-22T13:09:38.050005+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T13:09:38.050005+0000","last_undegraded":"2024-03-22T13:09:38.050005+0000","last_fullsized":"2024-03-22T13:09:38.050005+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:38.315172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:59.458076+0000","last_change":"2024-03-22T12:56:30.315105+0000","last_active":"2024-03-22T13:09:59.458076+0000","last_peered":"2024-03-22T13:09:59.458076+0000","last_clean":"2024-03-22T13:09:59.458076+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T13:09:59.458076+0000","last_undegraded":"2024-03-22T13:09:59.458076+0000","last_fullsized":"2024-03-22T13:09:59.458076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:49:33.758172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"31'4","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:10.291942+0000","last_change":"2024-03-22T12:56:28.771044+0000","last_active":"2024-03-22T13:12:10.291942+0000","last_peered":"2024-03-22T13:12:10.291942+0000","last_clean":"2024-03-22T13:12:10.291942+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T13:12:10.291942+0000","last_undegraded":"2024-03-22T13:12:10.291942+0000","last_fullsized":"2024-03-22T13:12:10.291942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:09.442100+0000","last_change":"2024-03-22T12:56:37.053158+0000","last_active":"2024-03-22T13:10:09.442100+0000","last_peered":"2024-03-22T13:10:09.442100+0000","last_clean":"2024-03-22T13:10:09.442100+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T13:10:09.442100+0000","last_undegraded":"2024-03-22T13:10:09.442100+0000","last_fullsized":"2024-03-22T13:10:09.442100+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086812+0000","last_change":"2024-03-22T12:56:30.308574+0000","last_active":"2024-03-22T13:07:48.086812+0000","last_peered":"2024-03-22T13:07:48.086812+0000","last_clean":"2024-03-22T13:07:48.086812+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T13:07:48.086812+0000","last_undegraded":"2024-03-22T13:07:48.086812+0000","last_fullsized":"2024-03-22T13:07:48.086812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:41:48.291368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"31'6","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:11.421997+0000","last_change":"2024-03-22T12:56:28.766625+0000","last_active":"2024-03-22T13:12:11.421997+0000","last_peered":"2024-03-22T13:12:11.421997+0000","last_clean":"2024-03-22T13:12:11.421997+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T13:12:11.421997+0000","last_undegraded":"2024-03-22T13:12:11.421997+0000","last_fullsized":"2024-03-22T13:12:11.421997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:39.456333+0000","last_change":"2024-03-22T12:56:37.234189+0000","last_active":"2024-03-22T13:09:39.456333+0000","last_peered":"2024-03-22T13:09:39.456333+0000","last_clean":"2024-03-22T13:09:39.456333+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T13:09:39.456333+0000","last_undegraded":"2024-03-22T13:09:39.456333+0000","last_fullsized":"2024-03-22T13:09:39.456333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:54:15.232423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.086795+0000","last_change":"2024-03-22T12:56:30.308025+0000","last_active":"2024-03-22T13:07:48.086795+0000","last_peered":"2024-03-22T13:07:48.086795+0000","last_clean":"2024-03-22T13:07:48.086795+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T13:07:48.086795+0000","last_undegraded":"2024-03-22T13:07:48.086795+0000","last_fullsized":"2024-03-22T13:07:48.086795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:33:11.390503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:59.480244+0000","last_change":"2024-03-22T12:56:28.769703+0000","last_active":"2024-03-22T13:09:59.480244+0000","last_peered":"2024-03-22T13:09:59.480244+0000","last_clean":"2024-03-22T13:09:59.480244+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T13:09:59.480244+0000","last_undegraded":"2024-03-22T13:09:59.480244+0000","last_fullsized":"2024-03-22T13:09:59.480244+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:49.409207+0000","last_change":"2024-03-22T12:56:37.039340+0000","last_active":"2024-03-22T13:11:49.409207+0000","last_peered":"2024-03-22T13:11:49.409207+0000","last_clean":"2024-03-22T13:11:49.409207+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T13:11:49.409207+0000","last_undegraded":"2024-03-22T13:11:49.409207+0000","last_fullsized":"2024-03-22T13:11:49.409207+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089125+0000","last_change":"2024-03-22T12:56:30.309165+0000","last_active":"2024-03-22T13:08:13.089125+0000","last_peered":"2024-03-22T13:08:13.089125+0000","last_clean":"2024-03-22T13:08:13.089125+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T13:08:13.089125+0000","last_undegraded":"2024-03-22T13:08:13.089125+0000","last_fullsized":"2024-03-22T13:08:13.089125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:23:44.494899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"31'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:28.049834+0000","last_change":"2024-03-22T12:56:28.774915+0000","last_active":"2024-03-22T13:10:28.049834+0000","last_peered":"2024-03-22T13:10:28.049834+0000","last_clean":"2024-03-22T13:10:28.049834+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T13:10:28.049834+0000","last_undegraded":"2024-03-22T13:10:28.049834+0000","last_fullsized":"2024-03-22T13:10:28.049834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:09.449371+0000","last_change":"2024-03-22T12:56:37.049839+0000","last_active":"2024-03-22T13:11:09.449371+0000","last_peered":"2024-03-22T13:11:09.449371+0000","last_clean":"2024-03-22T13:11:09.449371+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T13:11:09.449371+0000","last_undegraded":"2024-03-22T13:11:09.449371+0000","last_fullsized":"2024-03-22T13:11:09.449371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.042504+0000","last_change":"2024-03-22T12:56:30.302584+0000","last_active":"2024-03-22T13:08:13.042504+0000","last_peered":"2024-03-22T13:08:13.042504+0000","last_clean":"2024-03-22T13:08:13.042504+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T13:08:13.042504+0000","last_undegraded":"2024-03-22T13:08:13.042504+0000","last_fullsized":"2024-03-22T13:08:13.042504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:26:20.058709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.041967+0000","last_change":"2024-03-22T12:56:28.754681+0000","last_active":"2024-03-22T13:08:08.041967+0000","last_peered":"2024-03-22T13:08:08.041967+0000","last_clean":"2024-03-22T13:08:08.041967+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T13:08:08.041967+0000","last_undegraded":"2024-03-22T13:08:08.041967+0000","last_fullsized":"2024-03-22T13:08:08.041967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:44.777526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:09.378791+0000","last_change":"2024-03-22T12:56:37.234858+0000","last_active":"2024-03-22T13:11:09.378791+0000","last_peered":"2024-03-22T13:11:09.378791+0000","last_clean":"2024-03-22T13:11:09.378791+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T13:11:09.378791+0000","last_undegraded":"2024-03-22T13:11:09.378791+0000","last_fullsized":"2024-03-22T13:11:09.378791+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:13.089153+0000","last_change":"2024-03-22T12:56:30.308102+0000","last_active":"2024-03-22T13:08:13.089153+0000","last_peered":"2024-03-22T13:08:13.089153+0000","last_clean":"2024-03-22T13:08:13.089153+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T13:08:13.089153+0000","last_undegraded":"2024-03-22T13:08:13.089153+0000","last_fullsized":"2024-03-22T13:08:13.089153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:47:34.177135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447799+0000","last_change":"2024-03-22T12:56:28.777102+0000","last_active":"2024-03-22T13:08:09.447799+0000","last_peered":"2024-03-22T13:08:09.447799+0000","last_clean":"2024-03-22T13:08:09.447799+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T13:08:09.447799+0000","last_undegraded":"2024-03-22T13:08:09.447799+0000","last_fullsized":"2024-03-22T13:08:09.447799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:10:33.238964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:39.374543+0000","last_change":"2024-03-22T12:57:04.949816+0000","last_active":"2024-03-22T13:10:39.374543+0000","last_peered":"2024-03-22T13:10:39.374543+0000","last_clean":"2024-03-22T13:10:39.374543+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T13:10:39.374543+0000","last_undegraded":"2024-03-22T13:10:39.374543+0000","last_fullsized":"2024-03-22T13:10:39.374543+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.536903+0000","last_change":"2024-03-22T13:12:12.536941+0000","last_active":"2024-03-22T13:12:12.536903+0000","last_peered":"2024-03-22T13:12:12.536903+0000","last_clean":"2024-03-22T13:12:12.536903+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T13:12:12.536903+0000","last_undegraded":"2024-03-22T13:12:12.536903+0000","last_fullsized":"2024-03-22T13:12:12.536903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:12.536765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:12.536765+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.536765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:09:23.060896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"31'15","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:09.527931+0000","last_change":"2024-03-22T13:12:09.527981+0000","last_active":"2024-03-22T13:12:09.527931+0000","last_peered":"2024-03-22T13:12:09.527931+0000","last_clean":"2024-03-22T13:12:09.527931+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T13:12:09.527931+0000","last_undegraded":"2024-03-22T13:12:09.527931+0000","last_fullsized":"2024-03-22T13:12:09.527931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-03-22T13:12:09.527826+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-03-22T13:12:09.527826+0000","last_clean_scrub_stamp":"2024-03-22T13:12:09.527826+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:45:45.449957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:44.451139+0000","last_change":"2024-03-22T12:56:37.235073+0000","last_active":"2024-03-22T13:08:44.451139+0000","last_peered":"2024-03-22T13:08:44.451139+0000","last_clean":"2024-03-22T13:08:44.451139+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T13:08:44.451139+0000","last_undegraded":"2024-03-22T13:08:44.451139+0000","last_fullsized":"2024-03-22T13:08:44.451139+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:27:43.061889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091724+0000","last_change":"2024-03-22T12:56:30.272047+0000","last_active":"2024-03-22T13:08:38.091724+0000","last_peered":"2024-03-22T13:08:38.091724+0000","last_clean":"2024-03-22T13:08:38.091724+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T13:08:38.091724+0000","last_undegraded":"2024-03-22T13:08:38.091724+0000","last_fullsized":"2024-03-22T13:08:38.091724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:53:31.296655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'14","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:08.042011+0000","last_change":"2024-03-22T12:56:28.759729+0000","last_active":"2024-03-22T13:08:08.042011+0000","last_peered":"2024-03-22T13:08:08.042011+0000","last_clean":"2024-03-22T13:08:08.042011+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T13:08:08.042011+0000","last_undegraded":"2024-03-22T13:08:08.042011+0000","last_fullsized":"2024-03-22T13:08:08.042011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:46:57.848367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:39.552565+0000","last_change":"2024-03-22T12:56:37.056949+0000","last_active":"2024-03-22T13:10:39.552565+0000","last_peered":"2024-03-22T13:10:39.552565+0000","last_clean":"2024-03-22T13:10:39.552565+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T13:10:39.552565+0000","last_undegraded":"2024-03-22T13:10:39.552565+0000","last_fullsized":"2024-03-22T13:10:39.552565+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:29.482888+0000","last_change":"2024-03-22T12:56:30.305244+0000","last_active":"2024-03-22T13:10:29.482888+0000","last_peered":"2024-03-22T13:10:29.482888+0000","last_clean":"2024-03-22T13:10:29.482888+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T13:10:29.482888+0000","last_undegraded":"2024-03-22T13:10:29.482888+0000","last_fullsized":"2024-03-22T13:10:29.482888+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:13.900173+0000","last_change":"2024-03-22T13:12:13.900222+0000","last_active":"2024-03-22T13:12:13.900173+0000","last_peered":"2024-03-22T13:12:13.900173+0000","last_clean":"2024-03-22T13:12:13.900173+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T13:12:13.900173+0000","last_undegraded":"2024-03-22T13:12:13.900173+0000","last_fullsized":"2024-03-22T13:12:13.900173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-03-22T13:12:13.900078+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-03-22T13:12:13.900078+0000","last_clean_scrub_stamp":"2024-03-22T13:12:13.900078+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:10:07.120053+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:39.483866+0000","last_change":"2024-03-22T12:56:37.038286+0000","last_active":"2024-03-22T13:10:39.483866+0000","last_peered":"2024-03-22T13:10:39.483866+0000","last_clean":"2024-03-22T13:10:39.483866+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T13:10:39.483866+0000","last_undegraded":"2024-03-22T13:10:39.483866+0000","last_fullsized":"2024-03-22T13:10:39.483866+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:59.448277+0000","last_change":"2024-03-22T12:56:30.308673+0000","last_active":"2024-03-22T13:10:59.448277+0000","last_peered":"2024-03-22T13:10:59.448277+0000","last_clean":"2024-03-22T13:10:59.448277+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T13:10:59.448277+0000","last_undegraded":"2024-03-22T13:10:59.448277+0000","last_fullsized":"2024-03-22T13:10:59.448277+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450282+0000","last_change":"2024-03-22T12:56:28.762179+0000","last_active":"2024-03-22T13:08:34.450282+0000","last_peered":"2024-03-22T13:08:34.450282+0000","last_clean":"2024-03-22T13:08:34.450282+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T13:08:34.450282+0000","last_undegraded":"2024-03-22T13:08:34.450282+0000","last_fullsized":"2024-03-22T13:08:34.450282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:37:31.911570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:39.374472+0000","last_change":"2024-03-22T12:56:37.046470+0000","last_active":"2024-03-22T13:10:39.374472+0000","last_peered":"2024-03-22T13:10:39.374472+0000","last_clean":"2024-03-22T13:10:39.374472+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T13:10:39.374472+0000","last_undegraded":"2024-03-22T13:10:39.374472+0000","last_fullsized":"2024-03-22T13:10:39.374472+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450270+0000","last_change":"2024-03-22T12:56:30.312593+0000","last_active":"2024-03-22T13:08:34.450270+0000","last_peered":"2024-03-22T13:08:34.450270+0000","last_clean":"2024-03-22T13:08:34.450270+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T13:08:34.450270+0000","last_undegraded":"2024-03-22T13:08:34.450270+0000","last_fullsized":"2024-03-22T13:08:34.450270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:37:49.797203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:33.044141+0000","last_change":"2024-03-22T12:56:28.762467+0000","last_active":"2024-03-22T13:08:33.044141+0000","last_peered":"2024-03-22T13:08:33.044141+0000","last_clean":"2024-03-22T13:08:33.044141+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T13:08:33.044141+0000","last_undegraded":"2024-03-22T13:08:33.044141+0000","last_fullsized":"2024-03-22T13:08:33.044141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:24:32.903883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:39.374500+0000","last_change":"2024-03-22T12:56:37.235273+0000","last_active":"2024-03-22T13:10:39.374500+0000","last_peered":"2024-03-22T13:10:39.374500+0000","last_clean":"2024-03-22T13:10:39.374500+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T13:10:39.374500+0000","last_undegraded":"2024-03-22T13:10:39.374500+0000","last_fullsized":"2024-03-22T13:10:39.374500+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:39.446282+0000","last_change":"2024-03-22T12:56:37.239640+0000","last_active":"2024-03-22T13:10:39.446282+0000","last_peered":"2024-03-22T13:10:39.446282+0000","last_clean":"2024-03-22T13:10:39.446282+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T13:10:39.446282+0000","last_undegraded":"2024-03-22T13:10:39.446282+0000","last_fullsized":"2024-03-22T13:10:39.446282+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:59.554978+0000","last_change":"2024-03-22T12:56:30.269943+0000","last_active":"2024-03-22T13:10:59.554978+0000","last_peered":"2024-03-22T13:10:59.554978+0000","last_clean":"2024-03-22T13:10:59.554978+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T13:10:59.554978+0000","last_undegraded":"2024-03-22T13:10:59.554978+0000","last_fullsized":"2024-03-22T13:10:59.554978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:34.450228+0000","last_change":"2024-03-22T12:56:28.761491+0000","last_active":"2024-03-22T13:08:34.450228+0000","last_peered":"2024-03-22T13:08:34.450228+0000","last_clean":"2024-03-22T13:08:34.450228+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T13:08:34.450228+0000","last_undegraded":"2024-03-22T13:08:34.450228+0000","last_fullsized":"2024-03-22T13:08:34.450228+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:30:47.612660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:08.047276+0000","last_change":"2024-03-22T12:56:37.051752+0000","last_active":"2024-03-22T13:09:08.047276+0000","last_peered":"2024-03-22T13:09:08.047276+0000","last_clean":"2024-03-22T13:09:08.047276+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T13:09:08.047276+0000","last_undegraded":"2024-03-22T13:09:08.047276+0000","last_fullsized":"2024-03-22T13:09:08.047276+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:47:08.411008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:59.377621+0000","last_change":"2024-03-22T12:56:30.298704+0000","last_active":"2024-03-22T13:10:59.377621+0000","last_peered":"2024-03-22T13:10:59.377621+0000","last_clean":"2024-03-22T13:10:59.377621+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T13:10:59.377621+0000","last_undegraded":"2024-03-22T13:10:59.377621+0000","last_fullsized":"2024-03-22T13:10:59.377621+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:59.377659+0000","last_change":"2024-03-22T12:56:28.756194+0000","last_active":"2024-03-22T13:10:59.377659+0000","last_peered":"2024-03-22T13:10:59.377659+0000","last_clean":"2024-03-22T13:10:59.377659+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T13:10:59.377659+0000","last_undegraded":"2024-03-22T13:10:59.377659+0000","last_fullsized":"2024-03-22T13:10:59.377659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:58.046243+0000","last_change":"2024-03-22T12:56:28.754733+0000","last_active":"2024-03-22T13:08:58.046243+0000","last_peered":"2024-03-22T13:08:58.046243+0000","last_clean":"2024-03-22T13:08:58.046243+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T13:08:58.046243+0000","last_undegraded":"2024-03-22T13:08:58.046243+0000","last_fullsized":"2024-03-22T13:08:58.046243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:37.717560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:08.064633+0000","last_change":"2024-03-22T12:57:05.045205+0000","last_active":"2024-03-22T13:12:08.064633+0000","last_peered":"2024-03-22T13:12:08.064633+0000","last_clean":"2024-03-22T13:12:08.064633+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T13:12:08.064633+0000","last_undegraded":"2024-03-22T13:12:08.064633+0000","last_fullsized":"2024-03-22T13:12:08.064633+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:38.059919+0000","last_change":"2024-03-22T12:56:37.063635+0000","last_active":"2024-03-22T13:11:38.059919+0000","last_peered":"2024-03-22T13:11:38.059919+0000","last_clean":"2024-03-22T13:11:38.059919+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T13:11:38.059919+0000","last_undegraded":"2024-03-22T13:11:38.059919+0000","last_fullsized":"2024-03-22T13:11:38.059919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:29.381459+0000","last_change":"2024-03-22T12:56:28.758267+0000","last_active":"2024-03-22T13:11:29.381459+0000","last_peered":"2024-03-22T13:11:29.381459+0000","last_clean":"2024-03-22T13:11:29.381459+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T13:11:29.381459+0000","last_undegraded":"2024-03-22T13:11:29.381459+0000","last_fullsized":"2024-03-22T13:11:29.381459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:27.224260+0000","last_clean_scrub_stamp":"2024-03-22T12:56:27.224260+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:29.407211+0000","last_change":"2024-03-22T12:56:30.311713+0000","last_active":"2024-03-22T13:11:29.407211+0000","last_peered":"2024-03-22T13:11:29.407211+0000","last_clean":"2024-03-22T13:11:29.407211+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T13:11:29.407211+0000","last_undegraded":"2024-03-22T13:11:29.407211+0000","last_fullsized":"2024-03-22T13:11:29.407211+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.382573+0000","last_change":"2024-03-22T12:56:37.052342+0000","last_active":"2024-03-22T13:11:39.382573+0000","last_peered":"2024-03-22T13:11:39.382573+0000","last_clean":"2024-03-22T13:11:39.382573+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T13:11:39.382573+0000","last_undegraded":"2024-03-22T13:11:39.382573+0000","last_fullsized":"2024-03-22T13:11:39.382573+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:35.512368+0000","last_clean_scrub_stamp":"2024-03-22T12:56:35.512368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.046714+0000","last_change":"2024-03-22T12:56:30.310512+0000","last_active":"2024-03-22T13:09:03.046714+0000","last_peered":"2024-03-22T13:09:03.046714+0000","last_clean":"2024-03-22T13:09:03.046714+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T13:09:03.046714+0000","last_undegraded":"2024-03-22T13:09:03.046714+0000","last_fullsized":"2024-03-22T13:09:03.046714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:49:21.713818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:08.055551+0000","last_change":"2024-03-22T12:57:04.188189+0000","last_active":"2024-03-22T13:11:08.055551+0000","last_peered":"2024-03-22T13:11:08.055551+0000","last_clean":"2024-03-22T13:11:08.055551+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T13:11:08.055551+0000","last_undegraded":"2024-03-22T13:11:08.055551+0000","last_fullsized":"2024-03-22T13:11:08.055551+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452433+0000","last_change":"2024-03-22T12:56:30.317891+0000","last_active":"2024-03-22T13:08:59.452433+0000","last_peered":"2024-03-22T13:08:59.452433+0000","last_clean":"2024-03-22T13:08:59.452433+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T13:08:59.452433+0000","last_undegraded":"2024-03-22T13:08:59.452433+0000","last_fullsized":"2024-03-22T13:08:59.452433+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:25:55.945700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:13.491821+0000","last_change":"2024-03-22T13:12:13.491856+0000","last_active":"2024-03-22T13:12:13.491821+0000","last_peered":"2024-03-22T13:12:13.491821+0000","last_clean":"2024-03-22T13:12:13.491821+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T13:12:13.491821+0000","last_undegraded":"2024-03-22T13:12:13.491821+0000","last_fullsized":"2024-03-22T13:12:13.491821+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:13.491717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:13.491717+0000","last_clean_scrub_stamp":"2024-03-22T13:12:13.491717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:10:06.711692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.382557+0000","last_change":"2024-03-22T12:56:30.311737+0000","last_active":"2024-03-22T13:11:39.382557+0000","last_peered":"2024-03-22T13:11:39.382557+0000","last_clean":"2024-03-22T13:11:39.382557+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T13:11:39.382557+0000","last_undegraded":"2024-03-22T13:11:39.382557+0000","last_fullsized":"2024-03-22T13:11:39.382557+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:39.560584+0000","last_change":"2024-03-22T12:56:30.273473+0000","last_active":"2024-03-22T13:11:39.560584+0000","last_peered":"2024-03-22T13:11:39.560584+0000","last_clean":"2024-03-22T13:11:39.560584+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T13:11:39.560584+0000","last_undegraded":"2024-03-22T13:11:39.560584+0000","last_fullsized":"2024-03-22T13:11:39.560584+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:33.049650+0000","last_change":"2024-03-22T12:56:30.227825+0000","last_active":"2024-03-22T13:09:33.049650+0000","last_peered":"2024-03-22T13:09:33.049650+0000","last_clean":"2024-03-22T13:09:33.049650+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T13:09:33.049650+0000","last_undegraded":"2024-03-22T13:09:33.049650+0000","last_fullsized":"2024-03-22T13:09:33.049650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:05:38.179330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:13.742657+0000","last_change":"2024-03-22T13:12:13.742696+0000","last_active":"2024-03-22T13:12:13.742657+0000","last_peered":"2024-03-22T13:12:13.742657+0000","last_clean":"2024-03-22T13:12:13.742657+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T13:12:13.742657+0000","last_undegraded":"2024-03-22T13:12:13.742657+0000","last_fullsized":"2024-03-22T13:12:13.742657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:13.742546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:13.742546+0000","last_clean_scrub_stamp":"2024-03-22T13:12:13.742546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:46:55.402197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:08.047238+0000","last_change":"2024-03-22T12:57:04.180841+0000","last_active":"2024-03-22T13:10:08.047238+0000","last_peered":"2024-03-22T13:10:08.047238+0000","last_clean":"2024-03-22T13:10:08.047238+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T13:10:08.047238+0000","last_undegraded":"2024-03-22T13:10:08.047238+0000","last_fullsized":"2024-03-22T13:10:08.047238+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:34.551809+0000","last_change":"2024-03-22T12:57:05.045995+0000","last_active":"2024-03-22T13:10:34.551809+0000","last_peered":"2024-03-22T13:10:34.551809+0000","last_clean":"2024-03-22T13:10:34.551809+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T13:10:34.551809+0000","last_undegraded":"2024-03-22T13:10:34.551809+0000","last_fullsized":"2024-03-22T13:10:34.551809+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:59.458043+0000","last_change":"2024-03-22T12:56:30.312287+0000","last_active":"2024-03-22T13:09:59.458043+0000","last_peered":"2024-03-22T13:09:59.458043+0000","last_clean":"2024-03-22T13:09:59.458043+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T13:09:59.458043+0000","last_undegraded":"2024-03-22T13:09:59.458043+0000","last_fullsized":"2024-03-22T13:09:59.458043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:27:22.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:11.536116+0000","last_change":"2024-03-22T13:12:11.536116+0000","last_active":"2024-03-22T13:12:11.536116+0000","last_peered":"2024-03-22T13:12:11.536116+0000","last_clean":"2024-03-22T13:12:11.536116+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T13:12:11.536116+0000","last_undegraded":"2024-03-22T13:12:11.536116+0000","last_fullsized":"2024-03-22T13:12:11.536116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:11.536023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:11.536023+0000","last_clean_scrub_stamp":"2024-03-22T13:12:11.536023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:07:32.268477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:07:48.040052+0000","last_change":"2024-03-22T12:56:30.309098+0000","last_active":"2024-03-22T13:07:48.040052+0000","last_peered":"2024-03-22T13:07:48.040052+0000","last_clean":"2024-03-22T13:07:48.040052+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T13:07:48.040052+0000","last_undegraded":"2024-03-22T13:07:48.040052+0000","last_fullsized":"2024-03-22T13:07:48.040052+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:56.782974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:34.551858+0000","last_change":"2024-03-22T12:57:04.187737+0000","last_active":"2024-03-22T13:10:34.551858+0000","last_peered":"2024-03-22T13:10:34.551858+0000","last_clean":"2024-03-22T13:10:34.551858+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T13:10:34.551858+0000","last_undegraded":"2024-03-22T13:10:34.551858+0000","last_fullsized":"2024-03-22T13:10:34.551858+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:33.050557+0000","last_change":"2024-03-22T12:56:30.314310+0000","last_active":"2024-03-22T13:10:33.050557+0000","last_peered":"2024-03-22T13:10:33.050557+0000","last_clean":"2024-03-22T13:10:33.050557+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T13:10:33.050557+0000","last_undegraded":"2024-03-22T13:10:33.050557+0000","last_fullsized":"2024-03-22T13:10:33.050557+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447831+0000","last_change":"2024-03-22T12:56:30.310422+0000","last_active":"2024-03-22T13:08:09.447831+0000","last_peered":"2024-03-22T13:08:09.447831+0000","last_clean":"2024-03-22T13:08:09.447831+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T13:08:09.447831+0000","last_undegraded":"2024-03-22T13:08:09.447831+0000","last_fullsized":"2024-03-22T13:08:09.447831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:25:36.010577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:09.447764+0000","last_change":"2024-03-22T12:56:30.321362+0000","last_active":"2024-03-22T13:08:09.447764+0000","last_peered":"2024-03-22T13:08:09.447764+0000","last_clean":"2024-03-22T13:08:09.447764+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T13:08:09.447764+0000","last_undegraded":"2024-03-22T13:08:09.447764+0000","last_fullsized":"2024-03-22T13:08:09.447764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:08:33.459214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:29.551283+0000","last_change":"2024-03-22T12:56:30.308322+0000","last_active":"2024-03-22T13:10:29.551283+0000","last_peered":"2024-03-22T13:10:29.551283+0000","last_clean":"2024-03-22T13:10:29.551283+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T13:10:29.551283+0000","last_undegraded":"2024-03-22T13:10:29.551283+0000","last_fullsized":"2024-03-22T13:10:29.551283+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:29.444588+0000","last_change":"2024-03-22T12:56:30.301998+0000","last_active":"2024-03-22T13:10:29.444588+0000","last_peered":"2024-03-22T13:10:29.444588+0000","last_clean":"2024-03-22T13:10:29.444588+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T13:10:29.444588+0000","last_undegraded":"2024-03-22T13:10:29.444588+0000","last_fullsized":"2024-03-22T13:10:29.444588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:59.377682+0000","last_change":"2024-03-22T12:56:30.266832+0000","last_active":"2024-03-22T13:10:59.377682+0000","last_peered":"2024-03-22T13:10:59.377682+0000","last_clean":"2024-03-22T13:10:59.377682+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T13:10:59.377682+0000","last_undegraded":"2024-03-22T13:10:59.377682+0000","last_fullsized":"2024-03-22T13:10:59.377682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:10:59.485561+0000","last_change":"2024-03-22T12:56:30.304930+0000","last_active":"2024-03-22T13:10:59.485561+0000","last_peered":"2024-03-22T13:10:59.485561+0000","last_clean":"2024-03-22T13:10:59.485561+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T13:10:59.485561+0000","last_undegraded":"2024-03-22T13:10:59.485561+0000","last_fullsized":"2024-03-22T13:10:59.485561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044564+0000","last_change":"2024-03-22T12:57:04.251063+0000","last_active":"2024-03-22T13:08:38.044564+0000","last_peered":"2024-03-22T13:08:38.044564+0000","last_clean":"2024-03-22T13:08:38.044564+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T13:08:38.044564+0000","last_undegraded":"2024-03-22T13:08:38.044564+0000","last_fullsized":"2024-03-22T13:08:38.044564+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:18:00.097151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.044582+0000","last_change":"2024-03-22T12:56:30.306945+0000","last_active":"2024-03-22T13:08:38.044582+0000","last_peered":"2024-03-22T13:08:38.044582+0000","last_clean":"2024-03-22T13:08:38.044582+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T13:08:38.044582+0000","last_undegraded":"2024-03-22T13:08:38.044582+0000","last_fullsized":"2024-03-22T13:08:38.044582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:44:41.633477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:38.091763+0000","last_change":"2024-03-22T12:56:30.306546+0000","last_active":"2024-03-22T13:08:38.091763+0000","last_peered":"2024-03-22T13:08:38.091763+0000","last_clean":"2024-03-22T13:08:38.091763+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T13:08:38.091763+0000","last_undegraded":"2024-03-22T13:08:38.091763+0000","last_fullsized":"2024-03-22T13:08:38.091763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:11:31.890272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.731100+0000","last_change":"2024-03-22T13:12:12.731119+0000","last_active":"2024-03-22T13:12:12.731100+0000","last_peered":"2024-03-22T13:12:12.731100+0000","last_clean":"2024-03-22T13:12:12.731100+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T13:12:12.731100+0000","last_undegraded":"2024-03-22T13:12:12.731100+0000","last_fullsized":"2024-03-22T13:12:12.731100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:12.731024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:12.731024+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.731024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:07:33.723276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094308+0000","last_change":"2024-03-22T12:56:30.300811+0000","last_active":"2024-03-22T13:09:03.094308+0000","last_peered":"2024-03-22T13:09:03.094308+0000","last_clean":"2024-03-22T13:09:03.094308+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T13:09:03.094308+0000","last_undegraded":"2024-03-22T13:09:03.094308+0000","last_fullsized":"2024-03-22T13:09:03.094308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:00:01.284887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:59.452409+0000","last_change":"2024-03-22T12:56:30.321374+0000","last_active":"2024-03-22T13:08:59.452409+0000","last_peered":"2024-03-22T13:08:59.452409+0000","last_clean":"2024-03-22T13:08:59.452409+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T13:08:59.452409+0000","last_undegraded":"2024-03-22T13:08:59.452409+0000","last_fullsized":"2024-03-22T13:08:59.452409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:01:31.297801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094342+0000","last_change":"2024-03-22T12:56:30.318449+0000","last_active":"2024-03-22T13:09:03.094342+0000","last_peered":"2024-03-22T13:09:03.094342+0000","last_clean":"2024-03-22T13:09:03.094342+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T13:09:03.094342+0000","last_undegraded":"2024-03-22T13:09:03.094342+0000","last_fullsized":"2024-03-22T13:09:03.094342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:09:56.844501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:08:29.449798+0000","last_change":"2024-03-22T12:56:30.315065+0000","last_active":"2024-03-22T13:08:29.449798+0000","last_peered":"2024-03-22T13:08:29.449798+0000","last_clean":"2024-03-22T13:08:29.449798+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T13:08:29.449798+0000","last_undegraded":"2024-03-22T13:08:29.449798+0000","last_fullsized":"2024-03-22T13:08:29.449798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:50:22.059831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:11:04.448826+0000","last_change":"2024-03-22T12:56:30.317188+0000","last_active":"2024-03-22T13:11:04.448826+0000","last_peered":"2024-03-22T13:11:04.448826+0000","last_clean":"2024-03-22T13:11:04.448826+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T13:11:04.448826+0000","last_undegraded":"2024-03-22T13:11:04.448826+0000","last_fullsized":"2024-03-22T13:11:04.448826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:10.884935+0000","last_change":"2024-03-22T13:12:10.884972+0000","last_active":"2024-03-22T13:12:10.884935+0000","last_peered":"2024-03-22T13:12:10.884935+0000","last_clean":"2024-03-22T13:12:10.884935+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T13:12:10.884935+0000","last_undegraded":"2024-03-22T13:12:10.884935+0000","last_fullsized":"2024-03-22T13:12:10.884935+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:10.884815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:10.884815+0000","last_clean_scrub_stamp":"2024-03-22T13:12:10.884815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:52:57.789842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:09:03.094374+0000","last_change":"2024-03-22T12:56:30.232613+0000","last_active":"2024-03-22T13:09:03.094374+0000","last_peered":"2024-03-22T13:09:03.094374+0000","last_clean":"2024-03-22T13:09:03.094374+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T13:09:03.094374+0000","last_undegraded":"2024-03-22T13:09:03.094374+0000","last_fullsized":"2024-03-22T13:09:03.094374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T12:56:28.493851+0000","last_clean_scrub_stamp":"2024-03-22T12:56:28.493851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:22:30.052572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509636,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":380928,"data_stored":142876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509636,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":356352,"data_stored":140274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509636,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":143610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509636,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":360448,"data_stored":140476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509635,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":425984,"data_stored":730186,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542341,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":434176,"data_stored":731412,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542341,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":143576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542340,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T13:12:17.916 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.916 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.916 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.916 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.916 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.916 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.917 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.918 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.918 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.918 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.918 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.918 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.918 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.918 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.919 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.920 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-22T12:56:14.706607+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=14, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.921 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.922 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.923 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.924 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-22T12:56:27.224260+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=27, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-03-22T12:56:35.512368+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=35, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.925 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.926 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.927 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.927 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.927 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.927 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.927 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.927 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.927 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-22T12:56:28.493851+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=12, tm_min=56, tm_sec=28, tm_wday=4, tm_yday=82, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=22, tm_hour=13, tm_min=12, tm_sec=7, tm_wday=4, tm_yday=82, tm_isdst=0) 2024-03-22T13:12:17.928 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-22T13:12:33.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:12:33.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:12:37.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T13:12:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.123+0000 7fef0da00640 1 Processor -- start 2024-03-22T13:12:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.123+0000 7fef0da00640 1 -- start start 2024-03-22T13:12:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.125+0000 7fef0da00640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef080cc3b0 0x7fef080cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.125+0000 7fef0da00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cf770 0x7fef080cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.125+0000 7fef0da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080ce230 con 0x7fef080cccd0 2024-03-22T13:12:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.125+0000 7fef0da00640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef080ce510 con 0x7fef080cc3b0 2024-03-22T13:12:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.125+0000 7fef0da00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef080ce7f0 con 0x7fef080cf770 2024-03-22T13:12:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.126+0000 7feefe000640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cf770 0x7fef080cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.126+0000 7fef07400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fef080cccd0 0x7fef0815ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60030/0 (socket says 172.21.15.6:60030) 2024-03-22T13:12:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.126+0000 7fef07400640 1 -- 172.21.15.6:0/825519764 learned_addr learned my addr 172.21.15.6:0/825519764 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.126+0000 7fef06a00640 1 --2- 172.21.15.6:0/825519764 >> v2:172.21.15.92:3300/0 conn(0x7fef080cc3b0 0x7fef080cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.129+0000 7fef06000640 1 -- 172.21.15.6:0/825519764 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758532906 0 0) 0x7fef080ce230 con 0x7fef080cccd0 2024-03-22T13:12:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.129+0000 7fef06000640 1 -- 172.21.15.6:0/825519764 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec003520 con 0x7fef080cccd0 2024-03-22T13:12:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.129+0000 7feefe000640 1 -- 172.21.15.6:0/825519764 >> v2:172.21.15.92:3300/0 conn(0x7fef080cc3b0 msgr2=0x7fef080cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.129+0000 7feefe000640 1 --2- 172.21.15.6:0/825519764 >> v2:172.21.15.92:3300/0 conn(0x7fef080cc3b0 0x7fef080cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.129+0000 7feefe000640 1 -- 172.21.15.6:0/825519764 >> v1:172.21.15.6:6789/0 conn(0x7fef080cccd0 legacy=0x7fef0815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.129+0000 7feefe000640 1 -- 172.21.15.6:0/825519764 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080cead0 con 0x7fef080cf770 2024-03-22T13:12:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.132+0000 7feefe000640 1 --2- 172.21.15.6:0/825519764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cf770 0x7fef080cc790 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7feef00097a0 tx=0x7feef002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a8fa82eb0980e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:38.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.137+0000 7fef06000640 1 -- 172.21.15.6:0/825519764 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00315a0 con 0x7fef080cf770 2024-03-22T13:12:38.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.137+0000 7fef06000640 1 -- 172.21.15.6:0/825519764 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef0037070 con 0x7fef080cf770 2024-03-22T13:12:38.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.137+0000 7fef06000640 1 -- 172.21.15.6:0/825519764 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0031960 con 0x7fef080cf770 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 -- 172.21.15.6:0/825519764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cf770 msgr2=0x7fef080cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 --2- 172.21.15.6:0/825519764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cf770 0x7fef080cc790 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7feef00097a0 tx=0x7feef002b840 comp rx=0 tx=0).stop 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 -- 172.21.15.6:0/825519764 shutdown_connections 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 --2- 172.21.15.6:0/825519764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cf770 0x7fef080cc790 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 --2- 172.21.15.6:0/825519764 >> v2:172.21.15.92:3300/0 conn(0x7fef080cc3b0 0x7fef080cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 -- 172.21.15.6:0/825519764 >> 172.21.15.6:0/825519764 conn(0x7fef080b16d0 msgr2=0x7fef080b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 -- 172.21.15.6:0/825519764 shutdown_connections 2024-03-22T13:12:38.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.138+0000 7fef0da00640 1 -- 172.21.15.6:0/825519764 wait complete. 2024-03-22T13:12:38.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.140+0000 7fef0da00640 1 Processor -- start 2024-03-22T13:12:38.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.140+0000 7fef0da00640 1 -- start start 2024-03-22T13:12:38.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.141+0000 7fef0da00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cc3b0 0x7fef08156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.141+0000 7fef0da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef080cccd0 0x7fef08155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.141+0000 7fef0da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef080cf770 0x7fef081533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.141+0000 7fef0da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0815a5d0 con 0x7fef080cccd0 2024-03-22T13:12:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.141+0000 7fef0da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0806dc10 con 0x7fef080cf770 2024-03-22T13:12:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.141+0000 7fef0da00640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef08154bb0 con 0x7fef080cc3b0 2024-03-22T13:12:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.141+0000 7fef06a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cc3b0 0x7fef08156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.142+0000 7fef07400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef080cf770 0x7fef081533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.142+0000 7feefe000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef080cccd0 0x7fef08155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.142+0000 7fef06a00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cc3b0 0x7fef08156f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40776/0 (socket says 172.21.15.6:40776) 2024-03-22T13:12:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.142+0000 7fef06a00640 1 -- 172.21.15.6:0/4222705633 learned_addr learned my addr 172.21.15.6:0/4222705633 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.146+0000 7fef07400640 1 -- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cc3b0 msgr2=0x7fef08156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.146+0000 7fef07400640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cc3b0 0x7fef08156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.146+0000 7fef07400640 1 -- 172.21.15.6:0/4222705633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef080cccd0 msgr2=0x7fef08155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.146+0000 7fef07400640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef080cccd0 0x7fef08155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.146+0000 7fef07400640 1 -- 172.21.15.6:0/4222705633 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080b3b30 con 0x7fef080cf770 2024-03-22T13:12:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.149+0000 7fef07400640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef080cf770 0x7fef081533d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fef0000cae0 tx=0x7fef0000a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.151+0000 7feeff400640 1 -- 172.21.15.6:0/4222705633 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00017070 con 0x7fef080cf770 2024-03-22T13:12:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.151+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef081579e0 con 0x7fef080cf770 2024-03-22T13:12:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.152+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef080b50c0 con 0x7fef080cf770 2024-03-22T13:12:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.153+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef080756c0 con 0x7fef080cf770 2024-03-22T13:12:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.153+0000 7feeff400640 1 -- 172.21.15.6:0/4222705633 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0000ac40 con 0x7fef080cf770 2024-03-22T13:12:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.153+0000 7feeff400640 1 -- 172.21.15.6:0/4222705633 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00021060 con 0x7fef080cf770 2024-03-22T13:12:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.445+0000 7feeff400640 1 -- 172.21.15.6:0/4222705633 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef0001a070 con 0x7fef080cf770 2024-03-22T13:12:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.447+0000 7feeff400640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feee406bbd0 0x7feee406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.447+0000 7fef06a00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feee406bbd0 0x7feee406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.448+0000 7fef06a00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feee406bbd0 0x7feee406e090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7feef8009c00 tx=0x7feef8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:38.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.451+0000 7feeff400640 1 -- 172.21.15.6:0/4222705633 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7fef000571a0 con 0x7fef080cf770 2024-03-22T13:12:38.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.472+0000 7feeff400640 1 -- 172.21.15.6:0/4222705633 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef00059100 con 0x7fef080cf770 2024-03-22T13:12:38.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.737+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 --> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef0806de80 con 0x7feee406bbd0 2024-03-22T13:12:38.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.754+0000 7feeff400640 1 -- 172.21.15.6:0/4222705633 <== mgr.4106 v2:172.21.15.92:6800/2032190056 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332890 (secure 0 0 0) 0x7fef0806de80 con 0x7feee406bbd0 2024-03-22T13:12:38.758 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-22T13:12:38.760 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feee406bbd0 msgr2=0x7feee406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feee406bbd0 0x7feee406e090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7feef8009c00 tx=0x7feef8009190 comp rx=0 tx=0).stop 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef080cf770 msgr2=0x7fef081533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef080cf770 0x7fef081533d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fef0000cae0 tx=0x7fef0000a060 comp rx=0 tx=0).stop 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 shutdown_connections 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7feee406bbd0 0x7feee406e090 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef080cf770 0x7fef081533d0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef080cccd0 0x7fef08155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 --2- 172.21.15.6:0/4222705633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef080cc3b0 0x7fef08156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:38.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 >> 172.21.15.6:0/4222705633 conn(0x7fef080b16d0 msgr2=0x7fef0806d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:38.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 shutdown_connections 2024-03-22T13:12:38.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:38.759+0000 7fef0da00640 1 -- 172.21.15.6:0/4222705633 wait complete. 2024-03-22T13:12:38.776 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-03-22T13:12:37.920790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219104,"kb_used_data":2976,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755616,"statfs":{"total":773094113280,"available":772869750784,"internally_reserved":0,"allocated":3047424,"data_stored":2312816,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"2.1f","version":"31'19","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:10.518751+0000","last_change":"2024-03-22T13:12:10.518800+0000","last_active":"2024-03-22T13:12:10.518751+0000","last_peered":"2024-03-22T13:12:10.518751+0000","last_clean":"2024-03-22T13:12:10.518751+0000","last_became_active":"2024-03-22T12:56:28.777943+0000","last_became_peered":"2024-03-22T12:56:28.777943+0000","last_unstale":"2024-03-22T13:12:10.518751+0000","last_undegraded":"2024-03-22T13:12:10.518751+0000","last_fullsized":"2024-03-22T13:12:10.518751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-03-22T13:12:10.518656+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-03-22T13:12:10.518656+0000","last_clean_scrub_stamp":"2024-03-22T13:12:10.518656+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:21:36.449884+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.825499+0000","last_change":"2024-03-22T13:12:20.825499+0000","last_active":"2024-03-22T13:12:20.825499+0000","last_peered":"2024-03-22T13:12:20.825499+0000","last_clean":"2024-03-22T13:12:20.825499+0000","last_became_active":"2024-03-22T12:56:30.268482+0000","last_became_peered":"2024-03-22T12:56:30.268482+0000","last_unstale":"2024-03-22T13:12:20.825499+0000","last_undegraded":"2024-03-22T13:12:20.825499+0000","last_fullsized":"2024-03-22T13:12:20.825499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:20.825272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:20.825272+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.825272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:14:37.711363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:30.760064+0000","last_change":"2024-03-22T13:12:30.760105+0000","last_active":"2024-03-22T13:12:30.760064+0000","last_peered":"2024-03-22T13:12:30.760064+0000","last_clean":"2024-03-22T13:12:30.760064+0000","last_became_active":"2024-03-22T12:56:37.235711+0000","last_became_peered":"2024-03-22T12:56:37.235711+0000","last_unstale":"2024-03-22T13:12:30.760064+0000","last_undegraded":"2024-03-22T13:12:30.760064+0000","last_fullsized":"2024-03-22T13:12:30.760064+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:30.759957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:30.759957+0000","last_clean_scrub_stamp":"2024-03-22T13:12:30.759957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:13:35.858712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"30'2","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.747808+0000","last_change":"2024-03-22T13:12:18.747808+0000","last_active":"2024-03-22T13:12:18.747808+0000","last_peered":"2024-03-22T13:12:18.747808+0000","last_clean":"2024-03-22T13:12:18.747808+0000","last_became_active":"2024-03-22T12:56:28.782121+0000","last_became_peered":"2024-03-22T12:56:28.782121+0000","last_unstale":"2024-03-22T13:12:18.747808+0000","last_undegraded":"2024-03-22T13:12:18.747808+0000","last_fullsized":"2024-03-22T13:12:18.747808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-03-22T13:12:18.747663+0000","last_deep_scrub":"30'2","last_deep_scrub_stamp":"2024-03-22T13:12:18.747663+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.747663+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:36:16.011391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.383121+0000","last_change":"2024-03-22T13:12:18.383156+0000","last_active":"2024-03-22T13:12:18.383121+0000","last_peered":"2024-03-22T13:12:18.383121+0000","last_clean":"2024-03-22T13:12:18.383121+0000","last_became_active":"2024-03-22T12:56:30.313105+0000","last_became_peered":"2024-03-22T12:56:30.313105+0000","last_unstale":"2024-03-22T13:12:18.383121+0000","last_undegraded":"2024-03-22T13:12:18.383121+0000","last_fullsized":"2024-03-22T13:12:18.383121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:18.383021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:18.383021+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.383021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:18:15.883488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:27.265898+0000","last_change":"2024-03-22T13:12:27.265934+0000","last_active":"2024-03-22T13:12:27.265898+0000","last_peered":"2024-03-22T13:12:27.265898+0000","last_clean":"2024-03-22T13:12:27.265898+0000","last_became_active":"2024-03-22T12:56:37.050194+0000","last_became_peered":"2024-03-22T12:56:37.050194+0000","last_unstale":"2024-03-22T13:12:27.265898+0000","last_undegraded":"2024-03-22T13:12:27.265898+0000","last_fullsized":"2024-03-22T13:12:27.265898+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:27.265775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:27.265775+0000","last_clean_scrub_stamp":"2024-03-22T13:12:27.265775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:56:10.873138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"31'19","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.014569+0000","last_change":"2024-03-22T13:12:19.014604+0000","last_active":"2024-03-22T13:12:19.014569+0000","last_peered":"2024-03-22T13:12:19.014569+0000","last_clean":"2024-03-22T13:12:19.014569+0000","last_became_active":"2024-03-22T12:56:28.752826+0000","last_became_peered":"2024-03-22T12:56:28.752826+0000","last_unstale":"2024-03-22T13:12:19.014569+0000","last_undegraded":"2024-03-22T13:12:19.014569+0000","last_fullsized":"2024-03-22T13:12:19.014569+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-03-22T13:12:19.014508+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-03-22T13:12:19.014508+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.014508+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:10:56.654876+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:23.506281+0000","last_change":"2024-03-22T13:12:23.506317+0000","last_active":"2024-03-22T13:12:23.506281+0000","last_peered":"2024-03-22T13:12:23.506281+0000","last_clean":"2024-03-22T13:12:23.506281+0000","last_became_active":"2024-03-22T12:56:30.309629+0000","last_became_peered":"2024-03-22T12:56:30.309629+0000","last_unstale":"2024-03-22T13:12:23.506281+0000","last_undegraded":"2024-03-22T13:12:23.506281+0000","last_fullsized":"2024-03-22T13:12:23.506281+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:23.506169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:23.506169+0000","last_clean_scrub_stamp":"2024-03-22T13:12:23.506169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:29:32.776054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:24.281770+0000","last_change":"2024-03-22T13:12:24.281790+0000","last_active":"2024-03-22T13:12:24.281770+0000","last_peered":"2024-03-22T13:12:24.281770+0000","last_clean":"2024-03-22T13:12:24.281770+0000","last_became_active":"2024-03-22T12:56:37.045752+0000","last_became_peered":"2024-03-22T12:56:37.045752+0000","last_unstale":"2024-03-22T13:12:24.281770+0000","last_undegraded":"2024-03-22T13:12:24.281770+0000","last_fullsized":"2024-03-22T13:12:24.281770+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:24.281700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:24.281700+0000","last_clean_scrub_stamp":"2024-03-22T13:12:24.281700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:12:01.025399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"31'8","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.060697+0000","last_change":"2024-03-22T13:12:21.060744+0000","last_active":"2024-03-22T13:12:21.060697+0000","last_peered":"2024-03-22T13:12:21.060697+0000","last_clean":"2024-03-22T13:12:21.060697+0000","last_became_active":"2024-03-22T12:56:28.761148+0000","last_became_peered":"2024-03-22T12:56:28.761148+0000","last_unstale":"2024-03-22T13:12:21.060697+0000","last_undegraded":"2024-03-22T13:12:21.060697+0000","last_fullsized":"2024-03-22T13:12:21.060697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-03-22T13:12:21.060593+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-03-22T13:12:21.060593+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.060593+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:32:52.741346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.486316+0000","last_change":"2024-03-22T13:12:22.486355+0000","last_active":"2024-03-22T13:12:22.486316+0000","last_peered":"2024-03-22T13:12:22.486316+0000","last_clean":"2024-03-22T13:12:22.486316+0000","last_became_active":"2024-03-22T12:56:30.308106+0000","last_became_peered":"2024-03-22T12:56:30.308106+0000","last_unstale":"2024-03-22T13:12:22.486316+0000","last_undegraded":"2024-03-22T13:12:22.486316+0000","last_fullsized":"2024-03-22T13:12:22.486316+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:22.486203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:22.486203+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.486203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:36:19.749931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.311478+0000","last_change":"2024-03-22T13:12:22.311528+0000","last_active":"2024-03-22T13:12:22.311478+0000","last_peered":"2024-03-22T13:12:22.311478+0000","last_clean":"2024-03-22T13:12:22.311478+0000","last_became_active":"2024-03-22T12:56:37.043336+0000","last_became_peered":"2024-03-22T12:56:37.043336+0000","last_unstale":"2024-03-22T13:12:22.311478+0000","last_undegraded":"2024-03-22T13:12:22.311478+0000","last_fullsized":"2024-03-22T13:12:22.311478+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:22.311355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:22.311355+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.311355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:17:41.856472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:23.677254+0000","last_change":"2024-03-22T13:12:23.677287+0000","last_active":"2024-03-22T13:12:23.677254+0000","last_peered":"2024-03-22T13:12:23.677254+0000","last_clean":"2024-03-22T13:12:23.677254+0000","last_became_active":"2024-03-22T12:56:30.305393+0000","last_became_peered":"2024-03-22T12:56:30.305393+0000","last_unstale":"2024-03-22T13:12:23.677254+0000","last_undegraded":"2024-03-22T13:12:23.677254+0000","last_fullsized":"2024-03-22T13:12:23.677254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:23.677153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:23.677153+0000","last_clean_scrub_stamp":"2024-03-22T13:12:23.677153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:38:06.576115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"31'6","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.021438+0000","last_change":"2024-03-22T13:12:20.021496+0000","last_active":"2024-03-22T13:12:20.021438+0000","last_peered":"2024-03-22T13:12:20.021438+0000","last_clean":"2024-03-22T13:12:20.021438+0000","last_became_active":"2024-03-22T12:56:28.754251+0000","last_became_peered":"2024-03-22T12:56:28.754251+0000","last_unstale":"2024-03-22T13:12:20.021438+0000","last_undegraded":"2024-03-22T13:12:20.021438+0000","last_fullsized":"2024-03-22T13:12:20.021438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-03-22T13:12:20.021325+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-03-22T13:12:20.021325+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.021325+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:34:13.579010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:26.269501+0000","last_change":"2024-03-22T13:12:26.269541+0000","last_active":"2024-03-22T13:12:26.269501+0000","last_peered":"2024-03-22T13:12:26.269501+0000","last_clean":"2024-03-22T13:12:26.269501+0000","last_became_active":"2024-03-22T12:56:37.049377+0000","last_became_peered":"2024-03-22T12:56:37.049377+0000","last_unstale":"2024-03-22T13:12:26.269501+0000","last_undegraded":"2024-03-22T13:12:26.269501+0000","last_fullsized":"2024-03-22T13:12:26.269501+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:26.269386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:26.269386+0000","last_clean_scrub_stamp":"2024-03-22T13:12:26.269386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:14:43.155478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:15.676720+0000","last_change":"2024-03-22T13:12:15.676781+0000","last_active":"2024-03-22T13:12:15.676720+0000","last_peered":"2024-03-22T13:12:15.676720+0000","last_clean":"2024-03-22T13:12:15.676720+0000","last_became_active":"2024-03-22T12:56:30.305711+0000","last_became_peered":"2024-03-22T12:56:30.305711+0000","last_unstale":"2024-03-22T13:12:15.676720+0000","last_undegraded":"2024-03-22T13:12:15.676720+0000","last_fullsized":"2024-03-22T13:12:15.676720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:15.676501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:15.676501+0000","last_clean_scrub_stamp":"2024-03-22T13:12:15.676501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:33:20.170897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:15.887705+0000","last_change":"2024-03-22T13:12:15.887755+0000","last_active":"2024-03-22T13:12:15.887705+0000","last_peered":"2024-03-22T13:12:15.887705+0000","last_clean":"2024-03-22T13:12:15.887705+0000","last_became_active":"2024-03-22T12:56:28.772372+0000","last_became_peered":"2024-03-22T12:56:28.772372+0000","last_unstale":"2024-03-22T13:12:15.887705+0000","last_undegraded":"2024-03-22T13:12:15.887705+0000","last_fullsized":"2024-03-22T13:12:15.887705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-03-22T13:12:15.887571+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-03-22T13:12:15.887571+0000","last_clean_scrub_stamp":"2024-03-22T13:12:15.887571+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:33:20.381966+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:23.319642+0000","last_change":"2024-03-22T13:12:23.319642+0000","last_active":"2024-03-22T13:12:23.319642+0000","last_peered":"2024-03-22T13:12:23.319642+0000","last_clean":"2024-03-22T13:12:23.319642+0000","last_became_active":"2024-03-22T12:56:37.045673+0000","last_became_peered":"2024-03-22T12:56:37.045673+0000","last_unstale":"2024-03-22T13:12:23.319642+0000","last_undegraded":"2024-03-22T13:12:23.319642+0000","last_fullsized":"2024-03-22T13:12:23.319642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:23.319496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:23.319496+0000","last_clean_scrub_stamp":"2024-03-22T13:12:23.319496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:04:01.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.777432+0000","last_change":"2024-03-22T13:12:21.777485+0000","last_active":"2024-03-22T13:12:21.777432+0000","last_peered":"2024-03-22T13:12:21.777432+0000","last_clean":"2024-03-22T13:12:21.777432+0000","last_became_active":"2024-03-22T12:56:30.268354+0000","last_became_peered":"2024-03-22T12:56:30.268354+0000","last_unstale":"2024-03-22T13:12:21.777432+0000","last_undegraded":"2024-03-22T13:12:21.777432+0000","last_fullsized":"2024-03-22T13:12:21.777432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:21.777257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:21.777257+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.777257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:02:50.950789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"31'11","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:11.895621+0000","last_change":"2024-03-22T13:12:11.895680+0000","last_active":"2024-03-22T13:12:11.895621+0000","last_peered":"2024-03-22T13:12:11.895621+0000","last_clean":"2024-03-22T13:12:11.895621+0000","last_became_active":"2024-03-22T12:56:28.756819+0000","last_became_peered":"2024-03-22T12:56:28.756819+0000","last_unstale":"2024-03-22T13:12:11.895621+0000","last_undegraded":"2024-03-22T13:12:11.895621+0000","last_fullsized":"2024-03-22T13:12:11.895621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-03-22T13:12:11.895497+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-03-22T13:12:11.895497+0000","last_clean_scrub_stamp":"2024-03-22T13:12:11.895497+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:07:32.627951+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:24.905965+0000","last_change":"2024-03-22T13:12:24.906030+0000","last_active":"2024-03-22T13:12:24.905965+0000","last_peered":"2024-03-22T13:12:24.905965+0000","last_clean":"2024-03-22T13:12:24.905965+0000","last_became_active":"2024-03-22T12:56:37.238310+0000","last_became_peered":"2024-03-22T12:56:37.238310+0000","last_unstale":"2024-03-22T13:12:24.905965+0000","last_undegraded":"2024-03-22T13:12:24.905965+0000","last_fullsized":"2024-03-22T13:12:24.905965+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:24.905745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:24.905745+0000","last_clean_scrub_stamp":"2024-03-22T13:12:24.905745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:34:31.098880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.358841+0000","last_change":"2024-03-22T13:12:18.358880+0000","last_active":"2024-03-22T13:12:18.358841+0000","last_peered":"2024-03-22T13:12:18.358841+0000","last_clean":"2024-03-22T13:12:18.358841+0000","last_became_active":"2024-03-22T12:56:30.318720+0000","last_became_peered":"2024-03-22T12:56:30.318720+0000","last_unstale":"2024-03-22T13:12:18.358841+0000","last_undegraded":"2024-03-22T13:12:18.358841+0000","last_fullsized":"2024-03-22T13:12:18.358841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:18.358743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:18.358743+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.358743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:49:27.928242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"31'3","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.100217+0000","last_change":"2024-03-22T13:12:22.100260+0000","last_active":"2024-03-22T13:12:22.100217+0000","last_peered":"2024-03-22T13:12:22.100217+0000","last_clean":"2024-03-22T13:12:22.100217+0000","last_became_active":"2024-03-22T12:56:28.767074+0000","last_became_peered":"2024-03-22T12:56:28.767074+0000","last_unstale":"2024-03-22T13:12:22.100217+0000","last_undegraded":"2024-03-22T13:12:22.100217+0000","last_fullsized":"2024-03-22T13:12:22.100217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-03-22T13:12:22.100096+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-03-22T13:12:22.100096+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.100096+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:07:59.744401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:25.262218+0000","last_change":"2024-03-22T13:12:25.262253+0000","last_active":"2024-03-22T13:12:25.262218+0000","last_peered":"2024-03-22T13:12:25.262218+0000","last_clean":"2024-03-22T13:12:25.262218+0000","last_became_active":"2024-03-22T12:56:37.045587+0000","last_became_peered":"2024-03-22T12:56:37.045587+0000","last_unstale":"2024-03-22T13:12:25.262218+0000","last_undegraded":"2024-03-22T13:12:25.262218+0000","last_fullsized":"2024-03-22T13:12:25.262218+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:25.262130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:25.262130+0000","last_clean_scrub_stamp":"2024-03-22T13:12:25.262130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:59:29.927067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:16.436866+0000","last_change":"2024-03-22T13:12:16.436913+0000","last_active":"2024-03-22T13:12:16.436866+0000","last_peered":"2024-03-22T13:12:16.436866+0000","last_clean":"2024-03-22T13:12:16.436866+0000","last_became_active":"2024-03-22T12:56:30.310933+0000","last_became_peered":"2024-03-22T12:56:30.310933+0000","last_unstale":"2024-03-22T13:12:16.436866+0000","last_undegraded":"2024-03-22T13:12:16.436866+0000","last_fullsized":"2024-03-22T13:12:16.436866+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:16.436766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:16.436766+0000","last_clean_scrub_stamp":"2024-03-22T13:12:16.436766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:12:06.817476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"31'11","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:16.638580+0000","last_change":"2024-03-22T13:12:16.638629+0000","last_active":"2024-03-22T13:12:16.638580+0000","last_peered":"2024-03-22T13:12:16.638580+0000","last_clean":"2024-03-22T13:12:16.638580+0000","last_became_active":"2024-03-22T12:56:28.754380+0000","last_became_peered":"2024-03-22T12:56:28.754380+0000","last_unstale":"2024-03-22T13:12:16.638580+0000","last_undegraded":"2024-03-22T13:12:16.638580+0000","last_fullsized":"2024-03-22T13:12:16.638580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-03-22T13:12:16.638470+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-03-22T13:12:16.638470+0000","last_clean_scrub_stamp":"2024-03-22T13:12:16.638470+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:38:05.945127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:28.311275+0000","last_change":"2024-03-22T13:12:28.311331+0000","last_active":"2024-03-22T13:12:28.311275+0000","last_peered":"2024-03-22T13:12:28.311275+0000","last_clean":"2024-03-22T13:12:28.311275+0000","last_became_active":"2024-03-22T12:56:37.237604+0000","last_became_peered":"2024-03-22T12:56:37.237604+0000","last_unstale":"2024-03-22T13:12:28.311275+0000","last_undegraded":"2024-03-22T13:12:28.311275+0000","last_fullsized":"2024-03-22T13:12:28.311275+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:28.311153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:28.311153+0000","last_clean_scrub_stamp":"2024-03-22T13:12:28.311153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:48:05.455661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.822690+0000","last_change":"2024-03-22T13:12:22.822727+0000","last_active":"2024-03-22T13:12:22.822690+0000","last_peered":"2024-03-22T13:12:22.822690+0000","last_clean":"2024-03-22T13:12:22.822690+0000","last_became_active":"2024-03-22T12:56:30.269682+0000","last_became_peered":"2024-03-22T12:56:30.269682+0000","last_unstale":"2024-03-22T13:12:22.822690+0000","last_undegraded":"2024-03-22T13:12:22.822690+0000","last_fullsized":"2024-03-22T13:12:22.822690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:22.822538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:22.822538+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.822538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:42.352012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"31'11","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.919291+0000","last_change":"2024-03-22T13:12:12.919321+0000","last_active":"2024-03-22T13:12:12.919291+0000","last_peered":"2024-03-22T13:12:12.919291+0000","last_clean":"2024-03-22T13:12:12.919291+0000","last_became_active":"2024-03-22T12:56:28.767092+0000","last_became_peered":"2024-03-22T12:56:28.767092+0000","last_unstale":"2024-03-22T13:12:12.919291+0000","last_undegraded":"2024-03-22T13:12:12.919291+0000","last_fullsized":"2024-03-22T13:12:12.919291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-03-22T13:12:12.919229+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-03-22T13:12:12.919229+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.919229+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:09:23.443359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:27.836322+0000","last_change":"2024-03-22T13:12:27.836379+0000","last_active":"2024-03-22T13:12:27.836322+0000","last_peered":"2024-03-22T13:12:27.836322+0000","last_clean":"2024-03-22T13:12:27.836322+0000","last_became_active":"2024-03-22T12:56:37.232903+0000","last_became_peered":"2024-03-22T12:56:37.232903+0000","last_unstale":"2024-03-22T13:12:27.836322+0000","last_undegraded":"2024-03-22T13:12:27.836322+0000","last_fullsized":"2024-03-22T13:12:27.836322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:27.836147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:27.836147+0000","last_clean_scrub_stamp":"2024-03-22T13:12:27.836147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:58:18.132479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.874600+0000","last_change":"2024-03-22T13:12:21.874635+0000","last_active":"2024-03-22T13:12:21.874600+0000","last_peered":"2024-03-22T13:12:21.874600+0000","last_clean":"2024-03-22T13:12:21.874600+0000","last_became_active":"2024-03-22T12:56:30.317577+0000","last_became_peered":"2024-03-22T12:56:30.317577+0000","last_unstale":"2024-03-22T13:12:21.874600+0000","last_undegraded":"2024-03-22T13:12:21.874600+0000","last_fullsized":"2024-03-22T13:12:21.874600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:21.874497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:21.874497+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.874497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:47:59.019004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"31'7","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:24.137408+0000","last_change":"2024-03-22T13:12:24.137452+0000","last_active":"2024-03-22T13:12:24.137408+0000","last_peered":"2024-03-22T13:12:24.137408+0000","last_clean":"2024-03-22T13:12:24.137408+0000","last_became_active":"2024-03-22T12:56:28.776945+0000","last_became_peered":"2024-03-22T12:56:28.776945+0000","last_unstale":"2024-03-22T13:12:24.137408+0000","last_undegraded":"2024-03-22T13:12:24.137408+0000","last_fullsized":"2024-03-22T13:12:24.137408+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-03-22T13:12:24.137328+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-03-22T13:12:24.137328+0000","last_clean_scrub_stamp":"2024-03-22T13:12:24.137328+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:42:13.061056+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:23.426147+0000","last_change":"2024-03-22T13:12:23.426187+0000","last_active":"2024-03-22T13:12:23.426147+0000","last_peered":"2024-03-22T13:12:23.426147+0000","last_clean":"2024-03-22T13:12:23.426147+0000","last_became_active":"2024-03-22T12:56:37.237381+0000","last_became_peered":"2024-03-22T12:56:37.237381+0000","last_unstale":"2024-03-22T13:12:23.426147+0000","last_undegraded":"2024-03-22T13:12:23.426147+0000","last_fullsized":"2024-03-22T13:12:23.426147+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:23.426039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:23.426039+0000","last_clean_scrub_stamp":"2024-03-22T13:12:23.426039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:10:12.245690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:27.529155+0000","last_change":"2024-03-22T13:12:27.529188+0000","last_active":"2024-03-22T13:12:27.529155+0000","last_peered":"2024-03-22T13:12:27.529155+0000","last_clean":"2024-03-22T13:12:27.529155+0000","last_became_active":"2024-03-22T12:56:30.314943+0000","last_became_peered":"2024-03-22T12:56:30.314943+0000","last_unstale":"2024-03-22T13:12:27.529155+0000","last_undegraded":"2024-03-22T13:12:27.529155+0000","last_fullsized":"2024-03-22T13:12:27.529155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:27.529083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:27.529083+0000","last_clean_scrub_stamp":"2024-03-22T13:12:27.529083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:10:46.747478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"31'2","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:16.871962+0000","last_change":"2024-03-22T13:12:16.872003+0000","last_active":"2024-03-22T13:12:16.871962+0000","last_peered":"2024-03-22T13:12:16.871962+0000","last_clean":"2024-03-22T13:12:16.871962+0000","last_became_active":"2024-03-22T12:56:28.780135+0000","last_became_peered":"2024-03-22T12:56:28.780135+0000","last_unstale":"2024-03-22T13:12:16.871962+0000","last_undegraded":"2024-03-22T13:12:16.871962+0000","last_fullsized":"2024-03-22T13:12:16.871962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-03-22T13:12:16.871856+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-03-22T13:12:16.871856+0000","last_clean_scrub_stamp":"2024-03-22T13:12:16.871856+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:12:07.252566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.333813+0000","last_change":"2024-03-22T13:12:21.333854+0000","last_active":"2024-03-22T13:12:21.333813+0000","last_peered":"2024-03-22T13:12:21.333813+0000","last_clean":"2024-03-22T13:12:21.333813+0000","last_became_active":"2024-03-22T12:57:05.043480+0000","last_became_peered":"2024-03-22T12:57:05.043480+0000","last_unstale":"2024-03-22T13:12:21.333813+0000","last_undegraded":"2024-03-22T13:12:21.333813+0000","last_fullsized":"2024-03-22T13:12:21.333813+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:21.333710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:21.333710+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.333710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:30:17.070947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4,0],"acting":[2,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.348254+0000","last_change":"2024-03-22T13:12:19.348290+0000","last_active":"2024-03-22T13:12:19.348254+0000","last_peered":"2024-03-22T13:12:19.348254+0000","last_clean":"2024-03-22T13:12:19.348254+0000","last_became_active":"2024-03-22T12:56:30.319147+0000","last_became_peered":"2024-03-22T12:56:30.319147+0000","last_unstale":"2024-03-22T13:12:19.348254+0000","last_undegraded":"2024-03-22T13:12:19.348254+0000","last_fullsized":"2024-03-22T13:12:19.348254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:19.348138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:19.348138+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.348138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:07:40.340390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"31'7","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.464571+0000","last_change":"2024-03-22T13:12:20.464616+0000","last_active":"2024-03-22T13:12:20.464571+0000","last_peered":"2024-03-22T13:12:20.464571+0000","last_clean":"2024-03-22T13:12:20.464571+0000","last_became_active":"2024-03-22T12:56:28.773969+0000","last_became_peered":"2024-03-22T12:56:28.773969+0000","last_unstale":"2024-03-22T13:12:20.464571+0000","last_undegraded":"2024-03-22T13:12:20.464571+0000","last_fullsized":"2024-03-22T13:12:20.464571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-03-22T13:12:20.464462+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-03-22T13:12:20.464462+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.464462+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:03:58.820012+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:23.864108+0000","last_change":"2024-03-22T13:12:23.864148+0000","last_active":"2024-03-22T13:12:23.864108+0000","last_peered":"2024-03-22T13:12:23.864108+0000","last_clean":"2024-03-22T13:12:23.864108+0000","last_became_active":"2024-03-22T12:56:37.057215+0000","last_became_peered":"2024-03-22T12:56:37.057215+0000","last_unstale":"2024-03-22T13:12:23.864108+0000","last_undegraded":"2024-03-22T13:12:23.864108+0000","last_fullsized":"2024-03-22T13:12:23.864108+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:23.863997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:23.863997+0000","last_clean_scrub_stamp":"2024-03-22T13:12:23.863997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:39:09.324698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.898381+0000","last_change":"2024-03-22T13:12:20.898381+0000","last_active":"2024-03-22T13:12:20.898381+0000","last_peered":"2024-03-22T13:12:20.898381+0000","last_clean":"2024-03-22T13:12:20.898381+0000","last_became_active":"2024-03-22T12:56:30.314122+0000","last_became_peered":"2024-03-22T12:56:30.314122+0000","last_unstale":"2024-03-22T13:12:20.898381+0000","last_undegraded":"2024-03-22T13:12:20.898381+0000","last_fullsized":"2024-03-22T13:12:20.898381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:20.898193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:20.898193+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.898193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:56:04.505555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"31'8","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.270084+0000","last_change":"2024-03-22T13:12:12.270131+0000","last_active":"2024-03-22T13:12:12.270084+0000","last_peered":"2024-03-22T13:12:12.270084+0000","last_clean":"2024-03-22T13:12:12.270084+0000","last_became_active":"2024-03-22T12:56:28.762862+0000","last_became_peered":"2024-03-22T12:56:28.762862+0000","last_unstale":"2024-03-22T13:12:12.270084+0000","last_undegraded":"2024-03-22T13:12:12.270084+0000","last_fullsized":"2024-03-22T13:12:12.270084+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-03-22T13:12:12.269985+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-03-22T13:12:12.269985+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.269985+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:30:28.836151+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.383713+0000","last_change":"2024-03-22T13:12:20.383755+0000","last_active":"2024-03-22T13:12:20.383713+0000","last_peered":"2024-03-22T13:12:20.383713+0000","last_clean":"2024-03-22T13:12:20.383713+0000","last_became_active":"2024-03-22T12:57:04.948041+0000","last_became_peered":"2024-03-22T12:57:04.948041+0000","last_unstale":"2024-03-22T13:12:20.383713+0000","last_undegraded":"2024-03-22T13:12:20.383713+0000","last_fullsized":"2024-03-22T13:12:20.383713+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:20.383572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:20.383572+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.383572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:27:46.636564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0,3],"acting":[2,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.416675+0000","last_change":"2024-03-22T13:12:17.416710+0000","last_active":"2024-03-22T13:12:17.416675+0000","last_peered":"2024-03-22T13:12:17.416675+0000","last_clean":"2024-03-22T13:12:17.416675+0000","last_became_active":"2024-03-22T12:56:30.313632+0000","last_became_peered":"2024-03-22T12:56:30.313632+0000","last_unstale":"2024-03-22T13:12:17.416675+0000","last_undegraded":"2024-03-22T13:12:17.416675+0000","last_fullsized":"2024-03-22T13:12:17.416675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:17.416574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:17.416574+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.416574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:08:43.237158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"31'5","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:23.100789+0000","last_change":"2024-03-22T13:12:23.100789+0000","last_active":"2024-03-22T13:12:23.100789+0000","last_peered":"2024-03-22T13:12:23.100789+0000","last_clean":"2024-03-22T13:12:23.100789+0000","last_became_active":"2024-03-22T12:56:28.769136+0000","last_became_peered":"2024-03-22T12:56:28.769136+0000","last_unstale":"2024-03-22T13:12:23.100789+0000","last_undegraded":"2024-03-22T13:12:23.100789+0000","last_fullsized":"2024-03-22T13:12:23.100789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-03-22T13:12:23.100679+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-03-22T13:12:23.100679+0000","last_clean_scrub_stamp":"2024-03-22T13:12:23.100679+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:21:12.251501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.456435+0000","last_change":"2024-03-22T13:12:22.456474+0000","last_active":"2024-03-22T13:12:22.456435+0000","last_peered":"2024-03-22T13:12:22.456435+0000","last_clean":"2024-03-22T13:12:22.456435+0000","last_became_active":"2024-03-22T12:56:37.234793+0000","last_became_peered":"2024-03-22T12:56:37.234793+0000","last_unstale":"2024-03-22T13:12:22.456435+0000","last_undegraded":"2024-03-22T13:12:22.456435+0000","last_fullsized":"2024-03-22T13:12:22.456435+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:22.456333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:22.456333+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.456333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:47:59.600840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:14.720852+0000","last_change":"2024-03-22T13:12:14.720998+0000","last_active":"2024-03-22T13:12:14.720852+0000","last_peered":"2024-03-22T13:12:14.720852+0000","last_clean":"2024-03-22T13:12:14.720852+0000","last_became_active":"2024-03-22T12:56:30.308645+0000","last_became_peered":"2024-03-22T12:56:30.308645+0000","last_unstale":"2024-03-22T13:12:14.720852+0000","last_undegraded":"2024-03-22T13:12:14.720852+0000","last_fullsized":"2024-03-22T13:12:14.720852+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:14.720643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:14.720643+0000","last_clean_scrub_stamp":"2024-03-22T13:12:14.720643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:14:28.359283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"31'11","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.480556+0000","last_change":"2024-03-22T13:12:19.480605+0000","last_active":"2024-03-22T13:12:19.480556+0000","last_peered":"2024-03-22T13:12:19.480556+0000","last_clean":"2024-03-22T13:12:19.480556+0000","last_became_active":"2024-03-22T12:56:28.765540+0000","last_became_peered":"2024-03-22T12:56:28.765540+0000","last_unstale":"2024-03-22T13:12:19.480556+0000","last_undegraded":"2024-03-22T13:12:19.480556+0000","last_fullsized":"2024-03-22T13:12:19.480556+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-03-22T13:12:19.480445+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-03-22T13:12:19.480445+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.480445+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:17:39.025562+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"31'119","reported_seq":206,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.040230+0000","last_change":"2024-03-22T13:12:18.040230+0000","last_active":"2024-03-22T13:12:18.040230+0000","last_peered":"2024-03-22T13:12:18.040230+0000","last_clean":"2024-03-22T13:12:18.040230+0000","last_became_active":"2024-03-22T12:56:18.198637+0000","last_became_peered":"2024-03-22T12:56:18.198637+0000","last_unstale":"2024-03-22T13:12:18.040230+0000","last_undegraded":"2024-03-22T13:12:18.040230+0000","last_fullsized":"2024-03-22T13:12:18.040230+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'119","last_scrub_stamp":"2024-03-22T13:12:18.040138+0000","last_deep_scrub":"31'119","last_deep_scrub_stamp":"2024-03-22T13:12:18.040138+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.040138+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:30:13.777375+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:29.745932+0000","last_change":"2024-03-22T13:12:29.745969+0000","last_active":"2024-03-22T13:12:29.745932+0000","last_peered":"2024-03-22T13:12:29.745932+0000","last_clean":"2024-03-22T13:12:29.745932+0000","last_became_active":"2024-03-22T12:56:37.234992+0000","last_became_peered":"2024-03-22T12:56:37.234992+0000","last_unstale":"2024-03-22T13:12:29.745932+0000","last_undegraded":"2024-03-22T13:12:29.745932+0000","last_fullsized":"2024-03-22T13:12:29.745932+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:29.745853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:29.745853+0000","last_clean_scrub_stamp":"2024-03-22T13:12:29.745853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:06:20.613667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:28.024539+0000","last_change":"2024-03-22T13:12:28.024574+0000","last_active":"2024-03-22T13:12:28.024539+0000","last_peered":"2024-03-22T13:12:28.024539+0000","last_clean":"2024-03-22T13:12:28.024539+0000","last_became_active":"2024-03-22T12:56:30.306879+0000","last_became_peered":"2024-03-22T12:56:30.306879+0000","last_unstale":"2024-03-22T13:12:28.024539+0000","last_undegraded":"2024-03-22T13:12:28.024539+0000","last_fullsized":"2024-03-22T13:12:28.024539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:28.024430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:28.024430+0000","last_clean_scrub_stamp":"2024-03-22T13:12:28.024430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:07:43.133581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"31'9","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:13.298804+0000","last_change":"2024-03-22T13:12:13.298852+0000","last_active":"2024-03-22T13:12:13.298804+0000","last_peered":"2024-03-22T13:12:13.298804+0000","last_clean":"2024-03-22T13:12:13.298804+0000","last_became_active":"2024-03-22T12:56:28.762922+0000","last_became_peered":"2024-03-22T12:56:28.762922+0000","last_unstale":"2024-03-22T13:12:13.298804+0000","last_undegraded":"2024-03-22T13:12:13.298804+0000","last_fullsized":"2024-03-22T13:12:13.298804+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-03-22T13:12:13.298705+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-03-22T13:12:13.298705+0000","last_clean_scrub_stamp":"2024-03-22T13:12:13.298705+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:46:33.663284+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:28.763211+0000","last_change":"2024-03-22T13:12:28.763271+0000","last_active":"2024-03-22T13:12:28.763211+0000","last_peered":"2024-03-22T13:12:28.763211+0000","last_clean":"2024-03-22T13:12:28.763211+0000","last_became_active":"2024-03-22T12:56:37.233436+0000","last_became_peered":"2024-03-22T12:56:37.233436+0000","last_unstale":"2024-03-22T13:12:28.763211+0000","last_undegraded":"2024-03-22T13:12:28.763211+0000","last_fullsized":"2024-03-22T13:12:28.763211+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:28.763081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:28.763081+0000","last_clean_scrub_stamp":"2024-03-22T13:12:28.763081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:26:18.317732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:26.575174+0000","last_change":"2024-03-22T13:12:26.575215+0000","last_active":"2024-03-22T13:12:26.575174+0000","last_peered":"2024-03-22T13:12:26.575174+0000","last_clean":"2024-03-22T13:12:26.575174+0000","last_became_active":"2024-03-22T12:56:30.314820+0000","last_became_peered":"2024-03-22T12:56:30.314820+0000","last_unstale":"2024-03-22T13:12:26.575174+0000","last_undegraded":"2024-03-22T13:12:26.575174+0000","last_fullsized":"2024-03-22T13:12:26.575174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:26.575064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:26.575064+0000","last_clean_scrub_stamp":"2024-03-22T13:12:26.575064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:29:46.104538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"31'5","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:14.888195+0000","last_change":"2024-03-22T13:12:14.888228+0000","last_active":"2024-03-22T13:12:14.888195+0000","last_peered":"2024-03-22T13:12:14.888195+0000","last_clean":"2024-03-22T13:12:14.888195+0000","last_became_active":"2024-03-22T12:56:28.768651+0000","last_became_peered":"2024-03-22T12:56:28.768651+0000","last_unstale":"2024-03-22T13:12:14.888195+0000","last_undegraded":"2024-03-22T13:12:14.888195+0000","last_fullsized":"2024-03-22T13:12:14.888195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-03-22T13:12:14.888099+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-03-22T13:12:14.888099+0000","last_clean_scrub_stamp":"2024-03-22T13:12:14.888099+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:43:09.951561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.408522+0000","last_change":"2024-03-22T13:12:21.408522+0000","last_active":"2024-03-22T13:12:21.408522+0000","last_peered":"2024-03-22T13:12:21.408522+0000","last_clean":"2024-03-22T13:12:21.408522+0000","last_became_active":"2024-03-22T12:56:37.052585+0000","last_became_peered":"2024-03-22T12:56:37.052585+0000","last_unstale":"2024-03-22T13:12:21.408522+0000","last_undegraded":"2024-03-22T13:12:21.408522+0000","last_fullsized":"2024-03-22T13:12:21.408522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:21.408375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:21.408375+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.408375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:56:05.015737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:31.009134+0000","last_change":"2024-03-22T13:12:31.009180+0000","last_active":"2024-03-22T13:12:31.009134+0000","last_peered":"2024-03-22T13:12:31.009134+0000","last_clean":"2024-03-22T13:12:31.009134+0000","last_became_active":"2024-03-22T12:56:30.308303+0000","last_became_peered":"2024-03-22T12:56:30.308303+0000","last_unstale":"2024-03-22T13:12:31.009134+0000","last_undegraded":"2024-03-22T13:12:31.009134+0000","last_fullsized":"2024-03-22T13:12:31.009134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:31.009018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:31.009018+0000","last_clean_scrub_stamp":"2024-03-22T13:12:31.009018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:17:18.739723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"31'8","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:14.281831+0000","last_change":"2024-03-22T13:12:14.281886+0000","last_active":"2024-03-22T13:12:14.281831+0000","last_peered":"2024-03-22T13:12:14.281831+0000","last_clean":"2024-03-22T13:12:14.281831+0000","last_became_active":"2024-03-22T12:56:28.766124+0000","last_became_peered":"2024-03-22T12:56:28.766124+0000","last_unstale":"2024-03-22T13:12:14.281831+0000","last_undegraded":"2024-03-22T13:12:14.281831+0000","last_fullsized":"2024-03-22T13:12:14.281831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-03-22T13:12:14.281728+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-03-22T13:12:14.281728+0000","last_clean_scrub_stamp":"2024-03-22T13:12:14.281728+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:59:51.862511+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:31.436650+0000","last_change":"2024-03-22T13:12:31.436689+0000","last_active":"2024-03-22T13:12:31.436650+0000","last_peered":"2024-03-22T13:12:31.436650+0000","last_clean":"2024-03-22T13:12:31.436650+0000","last_became_active":"2024-03-22T12:56:37.233813+0000","last_became_peered":"2024-03-22T12:56:37.233813+0000","last_unstale":"2024-03-22T13:12:31.436650+0000","last_undegraded":"2024-03-22T13:12:31.436650+0000","last_fullsized":"2024-03-22T13:12:31.436650+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:31.436535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:31.436535+0000","last_clean_scrub_stamp":"2024-03-22T13:12:31.436535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:58:21.732867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:29.064288+0000","last_change":"2024-03-22T13:12:29.064338+0000","last_active":"2024-03-22T13:12:29.064288+0000","last_peered":"2024-03-22T13:12:29.064288+0000","last_clean":"2024-03-22T13:12:29.064288+0000","last_became_active":"2024-03-22T12:56:30.307757+0000","last_became_peered":"2024-03-22T12:56:30.307757+0000","last_unstale":"2024-03-22T13:12:29.064288+0000","last_undegraded":"2024-03-22T13:12:29.064288+0000","last_fullsized":"2024-03-22T13:12:29.064288+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:29.064173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:29.064173+0000","last_clean_scrub_stamp":"2024-03-22T13:12:29.064173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:21:55.076406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:14.418701+0000","last_change":"2024-03-22T13:12:14.418701+0000","last_active":"2024-03-22T13:12:14.418701+0000","last_peered":"2024-03-22T13:12:14.418701+0000","last_clean":"2024-03-22T13:12:14.418701+0000","last_became_active":"2024-03-22T12:56:28.769366+0000","last_became_peered":"2024-03-22T12:56:28.769366+0000","last_unstale":"2024-03-22T13:12:14.418701+0000","last_undegraded":"2024-03-22T13:12:14.418701+0000","last_fullsized":"2024-03-22T13:12:14.418701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-03-22T13:12:14.418560+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-03-22T13:12:14.418560+0000","last_clean_scrub_stamp":"2024-03-22T13:12:14.418560+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:00:26.706022+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.329616+0000","last_change":"2024-03-22T13:12:21.329636+0000","last_active":"2024-03-22T13:12:21.329616+0000","last_peered":"2024-03-22T13:12:21.329616+0000","last_clean":"2024-03-22T13:12:21.329616+0000","last_became_active":"2024-03-22T12:56:37.038952+0000","last_became_peered":"2024-03-22T12:56:37.038952+0000","last_unstale":"2024-03-22T13:12:21.329616+0000","last_undegraded":"2024-03-22T13:12:21.329616+0000","last_fullsized":"2024-03-22T13:12:21.329616+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:21.329549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:21.329549+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.329549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:09:44.714899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:32.013995+0000","last_change":"2024-03-22T13:12:32.014029+0000","last_active":"2024-03-22T13:12:32.013995+0000","last_peered":"2024-03-22T13:12:32.013995+0000","last_clean":"2024-03-22T13:12:32.013995+0000","last_became_active":"2024-03-22T12:56:30.308920+0000","last_became_peered":"2024-03-22T12:56:30.308920+0000","last_unstale":"2024-03-22T13:12:32.013995+0000","last_undegraded":"2024-03-22T13:12:32.013995+0000","last_fullsized":"2024-03-22T13:12:32.013995+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:32.013922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:32.013922+0000","last_clean_scrub_stamp":"2024-03-22T13:12:32.013922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:55:20.016383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"31'26","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:15.315156+0000","last_change":"2024-03-22T13:12:15.315197+0000","last_active":"2024-03-22T13:12:15.315156+0000","last_peered":"2024-03-22T13:12:15.315156+0000","last_clean":"2024-03-22T13:12:15.315156+0000","last_became_active":"2024-03-22T12:56:28.774459+0000","last_became_peered":"2024-03-22T12:56:28.774459+0000","last_unstale":"2024-03-22T13:12:15.315156+0000","last_undegraded":"2024-03-22T13:12:15.315156+0000","last_fullsized":"2024-03-22T13:12:15.315156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'26","last_scrub_stamp":"2024-03-22T13:12:15.315089+0000","last_deep_scrub":"31'26","last_deep_scrub_stamp":"2024-03-22T13:12:15.315089+0000","last_clean_scrub_stamp":"2024-03-22T13:12:15.315089+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:11:32.688543+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.388347+0000","last_change":"2024-03-22T13:12:20.388403+0000","last_active":"2024-03-22T13:12:20.388347+0000","last_peered":"2024-03-22T13:12:20.388347+0000","last_clean":"2024-03-22T13:12:20.388347+0000","last_became_active":"2024-03-22T12:56:37.048415+0000","last_became_peered":"2024-03-22T12:56:37.048415+0000","last_unstale":"2024-03-22T13:12:20.388347+0000","last_undegraded":"2024-03-22T13:12:20.388347+0000","last_fullsized":"2024-03-22T13:12:20.388347+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:20.388105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:20.388105+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.388105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:14:37.274197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.724368+0000","last_change":"2024-03-22T13:12:22.724403+0000","last_active":"2024-03-22T13:12:22.724368+0000","last_peered":"2024-03-22T13:12:22.724368+0000","last_clean":"2024-03-22T13:12:22.724368+0000","last_became_active":"2024-03-22T12:56:30.302305+0000","last_became_peered":"2024-03-22T12:56:30.302305+0000","last_unstale":"2024-03-22T13:12:22.724368+0000","last_undegraded":"2024-03-22T13:12:22.724368+0000","last_fullsized":"2024-03-22T13:12:22.724368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:22.724260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:22.724260+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.724260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:34:16.281945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"31'8","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.604241+0000","last_change":"2024-03-22T13:12:17.604285+0000","last_active":"2024-03-22T13:12:17.604241+0000","last_peered":"2024-03-22T13:12:17.604241+0000","last_clean":"2024-03-22T13:12:17.604241+0000","last_became_active":"2024-03-22T12:56:28.754412+0000","last_became_peered":"2024-03-22T12:56:28.754412+0000","last_unstale":"2024-03-22T13:12:17.604241+0000","last_undegraded":"2024-03-22T13:12:17.604241+0000","last_fullsized":"2024-03-22T13:12:17.604241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-03-22T13:12:17.604140+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-03-22T13:12:17.604140+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.604140+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:27:23.392654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:28.787557+0000","last_change":"2024-03-22T13:12:28.787598+0000","last_active":"2024-03-22T13:12:28.787557+0000","last_peered":"2024-03-22T13:12:28.787557+0000","last_clean":"2024-03-22T13:12:28.787557+0000","last_became_active":"2024-03-22T12:56:37.234317+0000","last_became_peered":"2024-03-22T12:56:37.234317+0000","last_unstale":"2024-03-22T13:12:28.787557+0000","last_undegraded":"2024-03-22T13:12:28.787557+0000","last_fullsized":"2024-03-22T13:12:28.787557+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:28.787455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:28.787455+0000","last_clean_scrub_stamp":"2024-03-22T13:12:28.787455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:28:20.940207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:30.037517+0000","last_change":"2024-03-22T13:12:30.037550+0000","last_active":"2024-03-22T13:12:30.037517+0000","last_peered":"2024-03-22T13:12:30.037517+0000","last_clean":"2024-03-22T13:12:30.037517+0000","last_became_active":"2024-03-22T12:56:30.307844+0000","last_became_peered":"2024-03-22T12:56:30.307844+0000","last_unstale":"2024-03-22T13:12:30.037517+0000","last_undegraded":"2024-03-22T13:12:30.037517+0000","last_fullsized":"2024-03-22T13:12:30.037517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:30.037411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:30.037411+0000","last_clean_scrub_stamp":"2024-03-22T13:12:30.037411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:26:38.430182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"31'8","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.506637+0000","last_change":"2024-03-22T13:12:21.506678+0000","last_active":"2024-03-22T13:12:21.506637+0000","last_peered":"2024-03-22T13:12:21.506637+0000","last_clean":"2024-03-22T13:12:21.506637+0000","last_became_active":"2024-03-22T12:56:28.776700+0000","last_became_peered":"2024-03-22T12:56:28.776700+0000","last_unstale":"2024-03-22T13:12:21.506637+0000","last_undegraded":"2024-03-22T13:12:21.506637+0000","last_fullsized":"2024-03-22T13:12:21.506637+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-03-22T13:12:21.506569+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-03-22T13:12:21.506569+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.506569+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:11:58.250268+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:31.772248+0000","last_change":"2024-03-22T13:12:31.772286+0000","last_active":"2024-03-22T13:12:31.772248+0000","last_peered":"2024-03-22T13:12:31.772248+0000","last_clean":"2024-03-22T13:12:31.772248+0000","last_became_active":"2024-03-22T12:57:04.948424+0000","last_became_peered":"2024-03-22T12:57:04.948424+0000","last_unstale":"2024-03-22T13:12:31.772248+0000","last_undegraded":"2024-03-22T13:12:31.772248+0000","last_fullsized":"2024-03-22T13:12:31.772248+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:31.772137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:31.772137+0000","last_clean_scrub_stamp":"2024-03-22T13:12:31.772137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:55:12.578207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.536903+0000","last_change":"2024-03-22T13:12:12.536941+0000","last_active":"2024-03-22T13:12:12.536903+0000","last_peered":"2024-03-22T13:12:12.536903+0000","last_clean":"2024-03-22T13:12:12.536903+0000","last_became_active":"2024-03-22T12:56:30.298086+0000","last_became_peered":"2024-03-22T12:56:30.298086+0000","last_unstale":"2024-03-22T13:12:12.536903+0000","last_undegraded":"2024-03-22T13:12:12.536903+0000","last_fullsized":"2024-03-22T13:12:12.536903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:12.536765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:12.536765+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.536765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:09:23.060896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"31'15","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:09.527931+0000","last_change":"2024-03-22T13:12:09.527981+0000","last_active":"2024-03-22T13:12:09.527931+0000","last_peered":"2024-03-22T13:12:09.527931+0000","last_clean":"2024-03-22T13:12:09.527931+0000","last_became_active":"2024-03-22T12:56:28.776652+0000","last_became_peered":"2024-03-22T12:56:28.776652+0000","last_unstale":"2024-03-22T13:12:09.527931+0000","last_undegraded":"2024-03-22T13:12:09.527931+0000","last_fullsized":"2024-03-22T13:12:09.527931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-03-22T13:12:09.527826+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-03-22T13:12:09.527826+0000","last_clean_scrub_stamp":"2024-03-22T13:12:09.527826+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T17:45:45.449957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:32.444107+0000","last_change":"2024-03-22T13:12:32.444155+0000","last_active":"2024-03-22T13:12:32.444107+0000","last_peered":"2024-03-22T13:12:32.444107+0000","last_clean":"2024-03-22T13:12:32.444107+0000","last_became_active":"2024-03-22T12:56:37.234731+0000","last_became_peered":"2024-03-22T12:56:37.234731+0000","last_unstale":"2024-03-22T13:12:32.444107+0000","last_undegraded":"2024-03-22T13:12:32.444107+0000","last_fullsized":"2024-03-22T13:12:32.444107+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:32.443985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:32.443985+0000","last_clean_scrub_stamp":"2024-03-22T13:12:32.443985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:28:24.596737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.011006+0000","last_change":"2024-03-22T13:12:17.011040+0000","last_active":"2024-03-22T13:12:17.011006+0000","last_peered":"2024-03-22T13:12:17.011006+0000","last_clean":"2024-03-22T13:12:17.011006+0000","last_became_active":"2024-03-22T12:56:30.271855+0000","last_became_peered":"2024-03-22T12:56:30.271855+0000","last_unstale":"2024-03-22T13:12:17.011006+0000","last_undegraded":"2024-03-22T13:12:17.011006+0000","last_fullsized":"2024-03-22T13:12:17.011006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:17.010938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:17.010938+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.010938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:01:23.622983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"31'20","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.681148+0000","last_change":"2024-03-22T13:12:19.681219+0000","last_active":"2024-03-22T13:12:19.681148+0000","last_peered":"2024-03-22T13:12:19.681148+0000","last_clean":"2024-03-22T13:12:19.681148+0000","last_became_active":"2024-03-22T12:56:28.759477+0000","last_became_peered":"2024-03-22T12:56:28.759477+0000","last_unstale":"2024-03-22T13:12:19.681148+0000","last_undegraded":"2024-03-22T13:12:19.681148+0000","last_fullsized":"2024-03-22T13:12:19.681148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'20","last_scrub_stamp":"2024-03-22T13:12:19.681026+0000","last_deep_scrub":"31'20","last_deep_scrub_stamp":"2024-03-22T13:12:19.681026+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.681026+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:36:52.339806+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.910560+0000","last_change":"2024-03-22T13:12:22.910606+0000","last_active":"2024-03-22T13:12:22.910560+0000","last_peered":"2024-03-22T13:12:22.910560+0000","last_clean":"2024-03-22T13:12:22.910560+0000","last_became_active":"2024-03-22T12:56:37.055908+0000","last_became_peered":"2024-03-22T12:56:37.055908+0000","last_unstale":"2024-03-22T13:12:22.910560+0000","last_undegraded":"2024-03-22T13:12:22.910560+0000","last_fullsized":"2024-03-22T13:12:22.910560+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:22.910447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:22.910447+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.910447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:10:11.730098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:16.412317+0000","last_change":"2024-03-22T13:12:16.412336+0000","last_active":"2024-03-22T13:12:16.412317+0000","last_peered":"2024-03-22T13:12:16.412317+0000","last_clean":"2024-03-22T13:12:16.412317+0000","last_became_active":"2024-03-22T12:56:30.304837+0000","last_became_peered":"2024-03-22T12:56:30.304837+0000","last_unstale":"2024-03-22T13:12:16.412317+0000","last_undegraded":"2024-03-22T13:12:16.412317+0000","last_fullsized":"2024-03-22T13:12:16.412317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:16.412247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:16.412247+0000","last_clean_scrub_stamp":"2024-03-22T13:12:16.412247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:21:42.343475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:13.900173+0000","last_change":"2024-03-22T13:12:13.900222+0000","last_active":"2024-03-22T13:12:13.900173+0000","last_peered":"2024-03-22T13:12:13.900173+0000","last_clean":"2024-03-22T13:12:13.900173+0000","last_became_active":"2024-03-22T12:56:28.768658+0000","last_became_peered":"2024-03-22T12:56:28.768658+0000","last_unstale":"2024-03-22T13:12:13.900173+0000","last_undegraded":"2024-03-22T13:12:13.900173+0000","last_fullsized":"2024-03-22T13:12:13.900173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-03-22T13:12:13.900078+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-03-22T13:12:13.900078+0000","last_clean_scrub_stamp":"2024-03-22T13:12:13.900078+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:10:07.120053+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.312133+0000","last_change":"2024-03-22T13:12:20.312153+0000","last_active":"2024-03-22T13:12:20.312133+0000","last_peered":"2024-03-22T13:12:20.312133+0000","last_clean":"2024-03-22T13:12:20.312133+0000","last_became_active":"2024-03-22T12:56:37.037796+0000","last_became_peered":"2024-03-22T12:56:37.037796+0000","last_unstale":"2024-03-22T13:12:20.312133+0000","last_undegraded":"2024-03-22T13:12:20.312133+0000","last_fullsized":"2024-03-22T13:12:20.312133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:20.312066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:20.312066+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.312066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:47:01.971718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:15.475560+0000","last_change":"2024-03-22T13:12:15.475602+0000","last_active":"2024-03-22T13:12:15.475560+0000","last_peered":"2024-03-22T13:12:15.475560+0000","last_clean":"2024-03-22T13:12:15.475560+0000","last_became_active":"2024-03-22T12:56:30.308357+0000","last_became_peered":"2024-03-22T12:56:30.308357+0000","last_unstale":"2024-03-22T13:12:15.475560+0000","last_undegraded":"2024-03-22T13:12:15.475560+0000","last_fullsized":"2024-03-22T13:12:15.475560+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:15.475416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:15.475416+0000","last_clean_scrub_stamp":"2024-03-22T13:12:15.475416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T21:33:19.969812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"31'6","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.479847+0000","last_change":"2024-03-22T13:12:18.479897+0000","last_active":"2024-03-22T13:12:18.479847+0000","last_peered":"2024-03-22T13:12:18.479847+0000","last_clean":"2024-03-22T13:12:18.479847+0000","last_became_active":"2024-03-22T12:56:28.761907+0000","last_became_peered":"2024-03-22T12:56:28.761907+0000","last_unstale":"2024-03-22T13:12:18.479847+0000","last_undegraded":"2024-03-22T13:12:18.479847+0000","last_fullsized":"2024-03-22T13:12:18.479847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-03-22T13:12:18.479747+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-03-22T13:12:18.479747+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.479747+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:14:32.118387+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:25.829042+0000","last_change":"2024-03-22T13:12:25.829081+0000","last_active":"2024-03-22T13:12:25.829042+0000","last_peered":"2024-03-22T13:12:25.829042+0000","last_clean":"2024-03-22T13:12:25.829042+0000","last_became_active":"2024-03-22T12:56:37.046003+0000","last_became_peered":"2024-03-22T12:56:37.046003+0000","last_unstale":"2024-03-22T13:12:25.829042+0000","last_undegraded":"2024-03-22T13:12:25.829042+0000","last_fullsized":"2024-03-22T13:12:25.829042+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:25.828937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:25.828937+0000","last_clean_scrub_stamp":"2024-03-22T13:12:25.828937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:28:11.123094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:15.526139+0000","last_change":"2024-03-22T13:12:15.526176+0000","last_active":"2024-03-22T13:12:15.526139+0000","last_peered":"2024-03-22T13:12:15.526139+0000","last_clean":"2024-03-22T13:12:15.526139+0000","last_became_active":"2024-03-22T12:56:30.310373+0000","last_became_peered":"2024-03-22T12:56:30.310373+0000","last_unstale":"2024-03-22T13:12:15.526139+0000","last_undegraded":"2024-03-22T13:12:15.526139+0000","last_fullsized":"2024-03-22T13:12:15.526139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:15.526029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:15.526029+0000","last_clean_scrub_stamp":"2024-03-22T13:12:15.526029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:49:25.095529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"31'7","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:20.683658+0000","last_change":"2024-03-22T13:12:20.683684+0000","last_active":"2024-03-22T13:12:20.683658+0000","last_peered":"2024-03-22T13:12:20.683658+0000","last_clean":"2024-03-22T13:12:20.683658+0000","last_became_active":"2024-03-22T12:56:28.762176+0000","last_became_peered":"2024-03-22T12:56:28.762176+0000","last_unstale":"2024-03-22T13:12:20.683658+0000","last_undegraded":"2024-03-22T13:12:20.683658+0000","last_fullsized":"2024-03-22T13:12:20.683658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-03-22T13:12:20.683601+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-03-22T13:12:20.683601+0000","last_clean_scrub_stamp":"2024-03-22T13:12:20.683601+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:30:16.420838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:29.794089+0000","last_change":"2024-03-22T13:12:29.794127+0000","last_active":"2024-03-22T13:12:29.794089+0000","last_peered":"2024-03-22T13:12:29.794089+0000","last_clean":"2024-03-22T13:12:29.794089+0000","last_became_active":"2024-03-22T12:56:37.234289+0000","last_became_peered":"2024-03-22T12:56:37.234289+0000","last_unstale":"2024-03-22T13:12:29.794089+0000","last_undegraded":"2024-03-22T13:12:29.794089+0000","last_fullsized":"2024-03-22T13:12:29.794089+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:29.793984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:29.793984+0000","last_clean_scrub_stamp":"2024-03-22T13:12:29.793984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:21:56.537209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:24.464077+0000","last_change":"2024-03-22T13:12:24.464117+0000","last_active":"2024-03-22T13:12:24.464077+0000","last_peered":"2024-03-22T13:12:24.464077+0000","last_clean":"2024-03-22T13:12:24.464077+0000","last_became_active":"2024-03-22T12:56:37.239195+0000","last_became_peered":"2024-03-22T12:56:37.239195+0000","last_unstale":"2024-03-22T13:12:24.464077+0000","last_undegraded":"2024-03-22T13:12:24.464077+0000","last_fullsized":"2024-03-22T13:12:24.464077+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:24.463969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:24.463969+0000","last_clean_scrub_stamp":"2024-03-22T13:12:24.463969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:51:00.188647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.884343+0000","last_change":"2024-03-22T13:12:17.884382+0000","last_active":"2024-03-22T13:12:17.884343+0000","last_peered":"2024-03-22T13:12:17.884343+0000","last_clean":"2024-03-22T13:12:17.884343+0000","last_became_active":"2024-03-22T12:56:30.267725+0000","last_became_peered":"2024-03-22T12:56:30.267725+0000","last_unstale":"2024-03-22T13:12:17.884343+0000","last_undegraded":"2024-03-22T13:12:17.884343+0000","last_fullsized":"2024-03-22T13:12:17.884343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:17.884238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:17.884238+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.884238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:08:43.704822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.493898+0000","last_change":"2024-03-22T13:12:17.493921+0000","last_active":"2024-03-22T13:12:17.493898+0000","last_peered":"2024-03-22T13:12:17.493898+0000","last_clean":"2024-03-22T13:12:17.493898+0000","last_became_active":"2024-03-22T12:56:28.761217+0000","last_became_peered":"2024-03-22T12:56:28.761217+0000","last_unstale":"2024-03-22T13:12:17.493898+0000","last_undegraded":"2024-03-22T13:12:17.493898+0000","last_fullsized":"2024-03-22T13:12:17.493898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-03-22T13:12:17.493837+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-03-22T13:12:17.493837+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.493837+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:46:59.153488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:27.746199+0000","last_change":"2024-03-22T13:12:27.746241+0000","last_active":"2024-03-22T13:12:27.746199+0000","last_peered":"2024-03-22T13:12:27.746199+0000","last_clean":"2024-03-22T13:12:27.746199+0000","last_became_active":"2024-03-22T12:56:37.051322+0000","last_became_peered":"2024-03-22T12:56:37.051322+0000","last_unstale":"2024-03-22T13:12:27.746199+0000","last_undegraded":"2024-03-22T13:12:27.746199+0000","last_fullsized":"2024-03-22T13:12:27.746199+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:27.746073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:27.746073+0000","last_clean_scrub_stamp":"2024-03-22T13:12:27.746073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:41:34.043255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:23.795182+0000","last_change":"2024-03-22T13:12:23.795237+0000","last_active":"2024-03-22T13:12:23.795182+0000","last_peered":"2024-03-22T13:12:23.795182+0000","last_clean":"2024-03-22T13:12:23.795182+0000","last_became_active":"2024-03-22T12:56:30.298089+0000","last_became_peered":"2024-03-22T12:56:30.298089+0000","last_unstale":"2024-03-22T13:12:23.795182+0000","last_undegraded":"2024-03-22T13:12:23.795182+0000","last_fullsized":"2024-03-22T13:12:23.795182+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:23.795004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:23.795004+0000","last_clean_scrub_stamp":"2024-03-22T13:12:23.795004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:10:43.013399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"31'5","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:16.719287+0000","last_change":"2024-03-22T13:12:16.719335+0000","last_active":"2024-03-22T13:12:16.719287+0000","last_peered":"2024-03-22T13:12:16.719287+0000","last_clean":"2024-03-22T13:12:16.719287+0000","last_became_active":"2024-03-22T12:56:28.755146+0000","last_became_peered":"2024-03-22T12:56:28.755146+0000","last_unstale":"2024-03-22T13:12:16.719287+0000","last_undegraded":"2024-03-22T13:12:16.719287+0000","last_fullsized":"2024-03-22T13:12:16.719287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-03-22T13:12:16.719190+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-03-22T13:12:16.719190+0000","last_clean_scrub_stamp":"2024-03-22T13:12:16.719190+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:03:55.074741+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"31'10","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.648013+0000","last_change":"2024-03-22T13:12:18.648058+0000","last_active":"2024-03-22T13:12:18.648013+0000","last_peered":"2024-03-22T13:12:18.648013+0000","last_clean":"2024-03-22T13:12:18.648013+0000","last_became_active":"2024-03-22T12:56:28.754447+0000","last_became_peered":"2024-03-22T12:56:28.754447+0000","last_unstale":"2024-03-22T13:12:18.648013+0000","last_undegraded":"2024-03-22T13:12:18.648013+0000","last_fullsized":"2024-03-22T13:12:18.648013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-03-22T13:12:18.647938+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-03-22T13:12:18.647938+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.647938+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:01:25.259982+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:22.353575+0000","last_change":"2024-03-22T13:12:22.353611+0000","last_active":"2024-03-22T13:12:22.353575+0000","last_peered":"2024-03-22T13:12:22.353575+0000","last_clean":"2024-03-22T13:12:22.353575+0000","last_became_active":"2024-03-22T12:57:05.044195+0000","last_became_peered":"2024-03-22T12:57:05.044195+0000","last_unstale":"2024-03-22T13:12:22.353575+0000","last_undegraded":"2024-03-22T13:12:22.353575+0000","last_fullsized":"2024-03-22T13:12:22.353575+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:22.353431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:22.353431+0000","last_clean_scrub_stamp":"2024-03-22T13:12:22.353431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:10:59.993799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.346062+0000","last_change":"2024-03-22T13:12:17.346101+0000","last_active":"2024-03-22T13:12:17.346062+0000","last_peered":"2024-03-22T13:12:17.346062+0000","last_clean":"2024-03-22T13:12:17.346062+0000","last_became_active":"2024-03-22T12:56:37.062996+0000","last_became_peered":"2024-03-22T12:56:37.062996+0000","last_unstale":"2024-03-22T13:12:17.346062+0000","last_undegraded":"2024-03-22T13:12:17.346062+0000","last_fullsized":"2024-03-22T13:12:17.346062+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:17.345933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:17.345933+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.345933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T16:02:00.305974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"31'7","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.737124+0000","last_change":"2024-03-22T13:12:17.737195+0000","last_active":"2024-03-22T13:12:17.737124+0000","last_peered":"2024-03-22T13:12:17.737124+0000","last_clean":"2024-03-22T13:12:17.737124+0000","last_became_active":"2024-03-22T12:56:28.756018+0000","last_became_peered":"2024-03-22T12:56:28.756018+0000","last_unstale":"2024-03-22T13:12:17.737124+0000","last_undegraded":"2024-03-22T13:12:17.737124+0000","last_fullsized":"2024-03-22T13:12:17.737124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-03-22T13:12:17.736943+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-03-22T13:12:17.736943+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.736943+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:11:54.480642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:17.393622+0000","last_change":"2024-03-22T13:12:17.393659+0000","last_active":"2024-03-22T13:12:17.393622+0000","last_peered":"2024-03-22T13:12:17.393622+0000","last_clean":"2024-03-22T13:12:17.393622+0000","last_became_active":"2024-03-22T12:56:30.311461+0000","last_became_peered":"2024-03-22T12:56:30.311461+0000","last_unstale":"2024-03-22T13:12:17.393622+0000","last_undegraded":"2024-03-22T13:12:17.393622+0000","last_fullsized":"2024-03-22T13:12:17.393622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:17.393515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:17.393515+0000","last_clean_scrub_stamp":"2024-03-22T13:12:17.393515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:27:56.287690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:26.869280+0000","last_change":"2024-03-22T13:12:26.869320+0000","last_active":"2024-03-22T13:12:26.869280+0000","last_peered":"2024-03-22T13:12:26.869280+0000","last_clean":"2024-03-22T13:12:26.869280+0000","last_became_active":"2024-03-22T12:56:37.051935+0000","last_became_peered":"2024-03-22T12:56:37.051935+0000","last_unstale":"2024-03-22T13:12:26.869280+0000","last_undegraded":"2024-03-22T13:12:26.869280+0000","last_fullsized":"2024-03-22T13:12:26.869280+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:26.869175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:26.869175+0000","last_clean_scrub_stamp":"2024-03-22T13:12:26.869175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:29:54.054542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:26.719716+0000","last_change":"2024-03-22T13:12:26.719748+0000","last_active":"2024-03-22T13:12:26.719716+0000","last_peered":"2024-03-22T13:12:26.719716+0000","last_clean":"2024-03-22T13:12:26.719716+0000","last_became_active":"2024-03-22T12:56:30.310214+0000","last_became_peered":"2024-03-22T12:56:30.310214+0000","last_unstale":"2024-03-22T13:12:26.719716+0000","last_undegraded":"2024-03-22T13:12:26.719716+0000","last_fullsized":"2024-03-22T13:12:26.719716+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:26.719635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:26.719635+0000","last_clean_scrub_stamp":"2024-03-22T13:12:26.719635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:09:41.420173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.350390+0000","last_change":"2024-03-22T13:12:19.350425+0000","last_active":"2024-03-22T13:12:19.350390+0000","last_peered":"2024-03-22T13:12:19.350390+0000","last_clean":"2024-03-22T13:12:19.350390+0000","last_became_active":"2024-03-22T12:57:04.187801+0000","last_became_peered":"2024-03-22T12:57:04.187801+0000","last_unstale":"2024-03-22T13:12:19.350390+0000","last_undegraded":"2024-03-22T13:12:19.350390+0000","last_fullsized":"2024-03-22T13:12:19.350390+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:19.350256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:19.350256+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.350256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:57:19.313010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:28.518343+0000","last_change":"2024-03-22T13:12:28.518362+0000","last_active":"2024-03-22T13:12:28.518343+0000","last_peered":"2024-03-22T13:12:28.518343+0000","last_clean":"2024-03-22T13:12:28.518343+0000","last_became_active":"2024-03-22T12:56:30.317633+0000","last_became_peered":"2024-03-22T12:56:30.317633+0000","last_unstale":"2024-03-22T13:12:28.518343+0000","last_undegraded":"2024-03-22T13:12:28.518343+0000","last_fullsized":"2024-03-22T13:12:28.518343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:28.518282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:28.518282+0000","last_clean_scrub_stamp":"2024-03-22T13:12:28.518282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:26:26.630833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:13.491821+0000","last_change":"2024-03-22T13:12:13.491856+0000","last_active":"2024-03-22T13:12:13.491821+0000","last_peered":"2024-03-22T13:12:13.491821+0000","last_clean":"2024-03-22T13:12:13.491821+0000","last_became_active":"2024-03-22T12:56:30.299775+0000","last_became_peered":"2024-03-22T12:56:30.299775+0000","last_unstale":"2024-03-22T13:12:13.491821+0000","last_undegraded":"2024-03-22T13:12:13.491821+0000","last_fullsized":"2024-03-22T13:12:13.491821+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:13.491717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:13.491717+0000","last_clean_scrub_stamp":"2024-03-22T13:12:13.491717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:10:06.711692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:24.808603+0000","last_change":"2024-03-22T13:12:24.808662+0000","last_active":"2024-03-22T13:12:24.808603+0000","last_peered":"2024-03-22T13:12:24.808603+0000","last_clean":"2024-03-22T13:12:24.808603+0000","last_became_active":"2024-03-22T12:56:30.311339+0000","last_became_peered":"2024-03-22T12:56:30.311339+0000","last_unstale":"2024-03-22T13:12:24.808603+0000","last_undegraded":"2024-03-22T13:12:24.808603+0000","last_fullsized":"2024-03-22T13:12:24.808603+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:24.808443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:24.808443+0000","last_clean_scrub_stamp":"2024-03-22T13:12:24.808443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:26:22.920994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.873025+0000","last_change":"2024-03-22T13:12:18.873064+0000","last_active":"2024-03-22T13:12:18.873025+0000","last_peered":"2024-03-22T13:12:18.873025+0000","last_clean":"2024-03-22T13:12:18.873025+0000","last_became_active":"2024-03-22T12:56:30.272631+0000","last_became_peered":"2024-03-22T12:56:30.272631+0000","last_unstale":"2024-03-22T13:12:18.873025+0000","last_undegraded":"2024-03-22T13:12:18.873025+0000","last_fullsized":"2024-03-22T13:12:18.873025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:18.872925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:18.872925+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.872925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:59:23.537863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:21.700205+0000","last_change":"2024-03-22T13:12:21.700222+0000","last_active":"2024-03-22T13:12:21.700205+0000","last_peered":"2024-03-22T13:12:21.700205+0000","last_clean":"2024-03-22T13:12:21.700205+0000","last_became_active":"2024-03-22T12:56:30.227535+0000","last_became_peered":"2024-03-22T12:56:30.227535+0000","last_unstale":"2024-03-22T13:12:21.700205+0000","last_undegraded":"2024-03-22T13:12:21.700205+0000","last_fullsized":"2024-03-22T13:12:21.700205+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:21.700143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:21.700143+0000","last_clean_scrub_stamp":"2024-03-22T13:12:21.700143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:10:59.340511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:13.742657+0000","last_change":"2024-03-22T13:12:13.742696+0000","last_active":"2024-03-22T13:12:13.742657+0000","last_peered":"2024-03-22T13:12:13.742657+0000","last_clean":"2024-03-22T13:12:13.742657+0000","last_became_active":"2024-03-22T12:56:30.308562+0000","last_became_peered":"2024-03-22T12:56:30.308562+0000","last_unstale":"2024-03-22T13:12:13.742657+0000","last_undegraded":"2024-03-22T13:12:13.742657+0000","last_fullsized":"2024-03-22T13:12:13.742657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:13.742546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:13.742546+0000","last_clean_scrub_stamp":"2024-03-22T13:12:13.742546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:46:55.402197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:18.326253+0000","last_change":"2024-03-22T13:12:18.326304+0000","last_active":"2024-03-22T13:12:18.326253+0000","last_peered":"2024-03-22T13:12:18.326253+0000","last_clean":"2024-03-22T13:12:18.326253+0000","last_became_active":"2024-03-22T12:57:04.179855+0000","last_became_peered":"2024-03-22T12:57:04.179855+0000","last_unstale":"2024-03-22T13:12:18.326253+0000","last_undegraded":"2024-03-22T13:12:18.326253+0000","last_fullsized":"2024-03-22T13:12:18.326253+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:18.326089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:18.326089+0000","last_clean_scrub_stamp":"2024-03-22T13:12:18.326089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:28:19.403110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:26.880927+0000","last_change":"2024-03-22T13:12:26.880979+0000","last_active":"2024-03-22T13:12:26.880927+0000","last_peered":"2024-03-22T13:12:26.880927+0000","last_clean":"2024-03-22T13:12:26.880927+0000","last_became_active":"2024-03-22T12:57:05.044203+0000","last_became_peered":"2024-03-22T12:57:05.044203+0000","last_unstale":"2024-03-22T13:12:26.880927+0000","last_undegraded":"2024-03-22T13:12:26.880927+0000","last_fullsized":"2024-03-22T13:12:26.880927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:26.880731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:26.880731+0000","last_clean_scrub_stamp":"2024-03-22T13:12:26.880731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T23:58:17.177063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:16.518350+0000","last_change":"2024-03-22T13:12:16.518386+0000","last_active":"2024-03-22T13:12:16.518350+0000","last_peered":"2024-03-22T13:12:16.518350+0000","last_clean":"2024-03-22T13:12:16.518350+0000","last_became_active":"2024-03-22T12:56:30.310540+0000","last_became_peered":"2024-03-22T12:56:30.310540+0000","last_unstale":"2024-03-22T13:12:16.518350+0000","last_undegraded":"2024-03-22T13:12:16.518350+0000","last_fullsized":"2024-03-22T13:12:16.518350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:16.518232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:16.518232+0000","last_clean_scrub_stamp":"2024-03-22T13:12:16.518232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:07:37.510484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:11.536116+0000","last_change":"2024-03-22T13:12:11.536116+0000","last_active":"2024-03-22T13:12:11.536116+0000","last_peered":"2024-03-22T13:12:11.536116+0000","last_clean":"2024-03-22T13:12:11.536116+0000","last_became_active":"2024-03-22T12:56:30.298665+0000","last_became_peered":"2024-03-22T12:56:30.298665+0000","last_unstale":"2024-03-22T13:12:11.536116+0000","last_undegraded":"2024-03-22T13:12:11.536116+0000","last_fullsized":"2024-03-22T13:12:11.536116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:11.536023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:11.536023+0000","last_clean_scrub_stamp":"2024-03-22T13:12:11.536023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:07:32.268477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:25.669836+0000","last_change":"2024-03-22T13:12:25.669869+0000","last_active":"2024-03-22T13:12:25.669836+0000","last_peered":"2024-03-22T13:12:25.669836+0000","last_clean":"2024-03-22T13:12:25.669836+0000","last_became_active":"2024-03-22T12:56:30.308845+0000","last_became_peered":"2024-03-22T12:56:30.308845+0000","last_unstale":"2024-03-22T13:12:25.669836+0000","last_undegraded":"2024-03-22T13:12:25.669836+0000","last_fullsized":"2024-03-22T13:12:25.669836+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:25.669764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:25.669764+0000","last_clean_scrub_stamp":"2024-03-22T13:12:25.669764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:55:40.894852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:25.857533+0000","last_change":"2024-03-22T13:12:25.857637+0000","last_active":"2024-03-22T13:12:25.857533+0000","last_peered":"2024-03-22T13:12:25.857533+0000","last_clean":"2024-03-22T13:12:25.857533+0000","last_became_active":"2024-03-22T12:57:04.187416+0000","last_became_peered":"2024-03-22T12:57:04.187416+0000","last_unstale":"2024-03-22T13:12:25.857533+0000","last_undegraded":"2024-03-22T13:12:25.857533+0000","last_fullsized":"2024-03-22T13:12:25.857533+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:25.857281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:25.857281+0000","last_clean_scrub_stamp":"2024-03-22T13:12:25.857281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:29:53.042648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:16.330530+0000","last_change":"2024-03-22T13:12:16.330564+0000","last_active":"2024-03-22T13:12:16.330530+0000","last_peered":"2024-03-22T13:12:16.330530+0000","last_clean":"2024-03-22T13:12:16.330530+0000","last_became_active":"2024-03-22T12:56:30.313404+0000","last_became_peered":"2024-03-22T12:56:30.313404+0000","last_unstale":"2024-03-22T13:12:16.330530+0000","last_undegraded":"2024-03-22T13:12:16.330530+0000","last_fullsized":"2024-03-22T13:12:16.330530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:16.330431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:16.330431+0000","last_clean_scrub_stamp":"2024-03-22T13:12:16.330431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:15:08.333654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:24.499207+0000","last_change":"2024-03-22T13:12:24.499228+0000","last_active":"2024-03-22T13:12:24.499207+0000","last_peered":"2024-03-22T13:12:24.499207+0000","last_clean":"2024-03-22T13:12:24.499207+0000","last_became_active":"2024-03-22T12:56:30.309827+0000","last_became_peered":"2024-03-22T12:56:30.309827+0000","last_unstale":"2024-03-22T13:12:24.499207+0000","last_undegraded":"2024-03-22T13:12:24.499207+0000","last_fullsized":"2024-03-22T13:12:24.499207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:24.499142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:24.499142+0000","last_clean_scrub_stamp":"2024-03-22T13:12:24.499142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:17:17.151415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:29.472144+0000","last_change":"2024-03-22T13:12:29.472164+0000","last_active":"2024-03-22T13:12:29.472144+0000","last_peered":"2024-03-22T13:12:29.472144+0000","last_clean":"2024-03-22T13:12:29.472144+0000","last_became_active":"2024-03-22T12:56:30.321102+0000","last_became_peered":"2024-03-22T12:56:30.321102+0000","last_unstale":"2024-03-22T13:12:29.472144+0000","last_undegraded":"2024-03-22T13:12:29.472144+0000","last_fullsized":"2024-03-22T13:12:29.472144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:29.472083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:29.472083+0000","last_clean_scrub_stamp":"2024-03-22T13:12:29.472083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:28:14.766240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.917055+0000","last_change":"2024-03-22T13:12:19.917181+0000","last_active":"2024-03-22T13:12:19.917055+0000","last_peered":"2024-03-22T13:12:19.917055+0000","last_clean":"2024-03-22T13:12:19.917055+0000","last_became_active":"2024-03-22T12:56:30.307999+0000","last_became_peered":"2024-03-22T12:56:30.307999+0000","last_unstale":"2024-03-22T13:12:19.917055+0000","last_undegraded":"2024-03-22T13:12:19.917055+0000","last_fullsized":"2024-03-22T13:12:19.917055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:19.916825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:19.916825+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.916825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:14:36.802916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:14.446428+0000","last_change":"2024-03-22T13:12:14.446466+0000","last_active":"2024-03-22T13:12:14.446428+0000","last_peered":"2024-03-22T13:12:14.446428+0000","last_clean":"2024-03-22T13:12:14.446428+0000","last_became_active":"2024-03-22T12:56:30.301465+0000","last_became_peered":"2024-03-22T12:56:30.301465+0000","last_unstale":"2024-03-22T13:12:14.446428+0000","last_undegraded":"2024-03-22T13:12:14.446428+0000","last_fullsized":"2024-03-22T13:12:14.446428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:14.446325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:14.446325+0000","last_clean_scrub_stamp":"2024-03-22T13:12:14.446325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T15:14:28.084965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.778991+0000","last_change":"2024-03-22T13:12:19.779042+0000","last_active":"2024-03-22T13:12:19.778991+0000","last_peered":"2024-03-22T13:12:19.778991+0000","last_clean":"2024-03-22T13:12:19.778991+0000","last_became_active":"2024-03-22T12:56:30.266049+0000","last_became_peered":"2024-03-22T12:56:30.266049+0000","last_unstale":"2024-03-22T13:12:19.778991+0000","last_undegraded":"2024-03-22T13:12:19.778991+0000","last_fullsized":"2024-03-22T13:12:19.778991+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:19.778800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:19.778800+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.778800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:29:29.048686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:15.419199+0000","last_change":"2024-03-22T13:12:15.419238+0000","last_active":"2024-03-22T13:12:15.419199+0000","last_peered":"2024-03-22T13:12:15.419199+0000","last_clean":"2024-03-22T13:12:15.419199+0000","last_became_active":"2024-03-22T12:56:30.304680+0000","last_became_peered":"2024-03-22T12:56:30.304680+0000","last_unstale":"2024-03-22T13:12:15.419199+0000","last_undegraded":"2024-03-22T13:12:15.419199+0000","last_fullsized":"2024-03-22T13:12:15.419199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:15.419094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:15.419094+0000","last_clean_scrub_stamp":"2024-03-22T13:12:15.419094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:19:35.065213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:30.716940+0000","last_change":"2024-03-22T13:12:30.716976+0000","last_active":"2024-03-22T13:12:30.716940+0000","last_peered":"2024-03-22T13:12:30.716940+0000","last_clean":"2024-03-22T13:12:30.716940+0000","last_became_active":"2024-03-22T12:57:04.250856+0000","last_became_peered":"2024-03-22T12:57:04.250856+0000","last_unstale":"2024-03-22T13:12:30.716940+0000","last_undegraded":"2024-03-22T13:12:30.716940+0000","last_fullsized":"2024-03-22T13:12:30.716940+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:30.716832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:30.716832+0000","last_clean_scrub_stamp":"2024-03-22T13:12:30.716832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:07:45.825983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:24.700476+0000","last_change":"2024-03-22T13:12:24.700520+0000","last_active":"2024-03-22T13:12:24.700476+0000","last_peered":"2024-03-22T13:12:24.700476+0000","last_clean":"2024-03-22T13:12:24.700476+0000","last_became_active":"2024-03-22T12:56:30.306683+0000","last_became_peered":"2024-03-22T12:56:30.306683+0000","last_unstale":"2024-03-22T13:12:24.700476+0000","last_undegraded":"2024-03-22T13:12:24.700476+0000","last_fullsized":"2024-03-22T13:12:24.700476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:24.700380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:24.700380+0000","last_clean_scrub_stamp":"2024-03-22T13:12:24.700380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:12:27.963899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:27.029816+0000","last_change":"2024-03-22T13:12:27.029834+0000","last_active":"2024-03-22T13:12:27.029816+0000","last_peered":"2024-03-22T13:12:27.029816+0000","last_clean":"2024-03-22T13:12:27.029816+0000","last_became_active":"2024-03-22T12:56:30.306295+0000","last_became_peered":"2024-03-22T12:56:30.306295+0000","last_unstale":"2024-03-22T13:12:27.029816+0000","last_undegraded":"2024-03-22T13:12:27.029816+0000","last_fullsized":"2024-03-22T13:12:27.029816+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:27.029754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:27.029754+0000","last_clean_scrub_stamp":"2024-03-22T13:12:27.029754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:06:17.897568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:12.731100+0000","last_change":"2024-03-22T13:12:12.731119+0000","last_active":"2024-03-22T13:12:12.731100+0000","last_peered":"2024-03-22T13:12:12.731100+0000","last_clean":"2024-03-22T13:12:12.731100+0000","last_became_active":"2024-03-22T12:56:30.308446+0000","last_became_peered":"2024-03-22T12:56:30.308446+0000","last_unstale":"2024-03-22T13:12:12.731100+0000","last_undegraded":"2024-03-22T13:12:12.731100+0000","last_fullsized":"2024-03-22T13:12:12.731100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:12.731024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:12.731024+0000","last_clean_scrub_stamp":"2024-03-22T13:12:12.731024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:07:33.723276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:26.075255+0000","last_change":"2024-03-22T13:12:26.075275+0000","last_active":"2024-03-22T13:12:26.075255+0000","last_peered":"2024-03-22T13:12:26.075255+0000","last_clean":"2024-03-22T13:12:26.075255+0000","last_became_active":"2024-03-22T12:56:30.300564+0000","last_became_peered":"2024-03-22T12:56:30.300564+0000","last_unstale":"2024-03-22T13:12:26.075255+0000","last_undegraded":"2024-03-22T13:12:26.075255+0000","last_fullsized":"2024-03-22T13:12:26.075255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:26.075194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:26.075194+0000","last_clean_scrub_stamp":"2024-03-22T13:12:26.075194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T01:01:57.147304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:30.484003+0000","last_change":"2024-03-22T13:12:30.484044+0000","last_active":"2024-03-22T13:12:30.484003+0000","last_peered":"2024-03-22T13:12:30.484003+0000","last_clean":"2024-03-22T13:12:30.484003+0000","last_became_active":"2024-03-22T12:56:30.321118+0000","last_became_peered":"2024-03-22T12:56:30.321118+0000","last_unstale":"2024-03-22T13:12:30.484003+0000","last_undegraded":"2024-03-22T13:12:30.484003+0000","last_fullsized":"2024-03-22T13:12:30.484003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:30.483898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:30.483898+0000","last_clean_scrub_stamp":"2024-03-22T13:12:30.483898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T20:29:57.669264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:33.027619+0000","last_change":"2024-03-22T13:12:33.027679+0000","last_active":"2024-03-22T13:12:33.027619+0000","last_peered":"2024-03-22T13:12:33.027619+0000","last_clean":"2024-03-22T13:12:33.027619+0000","last_became_active":"2024-03-22T12:56:30.318165+0000","last_became_peered":"2024-03-22T12:56:30.318165+0000","last_unstale":"2024-03-22T13:12:33.027619+0000","last_undegraded":"2024-03-22T13:12:33.027619+0000","last_fullsized":"2024-03-22T13:12:33.027619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:33.027516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:33.027516+0000","last_clean_scrub_stamp":"2024-03-22T13:12:33.027516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T18:46:26.274065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:25.531140+0000","last_change":"2024-03-22T13:12:25.531140+0000","last_active":"2024-03-22T13:12:25.531140+0000","last_peered":"2024-03-22T13:12:25.531140+0000","last_clean":"2024-03-22T13:12:25.531140+0000","last_became_active":"2024-03-22T12:56:30.314797+0000","last_became_peered":"2024-03-22T12:56:30.314797+0000","last_unstale":"2024-03-22T13:12:25.531140+0000","last_undegraded":"2024-03-22T13:12:25.531140+0000","last_fullsized":"2024-03-22T13:12:25.531140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:25.530991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:25.530991+0000","last_clean_scrub_stamp":"2024-03-22T13:12:25.530991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:02:54.704524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:19.371983+0000","last_change":"2024-03-22T13:12:19.372018+0000","last_active":"2024-03-22T13:12:19.371983+0000","last_peered":"2024-03-22T13:12:19.371983+0000","last_clean":"2024-03-22T13:12:19.371983+0000","last_became_active":"2024-03-22T12:56:30.314577+0000","last_became_peered":"2024-03-22T12:56:30.314577+0000","last_unstale":"2024-03-22T13:12:19.371983+0000","last_undegraded":"2024-03-22T13:12:19.371983+0000","last_fullsized":"2024-03-22T13:12:19.371983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:19.371838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:19.371838+0000","last_clean_scrub_stamp":"2024-03-22T13:12:19.371838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T19:29:28.641723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:10.884935+0000","last_change":"2024-03-22T13:12:10.884972+0000","last_active":"2024-03-22T13:12:10.884935+0000","last_peered":"2024-03-22T13:12:10.884935+0000","last_clean":"2024-03-22T13:12:10.884935+0000","last_became_active":"2024-03-22T12:56:30.302852+0000","last_became_peered":"2024-03-22T12:56:30.302852+0000","last_unstale":"2024-03-22T13:12:10.884935+0000","last_undegraded":"2024-03-22T13:12:10.884935+0000","last_fullsized":"2024-03-22T13:12:10.884935+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:10.884815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:10.884815+0000","last_clean_scrub_stamp":"2024-03-22T13:12:10.884815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T00:52:57.789842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-22T13:12:25.111550+0000","last_change":"2024-03-22T13:12:25.111587+0000","last_active":"2024-03-22T13:12:25.111550+0000","last_peered":"2024-03-22T13:12:25.111550+0000","last_clean":"2024-03-22T13:12:25.111550+0000","last_became_active":"2024-03-22T12:56:30.232335+0000","last_became_peered":"2024-03-22T12:56:30.232335+0000","last_unstale":"2024-03-22T13:12:25.111550+0000","last_undegraded":"2024-03-22T13:12:25.111550+0000","last_fullsized":"2024-03-22T13:12:25.111550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T13:12:25.111419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T13:12:25.111419+0000","last_clean_scrub_stamp":"2024-03-22T13:12:25.111419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T22:23:22.662867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509640,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":380928,"data_stored":142876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509640,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":356352,"data_stored":140274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509640,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":143610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509640,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":360448,"data_stored":140476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509639,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":425984,"data_stored":730186,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542345,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":434176,"data_stored":731412,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542345,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":360448,"data_stored":143576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542344,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":140406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T13:12:38.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-22T13:12:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.007+0000 7f684e800640 1 Processor -- start 2024-03-22T13:12:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.007+0000 7f684e800640 1 -- start start 2024-03-22T13:12:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.009+0000 7f684e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6848151dd0 0x7f6848157de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.009+0000 7f684e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c7920 0x7f68480cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.009+0000 7f684e800640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6848158620 con 0x7f68480c74e0 2024-03-22T13:12:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.009+0000 7f684e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68480c3620 con 0x7f6848151dd0 2024-03-22T13:12:39.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.009+0000 7f684e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68480c7fb0 con 0x7f68480c7920 2024-03-22T13:12:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.010+0000 7f684de00640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c7920 0x7f68480cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.010+0000 7f684ca00640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f68480c74e0 0x7f68480c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.6:54232/0 (socket says 172.21.15.6:54232) 2024-03-22T13:12:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.010+0000 7f684ca00640 1 -- 172.21.15.6:0/1904418069 learned_addr learned my addr 172.21.15.6:0/1904418069 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-22T13:12:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.010+0000 7f684d400640 1 --2- 172.21.15.6:0/1904418069 >> v2:172.21.15.6:3300/0 conn(0x7f6848151dd0 0x7f6848157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:39.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.013+0000 7f684de00640 1 -- 172.21.15.6:0/1904418069 >> v2:172.21.15.6:3300/0 conn(0x7f6848151dd0 msgr2=0x7f6848157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T13:12:39.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.013+0000 7f684de00640 1 --2- 172.21.15.6:0/1904418069 >> v2:172.21.15.6:3300/0 conn(0x7f6848151dd0 0x7f6848157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.013+0000 7f684de00640 1 -- 172.21.15.6:0/1904418069 >> v1:172.21.15.92:6789/0 conn(0x7f68480c74e0 legacy=0x7f68480c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:39.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.013+0000 7f6847e00640 1 -- 172.21.15.6:0/1904418069 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706836919 0 0) 0x7f6848158620 con 0x7f68480c74e0 2024-03-22T13:12:39.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.013+0000 7f684de00640 1 -- 172.21.15.6:0/1904418069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6848158900 con 0x7f68480c7920 2024-03-22T13:12:39.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.016+0000 7f684de00640 1 --2- 172.21.15.6:0/1904418069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c7920 0x7f68480cb8c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6840009d90 tx=0x7f684002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc690b8db74ad618 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.022+0000 7f6847e00640 1 -- 172.21.15.6:0/1904418069 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840039070 con 0x7f68480c7920 2024-03-22T13:12:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.022+0000 7f6847e00640 1 -- 172.21.15.6:0/1904418069 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684002cbd0 con 0x7f68480c7920 2024-03-22T13:12:39.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.022+0000 7f6847e00640 1 -- 172.21.15.6:0/1904418069 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840034440 con 0x7f68480c7920 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 -- 172.21.15.6:0/1904418069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c7920 msgr2=0x7f68480cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 --2- 172.21.15.6:0/1904418069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c7920 0x7f68480cb8c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6840009d90 tx=0x7f684002be20 comp rx=0 tx=0).stop 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 -- 172.21.15.6:0/1904418069 shutdown_connections 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 --2- 172.21.15.6:0/1904418069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c7920 0x7f68480cb8c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 --2- 172.21.15.6:0/1904418069 >> v2:172.21.15.6:3300/0 conn(0x7f6848151dd0 0x7f6848157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 -- 172.21.15.6:0/1904418069 >> 172.21.15.6:0/1904418069 conn(0x7f684806f0e0 msgr2=0x7f684806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 -- 172.21.15.6:0/1904418069 shutdown_connections 2024-03-22T13:12:39.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.023+0000 7f684e800640 1 -- 172.21.15.6:0/1904418069 wait complete. 2024-03-22T13:12:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.026+0000 7f684e800640 1 Processor -- start 2024-03-22T13:12:39.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.026+0000 7f684e800640 1 -- start start 2024-03-22T13:12:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.027+0000 7f684e800640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c74e0 0x7f684815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.027+0000 7f684e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68480c7920 0x7f684814c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.027+0000 7f684e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6848151dd0 0x7f6848149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.027+0000 7f684e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68480c84e0 con 0x7f6848151dd0 2024-03-22T13:12:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.027+0000 7f684e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68480c8200 con 0x7f68480c7920 2024-03-22T13:12:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.027+0000 7f684e800640 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6848158320 con 0x7f68480c74e0 2024-03-22T13:12:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.028+0000 7f684d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c74e0 0x7f684815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.028+0000 7f684ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68480c7920 0x7f684814c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.028+0000 7f684d400640 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c74e0 0x7f684815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.6:40802/0 (socket says 172.21.15.6:40802) 2024-03-22T13:12:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.028+0000 7f684d400640 1 -- 172.21.15.6:0/760149202 learned_addr learned my addr 172.21.15.6:0/760149202 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-22T13:12:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.029+0000 7f684de00640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6848151dd0 0x7f6848149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:39.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.033+0000 7f684d400640 1 -- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68480c7920 msgr2=0x7f684814c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:39.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.033+0000 7f684d400640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68480c7920 0x7f684814c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.033+0000 7f684d400640 1 -- 172.21.15.6:0/760149202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6848151dd0 msgr2=0x7f6848149300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:39.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.033+0000 7f684d400640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6848151dd0 0x7f6848149300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.033+0000 7f684d400640 1 -- 172.21.15.6:0/760149202 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68481effa0 con 0x7f68480c74e0 2024-03-22T13:12:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.037+0000 7f684d400640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c74e0 0x7f684815b980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f683c00e9b0 tx=0x7f683c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.042+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c017070 con 0x7f68480c74e0 2024-03-22T13:12:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.042+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c00c9a0 con 0x7f68480c74e0 2024-03-22T13:12:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.043+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c00cd90 con 0x7f68480c74e0 2024-03-22T13:12:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.043+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68481f0260 con 0x7f68480c74e0 2024-03-22T13:12:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.043+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68481f0820 con 0x7f68480c74e0 2024-03-22T13:12:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.044+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68480c68b0 con 0x7f68480c74e0 2024-03-22T13:12:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.080+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f683c01a070 con 0x7f68480c74e0 2024-03-22T13:12:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.082+0000 7f6845600640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f682806bc20 0x7f682806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T13:12:39.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.082+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6879+0+0 (secure 0 0 0) 0x7f683c08b620 con 0x7f68480c74e0 2024-03-22T13:12:39.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.083+0000 7f684ca00640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f682806bc20 0x7f682806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T13:12:39.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.084+0000 7f684ca00640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f682806bc20 0x7f682806e0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f68480cc970 tx=0x7f6834007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T13:12:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.398+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f68480c68b0 con 0x7f68480c74e0 2024-03-22T13:12:39.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.631+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f6848075650 con 0x7f68480c74e0 2024-03-22T13:12:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.871+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f683c059680 con 0x7f68480c74e0 2024-03-22T13:12:39.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.871+0000 7f6845600640 1 -- 172.21.15.6:0/760149202 <== mon.2 v2:172.21.15.92:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7f683c05d720 con 0x7f68480c74e0 2024-03-22T13:12:39.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f682806bc20 msgr2=0x7f682806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:39.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f682806bc20 0x7f682806e0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f68480cc970 tx=0x7f6834007de0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c74e0 msgr2=0x7f684815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c74e0 0x7f684815b980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f683c00e9b0 tx=0x7f683c00ee70 comp rx=0 tx=0).stop 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 shutdown_connections 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6848151dd0 0x7f6848149300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:6800/2032190056,v1:172.21.15.92:6801/2032190056] conn(0x7f682806bc20 0x7f682806e0e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68480c7920 0x7f684814c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 --2- 172.21.15.6:0/760149202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68480c74e0 0x7f684815b980 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.875+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 >> 172.21.15.6:0/760149202 conn(0x7f684806f0e0 msgr2=0x7f684814b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.876+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 shutdown_connections 2024-03-22T13:12:39.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-22T13:12:39.876+0000 7f684e800640 1 -- 172.21.15.6:0/760149202 wait complete. 2024-03-22T13:12:39.900 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-22T13:12:39.902 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-22T13:12:39.902 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-22T13:12:39.902 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:12:53.505 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:17:11.840 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.576 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:11.966 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.577 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:11.968 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.609 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:17:11.943 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.668 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:12.058 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.706 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:17:12.096 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.708 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:17:12.098 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.754 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:17:12.088 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:12:53.869 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:17:12.259 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.108 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:17:20.442 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.118 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:17:20.452 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.149 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:17:20.483 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.199 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:17:20.588 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.202 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:17:20.592 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.209 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:20.599 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.210 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:20.600 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.225 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:17:20.615 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:02.226 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:20.615 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:13:04.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:13:04.012 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:13:34.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:13:34.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:13:35.372 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T13:13:35.337+0000 b5f1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a (PID: 40631) UID: 0 2024-03-22T13:13:35.372 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T13:13:35.341+0000 b5f1640 -1 mds.a *** got signal Terminated *** 2024-03-22T13:13:39.445 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T13:13:39.442+0000 cdf4640 -1 mgrc handle_mgr_configure dropping unexpected configure message 2024-03-22T13:14:04.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:14:04.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:14:34.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:14:34.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:15:03.545 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:19:21.879 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.661 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:22.051 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.666 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:22.056 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.671 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:19:22.061 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.675 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:19:22.010 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.676 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:19:22.066 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.718 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:22.108 37456== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.759 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:19:22.149 37457== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:03.820 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:19:22.154 36698== Warning: unimplemented fcntl command: 1036 2024-03-22T13:15:04.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:15:04.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:15:34.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:15:34.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:16:05.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:16:05.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:16:13.257 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T13:16:00.819+0000 a5ef640 -1 monclient: get_auth_request but no auth handler is set up 2024-03-22T13:16:13.692 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T13:16:13.689+0000 95ed640 -1 monclient: get_auth_request but no auth handler is set up 2024-03-22T13:16:35.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:16:35.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:17:05.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:17:05.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:17:35.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:17:35.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:17:38.925 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-22T13:17:38.924+0000 95ed640 -1 monclient: get_auth_request but no auth handler is set up 2024-03-22T13:17:38.980 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-22T13:17:38.980 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-22T13:17:38.980 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:17:38.981 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-22T13:17:38.981+0000 7f7b5da00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 0 (PID: 36968) UID: 0 2024-03-22T13:17:38.981 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-22T13:17:38.982+0000 7f7b5da00640 -1 osd.0 31 *** Got signal Terminated *** 2024-03-22T13:17:45.081 INFO:tasks.ceph.osd.0:Stopped 2024-03-22T13:17:45.082 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-22T13:17:45.082 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:17:45.083 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-22T13:17:45.083+0000 7f4dda000640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 1 (PID: 36970) UID: 0 2024-03-22T13:17:45.083 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-22T13:17:45.083+0000 7f4dda000640 -1 osd.1 33 *** Got signal Terminated *** 2024-03-22T13:17:49.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-22T13:17:51.184 INFO:tasks.ceph.osd.1:Stopped 2024-03-22T13:17:51.184 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-22T13:17:51.184 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:17:51.185 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-22T13:17:51.185+0000 7f5d56400640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 2 (PID: 36974) UID: 0 2024-03-22T13:17:51.185 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-22T13:17:51.185+0000 7f5d56400640 -1 osd.2 35 *** Got signal Terminated *** 2024-03-22T13:17:55.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-22T13:17:55.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-22T13:17:57.286 INFO:tasks.ceph.osd.2:Stopped 2024-03-22T13:17:57.286 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-22T13:17:57.286 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:17:57.288 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-22T13:17:57.288+0000 7f6b11200640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 3 (PID: 36976) UID: 0 2024-03-22T13:17:57.288 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-22T13:17:57.288+0000 7f6b11200640 -1 osd.3 37 *** Got signal Terminated *** 2024-03-22T13:18:02.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-22T13:18:02.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-22T13:18:03.388 INFO:tasks.ceph.osd.3:Stopped 2024-03-22T13:18:03.389 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-22T13:18:03.389 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:18:03.390 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-22T13:18:03.389+0000 7f0033e21640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 4 (PID: 38127) UID: 0 2024-03-22T13:18:03.390 INFO:tasks.ceph.osd.4.smithi092.stderr:2024-03-22T13:18:03.389+0000 7f0033e21640 -1 osd.4 39 *** Got signal Terminated *** 2024-03-22T13:18:05.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:18:05.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:18:08.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-22T13:18:08.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-22T13:18:09.491 INFO:tasks.ceph.osd.4:Stopped 2024-03-22T13:18:09.491 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-22T13:18:09.491 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:18:09.492 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-22T13:18:09.492+0000 7fa2bb4ca640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 5 (PID: 38129) UID: 0 2024-03-22T13:18:09.492 INFO:tasks.ceph.osd.5.smithi092.stderr:2024-03-22T13:18:09.492+0000 7fa2bb4ca640 -1 osd.5 41 *** Got signal Terminated *** 2024-03-22T13:18:14.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-22T13:18:14.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-03-22T13:18:15.592 INFO:tasks.ceph.osd.5:Stopped 2024-03-22T13:18:15.593 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-22T13:18:15.593 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:18:15.594 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-22T13:18:15.594+0000 7f04f6e3c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 6 (PID: 38130) UID: 0 2024-03-22T13:18:15.594 INFO:tasks.ceph.osd.6.smithi092.stderr:2024-03-22T13:18:15.594+0000 7f04f6e3c640 -1 osd.6 43 *** Got signal Terminated *** 2024-03-22T13:18:21.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-22T13:18:21.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-22T13:18:21.695 INFO:tasks.ceph.osd.6:Stopped 2024-03-22T13:18:21.695 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-22T13:18:21.695 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:18:21.696 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-22T13:18:21.697+0000 7fb27cc9d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 7 (PID: 38133) UID: 0 2024-03-22T13:18:21.696 INFO:tasks.ceph.osd.7.smithi092.stderr:2024-03-22T13:18:21.697+0000 7fb27cc9d640 -1 osd.7 43 *** Got signal Terminated *** 2024-03-22T13:18:27.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-22T13:18:27.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-22T13:18:27.797 INFO:tasks.ceph.osd.7:Stopped 2024-03-22T13:18:27.797 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-22T13:18:27.798 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-22T13:18:27.798 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:18:33.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-03-22T13:18:33.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-22T13:18:33.899 INFO:tasks.ceph.mgr.y:Stopped 2024-03-22T13:18:33.900 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-22T13:18:33.900 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:18:33.901 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T13:18:33.900+0000 7f0ef9b09640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 37454) UID: 0 2024-03-22T13:18:33.901 INFO:tasks.ceph.mgr.x.smithi092.stderr:2024-03-22T13:18:33.900+0000 7f0ef9b09640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-22T13:18:33.937 INFO:tasks.ceph.mgr.x:Stopped 2024-03-22T13:18:33.938 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-22T13:18:33.938 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-22T13:18:33.938 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:18:33.986 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-22T13:18:33.968+0000 9f9a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 36691) UID: 0 2024-03-22T13:18:33.987 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-22T13:18:33.970+0000 9f9a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-22T13:18:35.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:18:35.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:18:39.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-03-22T13:19:05.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:19:05.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:19:22.983 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:23:41.317 36698== 2024-03-22T13:19:22.983 INFO:tasks.ceph.mon.a.smithi006.stderr:==00:00:23:41.317 36698== Exit program on first error (--exit-on-first-error=yes) 2024-03-22T13:19:23.030 INFO:tasks.ceph.mon.a.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-22T13:19:26.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-22T13:19:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-22T13:19:28.852 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi006 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-22T13:19:28.853 INFO:tasks.ceph.mon.a:Stopped 2024-03-22T13:19:28.854 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-22T13:19:28.854 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:19:28.870 INFO:tasks.ceph.mon.b.smithi092.stderr:2024-03-22T13:19:28.859+0000 9f9a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 37449) UID: 0 2024-03-22T13:19:28.870 INFO:tasks.ceph.mon.b.smithi092.stderr:2024-03-22T13:19:28.860+0000 9f9a640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-22T13:19:32.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-22T13:19:35.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:19:35.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:20:06.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:20:06.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:20:36.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:20:36.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:21:06.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:21:06.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:21:36.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:21:36.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:21:40.706 INFO:tasks.ceph.mon.c.smithi092.stderr:2024-03-22T13:21:40.697+0000 d7a0640 -1 mon.c@2(probing) e1 get_health_metrics reporting 2 slow ops, oldest is mdsbeacon(4194/c up:resolve seq=350 v27) 2024-03-22T13:21:45.715 INFO:tasks.ceph.mon.c.smithi092.stderr:2024-03-22T13:21:45.714+0000 d7a0640 -1 mon.c@2(probing) e1 get_health_metrics reporting 3 slow ops, oldest is mdsbeacon(4194/c up:resolve seq=350 v27) 2024-03-22T13:21:50.724 INFO:tasks.ceph.mon.c.smithi092.stderr:2024-03-22T13:21:50.723+0000 d7a0640 -1 mon.c@2(probing) e1 get_health_metrics reporting 4 slow ops, oldest is mdsbeacon(4194/c up:resolve seq=350 v27) 2024-03-22T13:21:55.150 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:13.540 37456== 2024-03-22T13:21:55.150 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:13.540 37456== Exit program on first error (--exit-on-first-error=yes) 2024-03-22T13:21:55.160 INFO:tasks.ceph.mon.b.smithi092.stderr:daemon-helper: command failed with exit status 42 2024-03-22T13:21:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-22T13:21:55.292 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-22T13:21:55.292 INFO:tasks.ceph.mon.b:Stopped 2024-03-22T13:21:55.293 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-22T13:21:55.293 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T13:21:55.307 INFO:tasks.ceph.mon.c.smithi092.stderr:2024-03-22T13:21:55.297+0000 9f9a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 37451) UID: 0 2024-03-22T13:21:55.309 INFO:tasks.ceph.mon.c.smithi092.stderr:2024-03-22T13:21:55.299+0000 9f9a640 -1 mon.c@2(probing) e1 *** Got Signal Terminated *** 2024-03-22T13:22:06.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:06.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:32.681 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:26:51.070 37457== 2024-03-22T13:22:32.681 INFO:tasks.ceph.mon.c.smithi092.stderr:==00:00:26:51.070 37457== Exit program on first error (--exit-on-first-error=yes) 2024-03-22T13:22:32.690 INFO:tasks.ceph.mon.c.smithi092.stderr:daemon-helper: command failed with exit status 42 2024-03-22T13:22:36.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:36.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:37.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-22T13:22:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-22T13:22:38.003 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-22T13:22:38.004 INFO:tasks.ceph.mon.c:Stopped 2024-03-22T13:22:38.004 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-22T13:22:38.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v MON_DOWN | head -n 1 2024-03-22T13:22:38.116 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T13:22:38.116 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-22T13:22:38.387 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T13:22:38.387 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-22T13:22:38.518 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T13:22:38.518 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-22T13:22:38.656 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T13:22:38.656 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-22T13:22:38.813 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T13:22:38.813 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-22T13:22:38.925 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T13:22:38.925 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-22T13:22:38.999 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T13:22:38.999 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-22T13:22:39.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T13:22:39.056 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-22T13:22:39.123 INFO:tasks.ceph:Archiving mon data... 2024-03-22T13:22:39.124 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/data/mon.a.tgz 2024-03-22T13:22:39.124 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-22T13:22:39.174 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.jjYqpXgxTl 2024-03-22T13:22:39.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.jjYqpXgxTl 2024-03-22T13:22:39.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.jjYqpXgxTl 2024-03-22T13:22:39.451 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.jjYqpXgxTl is 316KB 2024-03-22T13:22:39.543 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.jjYqpXgxTl 2024-03-22T13:22:39.589 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/data/mon.b.tgz 2024-03-22T13:22:39.590 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-22T13:22:39.603 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.EbARJc9qug 2024-03-22T13:22:39.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.EbARJc9qug 2024-03-22T13:22:39.748 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.EbARJc9qug 2024-03-22T13:22:39.831 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.EbARJc9qug is 358KB 2024-03-22T13:22:39.896 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.EbARJc9qug 2024-03-22T13:22:39.911 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/data/mon.c.tgz 2024-03-22T13:22:39.912 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-22T13:22:39.967 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.FeyFTGB8Vk 2024-03-22T13:22:39.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.FeyFTGB8Vk 2024-03-22T13:22:40.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.FeyFTGB8Vk 2024-03-22T13:22:40.190 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.FeyFTGB8Vk is 369KB 2024-03-22T13:22:40.254 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.FeyFTGB8Vk 2024-03-22T13:22:40.268 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-22T13:22:40.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-22T13:22:40.275 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-22T13:22:40.488 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-22T13:22:40.534 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-22T13:22:40.581 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-22T13:22:40.657 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-22T13:22:40.728 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-22T13:22:40.745 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-22T13:22:40.791 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-22T13:22:40.829 INFO:tasks.ceph:Archiving crash dumps... 2024-03-22T13:22:40.831 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/remote/smithi006/crash 2024-03-22T13:22:40.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-22T13:22:40.913 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/remote/smithi092/crash 2024-03-22T13:22:40.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-22T13:22:40.940 INFO:tasks.ceph:Shutting down logrotate 2024-03-22T13:22:40.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:40.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:41.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:41.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-22T13:22:41.158 INFO:tasks.ceph:Compressing logs... 2024-03-22T13:22:41.158 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T13:22:41.164 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T13:22:41.228 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-22T13:22:41.228 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2024-03-22T13:22:41.228 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38035.log 2024-03-22T13:22:41.228 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37648.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.38035.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.f.log 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gz 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.b.log 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38035.log.gz 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/mds.f.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr: 55.1% -- replaced with /var/log/ceph/valgrind/mds.f.log.gz 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.d.log 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/mds.b.log: 55.2% -- replaced with /var/log/ceph/valgrind/mds.b.log.gz 2024-03-22T13:22:41.233 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37877.log 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/mds.d.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 55.1% -- replaced with /var/log/ceph/valgrind/mds.d.log.gz 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37746.log 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37877.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37877.log.gz 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37746.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.audit.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.37746.log.gz --verbose 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr: -- /var/log/ceph/ceph-osd.4.log 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose/var/log/ceph/ceph-mds.b.log: -- /var/log/ceph/ceph-client.admin.37931.log 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-22T13:22:41.234 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37931.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37931.log.gz 2024-03-22T13:22:41.236 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-22T13:22:41.236 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.7.log: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-22T13:22:41.236 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54752.log 2024-03-22T13:22:41.238 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48611.log 2024-03-22T13:22:41.238 INFO:teuthology.orchestra.run.smithi092.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-22T13:22:41.238 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-22T13:22:41.238 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37837.log 2024-03-22T13:22:41.239 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48359.log 2024-03-22T13:22:41.239 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54752.log.gz 2024-03-22T13:22:41.242 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47044.log 2024-03-22T13:22:41.242 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48611.log.gz 2024-03-22T13:22:41.242 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52435.log 2024-03-22T13:22:41.242 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48359.log.gz 2024-03-22T13:22:41.243 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-22T13:22:41.243 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37837.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37837.log.gz 2024-03-22T13:22:41.243 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-22T13:22:41.244 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mgr.x.log: gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping 2024-03-22T13:22:41.244 INFO:teuthology.orchestra.run.smithi092.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-22T13:22:41.245 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47650.log 2024-03-22T13:22:41.246 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47044.log.gz 2024-03-22T13:22:41.246 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50784.log 2024-03-22T13:22:41.246 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52435.log.gz 2024-03-22T13:22:41.246 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2024-03-22T13:22:41.246 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47650.log.gz 2024-03-22T13:22:41.246 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50219.log 2024-03-22T13:22:41.246 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50784.log.gz 2024-03-22T13:22:41.247 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53623.log 2024-03-22T13:22:41.247 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2024-03-22T13:22:41.248 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50219.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-22T13:22:41.248 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50219.log.gz 2024-03-22T13:22:41.249 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36980.log 2024-03-22T13:22:41.249 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53623.log.gz 2024-03-22T13:22:41.250 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50826.log 2024-03-22T13:22:41.252 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.36980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49697.log 2024-03-22T13:22:41.252 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36980.log.gz 2024-03-22T13:22:41.253 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49925.log 2024-03-22T13:22:41.253 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50826.log.gz 2024-03-22T13:22:41.254 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47440.log 2024-03-22T13:22:41.255 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-22T13:22:41.256 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49697.log.gz 2024-03-22T13:22:41.256 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42597.log 2024-03-22T13:22:41.256 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49925.log.gz 2024-03-22T13:22:41.257 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46465.log 2024-03-22T13:22:41.257 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47440.log.gz 2024-03-22T13:22:41.258 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56321.log 2024-03-22T13:22:41.259 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42597.log.gz 2024-03-22T13:22:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59265.log 2024-03-22T13:22:41.260 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46465.log.gz 2024-03-22T13:22:41.261 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48485.log 2024-03-22T13:22:41.261 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56321.log.gz 2024-03-22T13:22:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55796.log 2024-03-22T13:22:41.262 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59265.log.gz 2024-03-22T13:22:41.264 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.log: 93.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37688.log 2024-03-22T13:22:41.264 INFO:teuthology.orchestra.run.smithi092.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-22T13:22:41.265 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54482.log 2024-03-22T13:22:41.265 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48485.log.gz 2024-03-22T13:22:41.265 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52309.log 2024-03-22T13:22:41.265 INFO:teuthology.orchestra.run.smithi006.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55796.log.gz 2024-03-22T13:22:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51348.log 2024-03-22T13:22:41.266 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54482.log.gz 2024-03-22T13:22:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59123.log 2024-03-22T13:22:41.267 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52309.log.gz 2024-03-22T13:22:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45378.log 2024-03-22T13:22:41.269 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51348.log.gz 2024-03-22T13:22:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57659.log 2024-03-22T13:22:41.270 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59123.log.gz 2024-03-22T13:22:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54962.log 2024-03-22T13:22:41.272 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45378.log.gz 2024-03-22T13:22:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57659.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57659.log.gz -5 2024-03-22T13:22:41.273 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49529.log 2024-03-22T13:22:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42987.log 2024-03-22T13:22:41.274 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54962.log.gz 2024-03-22T13:22:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50093.log 2024-03-22T13:22:41.275 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49529.log.gz 2024-03-22T13:22:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43598.log 2024-03-22T13:22:41.277 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42987.log.gz 2024-03-22T13:22:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40727.log 2024-03-22T13:22:41.278 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50093.log.gz 2024-03-22T13:22:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48089.log 2024-03-22T13:22:41.279 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43598.log.gz 2024-03-22T13:22:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48173.log 2024-03-22T13:22:41.281 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40727.log.gz 2024-03-22T13:22:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54836.log 2024-03-22T13:22:41.282 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48089.log.gz 2024-03-22T13:22:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41397.log 2024-03-22T13:22:41.283 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.48173.log.gz 2024-03-22T13:22:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52705.log 2024-03-22T13:22:41.285 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54836.log.gz 2024-03-22T13:22:41.287 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-22T13:22:41.287 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.37688.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-22T13:22:41.287 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mds.d.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37688.log.gz 2024-03-22T13:22:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41397.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41397.log.gz -5 2024-03-22T13:22:41.288 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52957.log 2024-03-22T13:22:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44949.log 2024-03-22T13:22:41.289 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52705.log.gz 2024-03-22T13:22:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52957.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52957.log.gz 2024-03-22T13:22:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2024-03-22T13:22:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47272.log 2024-03-22T13:22:41.290 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44949.log.gz 2024-03-22T13:22:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40685.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2024-03-22T13:22:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54146.log 2024-03-22T13:22:41.292 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55358.log 2024-03-22T13:22:41.292 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47272.log.gz 2024-03-22T13:22:41.293 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54146.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54146.log.gz 2024-03-22T13:22:41.293 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48653.log 2024-03-22T13:22:41.294 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37786.log 2024-03-22T13:22:41.295 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57929.log 2024-03-22T13:22:41.295 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55358.log.gz 2024-03-22T13:22:41.296 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48653.log: gzip -5 --verbose 87.4% -- -- replaced with /var/log/ceph/ceph-client.admin.48653.log.gz /var/log/ceph/ceph-client.admin.54272.log 2024-03-22T13:22:41.296 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43844.log 2024-03-22T13:22:41.297 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57929.log.gz 2024-03-22T13:22:41.298 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54272.log: gzip -5 --verbose 89.0% -- /var/log/ceph/ceph-client.admin.49445.log -- replaced with /var/log/ceph/ceph-client.admin.54272.log.gz 2024-03-22T13:22:41.298 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59225.log 2024-03-22T13:22:41.299 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43844.log.gz 2024-03-22T13:22:41.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49445.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49445.log.gz 2024-03-22T13:22:41.301 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52789.log 2024-03-22T13:22:41.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59225.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.c.log 2024-03-22T13:22:41.302 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59225.log.gz 2024-03-22T13:22:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52789.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52789.log.gz 2024-03-22T13:22:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.a.log 2024-03-22T13:22:41.304 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.37786.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37786.log.gz 2024-03-22T13:22:41.304 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/mds.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-22T13:22:41.305 INFO:teuthology.orchestra.run.smithi006.stderr: 55.3% -- replaced with /var/log/ceph/valgrind/mds.c.log.gz 2024-03-22T13:22:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/mds.a.log: 55.3% -- replaced with /var/log/ceph/valgrind/mds.a.log.gz 2024-03-22T13:22:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.e.log 2024-03-22T13:22:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50868.log 2024-03-22T13:22:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/mon.a.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-22T13:22:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/valgrind/mds.e.log: 55.3% -- replaced with /var/log/ceph/valgrind/mds.e.log.gz 2024-03-22T13:22:41.308 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50261.log 2024-03-22T13:22:41.308 INFO:teuthology.orchestra.run.smithi092.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-22T13:22:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.50868.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44233.log 2024-03-22T13:22:41.309 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50868.log.gz 2024-03-22T13:22:41.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50261.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50261.log.gz 2024-03-22T13:22:41.310 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57617.log 2024-03-22T13:22:41.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51264.log 2024-03-22T13:22:41.312 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44233.log.gz 2024-03-22T13:22:41.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57617.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57617.log.gz 2024-03-22T13:22:41.313 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53437.log 2024-03-22T13:22:41.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52225.log 2024-03-22T13:22:41.314 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51264.log.gz 2024-03-22T13:22:41.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53437.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53437.log.gz 2024-03-22T13:22:41.315 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45252.log 2024-03-22T13:22:41.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46111.log 2024-03-22T13:22:41.316 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52225.log.gz 2024-03-22T13:22:41.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45252.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45252.log.gz 2024-03-22T13:22:41.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43029.log 2024-03-22T13:22:41.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54188.log 2024-03-22T13:22:41.319 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46111.log.gz 2024-03-22T13:22:41.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57575.log 2024-03-22T13:22:41.320 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43029.log.gz 2024-03-22T13:22:41.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45649.log 2024-03-22T13:22:41.321 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54188.log.gz 2024-03-22T13:22:41.322 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57575.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57575.log.gz 2024-03-22T13:22:41.322 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53227.log 2024-03-22T13:22:41.323 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51744.log 2024-03-22T13:22:41.324 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45649.log: /var/log/ceph/ceph-client.admin.53227.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45649.log.gz 2024-03-22T13:22:41.324 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53227.log.gz 2024-03-22T13:22:41.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58955.log 2024-03-22T13:22:41.325 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57011.log 2024-03-22T13:22:41.326 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51744.log.gz 2024-03-22T13:22:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58955.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58955.log.gz 2024-03-22T13:22:41.327 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48275.log 2024-03-22T13:22:41.328 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49781.log 2024-03-22T13:22:41.329 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57011.log.gz 2024-03-22T13:22:41.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48275.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48275.log.gz 2024-03-22T13:22:41.329 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53479.log 2024-03-22T13:22:41.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50574.log 2024-03-22T13:22:41.331 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49781.log.gz 2024-03-22T13:22:41.332 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53479.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53479.log.gz 2024-03-22T13:22:41.332 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50742.log 2024-03-22T13:22:41.333 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37099.log 2024-03-22T13:22:41.333 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50574.log.gz 2024-03-22T13:22:41.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50742.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50742.log.gz 2024-03-22T13:22:41.334 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54626.log 2024-03-22T13:22:41.335 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41156.log 2024-03-22T13:22:41.335 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37099.log.gz 2024-03-22T13:22:41.336 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54626.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54626.log.gz 2024-03-22T13:22:41.337 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45943.log 2024-03-22T13:22:41.337 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59081.log 2024-03-22T13:22:41.338 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41156.log.gz 2024-03-22T13:22:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45943.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45943.log.gz 2024-03-22T13:22:41.339 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44991.log 2024-03-22T13:22:41.340 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55922.log 2024-03-22T13:22:41.341 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44991.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59081.log.gz 2024-03-22T13:22:41.341 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55046.log 2024-03-22T13:22:41.341 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44991.log.gz 2024-03-22T13:22:41.343 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55922.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55922.log.gz 2024-03-22T13:22:41.343 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58223.log 2024-03-22T13:22:41.344 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58703.log 2024-03-22T13:22:41.344 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55046.log.gz 2024-03-22T13:22:41.345 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56489.log 89.0% 2024-03-22T13:22:41.345 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58223.log.gz 2024-03-22T13:22:41.347 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49403.log 2024-03-22T13:22:41.347 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58703.log.gz 2024-03-22T13:22:41.348 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55652.log 2024-03-22T13:22:41.348 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56489.log.gz 2024-03-22T13:22:41.349 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2024-03-22T13:22:41.350 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49403.log.gz 2024-03-22T13:22:41.350 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49655.log 2024-03-22T13:22:41.351 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55652.log.gz 2024-03-22T13:22:41.352 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46255.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2024-03-22T13:22:41.352 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43293.log 2024-03-22T13:22:41.353 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58013.log 2024-03-22T13:22:41.353 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49655.log.gz 2024-03-22T13:22:41.354 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51450.log 2024-03-22T13:22:41.355 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43293.log.gz 2024-03-22T13:22:41.355 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58013.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58013.log.gz 2024-03-22T13:22:41.356 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55838.log 2024-03-22T13:22:41.357 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55610.log 2024-03-22T13:22:41.357 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51450.log: 89.1%/var/log/ceph/ceph-client.admin.55838.log: -- replaced with /var/log/ceph/ceph-client.admin.51450.log.gz 2024-03-22T13:22:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53917.log 2024-03-22T13:22:41.358 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55838.log.gz 2024-03-22T13:22:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55610.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55610.log.gz 2024-03-22T13:22:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54710.log 2024-03-22T13:22:41.360 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49133.log 2024-03-22T13:22:41.361 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54710.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53917.log.gz 2024-03-22T13:22:41.362 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41673.log 2024-03-22T13:22:41.362 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54710.log.gz 2024-03-22T13:22:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53833.log 2024-03-22T13:22:41.363 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49133.log.gz 2024-03-22T13:22:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47398.log 2024-03-22T13:22:41.364 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41673.log.gz 2024-03-22T13:22:41.365 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2024-03-22T13:22:41.366 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47398.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53833.log.gz 2024-03-22T13:22:41.367 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47398.log.gz 2024-03-22T13:22:41.367 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51492.log 2024-03-22T13:22:41.368 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-03-22T13:22:41.368 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz 2024-03-22T13:22:41.370 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51492.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51492.log.gz 2024-03-22T13:22:41.370 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47693.log 2024-03-22T13:22:41.370 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53353.log 2024-03-22T13:22:41.371 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2%/var/log/ceph/ceph-client.admin.47693.log: -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-03-22T13:22:41.372 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47693.log.gz 2024-03-22T13:22:41.372 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54001.log 2024-03-22T13:22:41.373 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49175.log 2024-03-22T13:22:41.374 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53353.log.gz 2024-03-22T13:22:41.374 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59165.log 2024-03-22T13:22:41.375 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54001.log.gz 2024-03-22T13:22:41.376 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42371.log 2024-03-22T13:22:41.376 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49175.log.gz 2024-03-22T13:22:41.377 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57491.log 2024-03-22T13:22:41.377 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59165.log.gz 2024-03-22T13:22:41.378 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48737.log 2024-03-22T13:22:41.378 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42371.log.gz 2024-03-22T13:22:41.379 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57491.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57491.log.gz 2024-03-22T13:22:41.379 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42903.log 2024-03-22T13:22:41.380 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57365.log 2024-03-22T13:22:41.381 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48737.log.gz 2024-03-22T13:22:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42903.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42903.log.gz 2024-03-22T13:22:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49007.log 2024-03-22T13:22:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55880.log 2024-03-22T13:22:41.383 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57365.log.gz 2024-03-22T13:22:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40786.log 2024-03-22T13:22:41.384 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49007.log.gz 2024-03-22T13:22:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55880.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55880.log.gz 2024-03-22T13:22:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51660.log 2024-03-22T13:22:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44823.log 2024-03-22T13:22:41.387 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40786.log.gz 2024-03-22T13:22:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50009.log 2024-03-22T13:22:41.388 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51660.log.gz 2024-03-22T13:22:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55088.log 2024-03-22T13:22:41.389 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44823.log.gz 2024-03-22T13:22:41.390 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51870.log 2024-03-22T13:22:41.391 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50009.log.gz 2024-03-22T13:22:41.392 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44323.log 2024-03-22T13:22:41.392 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55088.log.gz 2024-03-22T13:22:41.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55004.log 2024-03-22T13:22:41.393 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51870.log.gz 2024-03-22T13:22:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41217.log 2024-03-22T13:22:41.395 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44323.log.gz 2024-03-22T13:22:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43987.log 2024-03-22T13:22:41.396 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55004.log.gz 2024-03-22T13:22:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42497.log 2024-03-22T13:22:41.397 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41217.log.gz 2024-03-22T13:22:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43987.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43987.log.gz 2024-03-22T13:22:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46297.log 2024-03-22T13:22:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45985.log 2024-03-22T13:22:41.400 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42497.log.gz 2024-03-22T13:22:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53959.log 2024-03-22T13:22:41.401 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46297.log.gz 2024-03-22T13:22:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45985.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45985.log.gz 2024-03-22T13:22:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45901.log 2024-03-22T13:22:41.403 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54566.log 2024-03-22T13:22:41.404 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45901.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53959.log.gz 2024-03-22T13:22:41.404 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45901.log.gz 2024-03-22T13:22:41.404 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2024-03-22T13:22:41.405 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54440.log 2024-03-22T13:22:41.406 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54566.log.gz 2024-03-22T13:22:41.407 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47921.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2024-03-22T13:22:41.407 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49883.log 2024-03-22T13:22:41.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53581.log 2024-03-22T13:22:41.408 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54440.log.gz 2024-03-22T13:22:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58055.log 2024-03-22T13:22:41.409 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49883.log.gz 2024-03-22T13:22:41.410 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53581.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53581.log.gz 2024-03-22T13:22:41.411 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57533.log 2024-03-22T13:22:41.411 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2024-03-22T13:22:41.412 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58055.log.gz 2024-03-22T13:22:41.413 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57533.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57533.log.gz 2024-03-22T13:22:41.413 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42539.log 2024-03-22T13:22:41.414 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52014.log 2024-03-22T13:22:41.415 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2024-03-22T13:22:41.416 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42539.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-22T13:22:41.416 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42539.log.gz 2024-03-22T13:22:41.417 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46960.log 2024-03-22T13:22:41.417 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52014.log.gz 2024-03-22T13:22:41.418 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43335.log 2024-03-22T13:22:41.419 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51306.log 2024-03-22T13:22:41.420 INFO:teuthology.orchestra.run.smithi006.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.46960.log.gz 2024-03-22T13:22:41.420 INFO:teuthology.orchestra.run.smithi006.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-22T13:22:41.420 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50051.log 2024-03-22T13:22:41.421 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43335.log.gz 2024-03-22T13:22:41.422 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56048.log 2024-03-22T13:22:41.422 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51306.log.gz 2024-03-22T13:22:41.423 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50051.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50051.log.gz 2024-03-22T13:22:41.423 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50531.log 2024-03-22T13:22:41.424 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56048.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56048.log.gz 2024-03-22T13:22:41.425 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49967.log 2024-03-22T13:22:41.426 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56447.log 2024-03-22T13:22:41.426 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50531.log.gz 2024-03-22T13:22:41.427 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49967.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49967.log.gz 2024-03-22T13:22:41.427 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49361.log 2024-03-22T13:22:41.428 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56447.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56447.log.gz 2024-03-22T13:22:41.429 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44865.log 2024-03-22T13:22:41.429 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46381.log 2024-03-22T13:22:41.430 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49361.log.gz 2024-03-22T13:22:41.431 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36697.log 2024-03-22T13:22:41.431 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44865.log.gz 2024-03-22T13:22:41.432 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46381.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46381.log.gz 2024-03-22T13:22:41.432 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52663.log 2024-03-22T13:22:41.433 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.36697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56363.log 2024-03-22T13:22:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52663.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.36697.log.gz 2024-03-22T13:22:41.435 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59039.log 2024-03-22T13:22:41.435 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52663.log.gz 2024-03-22T13:22:41.436 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56363.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56363.log.gz 2024-03-22T13:22:41.436 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55964.log 2024-03-22T13:22:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58829.log 2024-03-22T13:22:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.59039.log: /var/log/ceph/ceph-client.admin.55964.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59039.log.gz 2024-03-22T13:22:41.438 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55964.log.gz 2024-03-22T13:22:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-03-22T13:22:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51390.log 2024-03-22T13:22:41.440 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58829.log.gz 2024-03-22T13:22:41.441 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40946.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-03-22T13:22:41.441 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51786.log 2024-03-22T13:22:41.442 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55232.log 2024-03-22T13:22:41.442 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51390.log: /var/log/ceph/ceph-client.admin.51786.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51390.log.gz 2024-03-22T13:22:41.443 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51786.log.gz 2024-03-22T13:22:41.443 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46750.log 2024-03-22T13:22:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.55232.log: -- /var/log/ceph/ceph-client.admin.51972.log 2024-03-22T13:22:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46750.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55232.log.gz 2024-03-22T13:22:41.445 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46750.log.gz 2024-03-22T13:22:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56573.log 2024-03-22T13:22:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56657.log 2024-03-22T13:22:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51972.log: /var/log/ceph/ceph-client.admin.56573.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51972.log.gz 2024-03-22T13:22:41.447 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56573.log.gz 2024-03-22T13:22:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48923.log 2024-03-22T13:22:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41607.log 2024-03-22T13:22:41.449 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56657.log.gz 2024-03-22T13:22:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48923.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48923.log.gz 2024-03-22T13:22:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51828.log 2024-03-22T13:22:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46792.log 2024-03-22T13:22:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41607.log: /var/log/ceph/ceph-client.admin.51828.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51828.log.gz -5 2024-03-22T13:22:41.452 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose 90.6% -- -- replaced with /var/log/ceph/ceph-client.admin.41607.log.gz /var/log/ceph/ceph-client.admin.44283.log 2024-03-22T13:22:41.452 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54230.log 2024-03-22T13:22:41.453 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46792.log.gz 2024-03-22T13:22:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44283.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44283.log.gz 2024-03-22T13:22:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51096.log 2024-03-22T13:22:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2024-03-22T13:22:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54230.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54230.log.gz/var/log/ceph/ceph-client.admin.51096.log: 2024-03-22T13:22:41.456 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51096.log.gz 2024-03-22T13:22:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41523.log 2024-03-22T13:22:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50658.log 2024-03-22T13:22:41.458 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2024-03-22T13:22:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41523.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41523.log.gz 2024-03-22T13:22:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57323.log 2024-03-22T13:22:41.460 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58451.log 2024-03-22T13:22:41.460 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50658.log: /var/log/ceph/ceph-client.admin.57323.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50658.log.gz 2024-03-22T13:22:41.461 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57323.log.gz 2024-03-22T13:22:41.461 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43165.log 2024-03-22T13:22:41.462 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48401.log 2024-03-22T13:22:41.462 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58451.log.gz 2024-03-22T13:22:41.463 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43165.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43165.log.gz 2024-03-22T13:22:41.463 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46549.log 2024-03-22T13:22:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46027.log 2024-03-22T13:22:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48401.log: /var/log/ceph/ceph-client.admin.46549.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48401.log.gz 2024-03-22T13:22:41.465 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46549.log.gz 2024-03-22T13:22:41.466 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44739.log 2024-03-22T13:22:41.466 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51180.log 2024-03-22T13:22:41.467 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46027.log.gz 2024-03-22T13:22:41.467 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44739.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44739.log.gz 2024-03-22T13:22:41.468 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-03-22T13:22:41.469 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57053.log 2024-03-22T13:22:41.469 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51180.log: /var/log/ceph/ceph-client.admin.47795.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51180.log.gz 2024-03-22T13:22:41.470 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-03-22T13:22:41.470 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42413.log 2024-03-22T13:22:41.471 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41267.log 2024-03-22T13:22:41.472 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57053.log.gz 2024-03-22T13:22:41.472 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42413.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42413.log.gz 2024-03-22T13:22:41.472 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56615.log 2024-03-22T13:22:41.473 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48965.log 2024-03-22T13:22:41.474 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56615.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41267.log.gz 2024-03-22T13:22:41.475 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42681.log 2024-03-22T13:22:41.475 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56615.log.gz 2024-03-22T13:22:41.476 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48965.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48965.log.gz 2024-03-22T13:22:41.476 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-03-22T13:22:41.477 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51222.log 2024-03-22T13:22:41.477 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42681.log: /var/log/ceph/ceph-client.admin.41030.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42681.log.gz 2024-03-22T13:22:41.478 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-03-22T13:22:41.478 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50616.log 2024-03-22T13:22:41.479 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51222.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51222.log.gz 2024-03-22T13:22:41.480 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56801.log 2024-03-22T13:22:41.481 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50970.log 2024-03-22T13:22:41.481 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56801.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50616.log.gz 2024-03-22T13:22:41.482 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56801.log.gz 2024-03-22T13:22:41.482 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-03-22T13:22:41.483 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50970.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-22T13:22:41.483 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50970.log.gz 2024-03-22T13:22:41.485 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41481.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2024-03-22T13:22:41.485 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47566.log 2024-03-22T13:22:41.486 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52579.log 2024-03-22T13:22:41.487 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47566.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47566.log.gz 2024-03-22T13:22:41.487 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53143.log 2024-03-22T13:22:41.488 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54043.log 2024-03-22T13:22:41.488 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52579.log.gz 2024-03-22T13:22:41.490 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53143.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53143.log.gz 2024-03-22T13:22:41.490 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49301.log 2024-03-22T13:22:41.490 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57137.log 2024-03-22T13:22:41.491 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54043.log: 87.4%/var/log/ceph/ceph-client.admin.49301.log: -- replaced with /var/log/ceph/ceph-client.admin.54043.log.gz 2024-03-22T13:22:41.492 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49301.log.gz 2024-03-22T13:22:41.492 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54878.log 2024-03-22T13:22:41.493 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56699.log 2024-03-22T13:22:41.493 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57137.log.gz 2024-03-22T13:22:41.494 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54878.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54878.log.gz 2024-03-22T13:22:41.494 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45859.log 2024-03-22T13:22:41.495 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53269.log 2024-03-22T13:22:41.496 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56699.log.gz 2024-03-22T13:22:41.497 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44907.log 2024-03-22T13:22:41.497 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45859.log.gz 2024-03-22T13:22:41.498 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53269.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53269.log.gz 2024-03-22T13:22:41.498 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53791.log 2024-03-22T13:22:41.499 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56927.log 2024-03-22T13:22:41.499 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44907.log.gz 2024-03-22T13:22:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53791.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53791.log.gz 2024-03-22T13:22:41.501 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.45504.log 2024-03-22T13:22:41.501 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56927.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56927.log.gz 2024-03-22T13:22:41.502 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46633.log 2024-03-22T13:22:41.503 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43251.log 2024-03-22T13:22:41.504 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45504.log.gz 2024-03-22T13:22:41.504 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46633.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46633.log.gz 2024-03-22T13:22:41.504 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46171.log 2024-03-22T13:22:41.505 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.43251.log: --verbose -- /var/log/ceph/ceph-client.admin.48569.log 2024-03-22T13:22:41.506 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43251.log.gz 2024-03-22T13:22:41.507 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46171.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46171.log.gz 2024-03-22T13:22:41.507 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50345.log 2024-03-22T13:22:41.507 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50405.log 2024-03-22T13:22:41.508 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48569.log.gz 2024-03-22T13:22:41.509 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2024-03-22T13:22:41.509 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50345.log.gz 2024-03-22T13:22:41.510 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55316.log 2024-03-22T13:22:41.510 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50405.log.gz 2024-03-22T13:22:41.511 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52477.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2024-03-22T13:22:41.511 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47356.log 2024-03-22T13:22:41.513 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45775.log 2024-03-22T13:22:41.513 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55316.log.gz 2024-03-22T13:22:41.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41072.log 2024-03-22T13:22:41.514 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47356.log.gz 2024-03-22T13:22:41.515 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45775.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45775.log.gz 2024-03-22T13:22:41.515 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58745.log 2024-03-22T13:22:41.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50700.log 2024-03-22T13:22:41.517 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58745.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41072.log.gz 2024-03-22T13:22:41.518 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42105.log 2024-03-22T13:22:41.518 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58745.log.gz 2024-03-22T13:22:41.519 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50700.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50700.log.gz 2024-03-22T13:22:41.519 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43596.log 2024-03-22T13:22:41.520 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56132.log 2024-03-22T13:22:41.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43596.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56237.log 2024-03-22T13:22:41.521 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43596.log.gz 2024-03-22T13:22:41.523 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48005.log 2024-03-22T13:22:41.523 INFO:teuthology.orchestra.run.smithi006.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.42105.log.gz 2024-03-22T13:22:41.523 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56132.log.gz 2024-03-22T13:22:41.524 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51054.log 2024-03-22T13:22:41.524 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56237.log.gz 2024-03-22T13:22:41.525 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48005.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48005.log.gz 2024-03-22T13:22:41.525 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54668.log 2024-03-22T13:22:41.526 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.51054.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57701.log 2024-03-22T13:22:41.527 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51054.log.gz 2024-03-22T13:22:41.527 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54668.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54668.log.gz 2024-03-22T13:22:41.527 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41565.log 2024-03-22T13:22:41.528 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52831.log 2024-03-22T13:22:41.529 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57701.log.gz 2024-03-22T13:22:41.529 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41565.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41565.log.gz 2024-03-22T13:22:41.529 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52999.log 2024-03-22T13:22:41.530 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.52831.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57845.log 2024-03-22T13:22:41.531 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52831.log.gz 2024-03-22T13:22:41.531 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52999.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52999.log.gz 2024-03-22T13:22:41.532 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49571.log 2024-03-22T13:22:41.532 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58787.log 2024-03-22T13:22:41.533 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57845.log.gz 2024-03-22T13:22:41.534 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44211.log 2024-03-22T13:22:41.534 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49571.log.gz 2024-03-22T13:22:41.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55484.log 2024-03-22T13:22:41.536 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58787.log.gz 2024-03-22T13:22:41.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44211.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44211.log.gz 2024-03-22T13:22:41.537 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49259.log 2024-03-22T13:22:41.537 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51534.log 2024-03-22T13:22:41.538 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55484.log.gz 2024-03-22T13:22:41.539 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52056.log 2024-03-22T13:22:41.540 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49259.log.gz 2024-03-22T13:22:41.540 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51534.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51534.log.gz 2024-03-22T13:22:41.540 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-03-22T13:22:41.541 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56405.log 2024-03-22T13:22:41.542 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52056.log.gz 2024-03-22T13:22:41.543 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41355.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52140.log 2024-03-22T13:22:41.543 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-03-22T13:22:41.544 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43209.log 2024-03-22T13:22:41.544 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56405.log.gz 2024-03-22T13:22:41.545 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52140.log: gzip -5 --verbose 87.3% -- /var/log/ceph/ceph-osd.3.log -- replaced with /var/log/ceph/ceph-client.admin.52140.log.gz 2024-03-22T13:22:41.545 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.546 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52873.log 2024-03-22T13:22:41.547 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43209.log.gz 2024-03-22T13:22:41.548 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43122.log 2024-03-22T13:22:41.549 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45691.log 2024-03-22T13:22:41.549 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52873.log.gz 2024-03-22T13:22:41.550 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57407.log 2024-03-22T13:22:41.550 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43122.log.gz 2024-03-22T13:22:41.552 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56090.log 2024-03-22T13:22:41.552 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45691.log.gz 2024-03-22T13:22:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51576.log 2024-03-22T13:22:41.553 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57407.log.gz 2024-03-22T13:22:41.555 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52098.log 2024-03-22T13:22:41.555 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56090.log.gz 2024-03-22T13:22:41.556 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44469.log 2024-03-22T13:22:41.556 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51576.log.gz 2024-03-22T13:22:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45462.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52098.log.gz 2024-03-22T13:22:41.557 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.558 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44469.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44469.log.gz 2024-03-22T13:22:41.559 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46591.log 2024-03-22T13:22:41.560 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54314.log 2024-03-22T13:22:41.560 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45462.log.gz 2024-03-22T13:22:41.562 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46591.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46591.log.gz 2024-03-22T13:22:41.562 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54314.log: /var/log/ceph/ceph-client.admin.46834.log 2024-03-22T13:22:41.563 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54314.log.gz 2024-03-22T13:22:41.563 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57095.log 2024-03-22T13:22:41.565 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53749.log 2024-03-22T13:22:41.565 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46834.log.gz 2024-03-22T13:22:41.567 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57095.log: gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57095.log.gz 2024-03-22T13:22:41.567 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48779.log 2024-03-22T13:22:41.567 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53749.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53749.log.gz 2024-03-22T13:22:41.568 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43879.log 2024-03-22T13:22:41.570 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58367.log 2024-03-22T13:22:41.570 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48779.log.gz 2024-03-22T13:22:41.571 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43879.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43879.log.gz 2024-03-22T13:22:41.571 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48131.log 2024-03-22T13:22:41.573 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58367.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.58367.log.gz --verbose 2024-03-22T13:22:41.573 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.42945.log 2024-03-22T13:22:41.574 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51912.log 2024-03-22T13:22:41.574 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48131.log.gz 2024-03-22T13:22:41.575 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42945.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42945.log.gz 2024-03-22T13:22:41.576 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42455.log 2024-03-22T13:22:41.576 INFO:teuthology.orchestra.run.smithi092.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-22T13:22:41.577 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57179.log 2024-03-22T13:22:41.577 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51912.log.gz 2024-03-22T13:22:41.578 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56759.log 2024-03-22T13:22:41.578 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42455.log.gz 2024-03-22T13:22:41.580 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57179.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57179.log.gz 2024-03-22T13:22:41.580 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.56531.log 2024-03-22T13:22:41.581 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56759.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56759.log.gz 2024-03-22T13:22:41.581 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45336.log 2024-03-22T13:22:41.583 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55526.log 2024-03-22T13:22:41.583 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56531.log.gz 2024-03-22T13:22:41.584 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45336.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45336.log.gz 2024-03-22T13:22:41.584 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44697.log 2024-03-22T13:22:41.585 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55526.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55526.log.gz 2024-03-22T13:22:41.586 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55712.log 2024-03-22T13:22:41.587 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48443.log 2024-03-22T13:22:41.588 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44697.log.gz 2024-03-22T13:22:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58535.log 2024-03-22T13:22:41.589 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55712.log.gz 2024-03-22T13:22:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49049.log 2024-03-22T13:22:41.590 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48443.log.gz 2024-03-22T13:22:41.591 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56175.log 2024-03-22T13:22:41.592 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58535.log.gz 2024-03-22T13:22:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49049.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49049.log.gz 2024-03-22T13:22:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52393.log 2024-03-22T13:22:41.594 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56175.log.gz 2024-03-22T13:22:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42329.log 2024-03-22T13:22:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-03-22T13:22:41.597 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52393.log.gz 2024-03-22T13:22:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42329.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42329.log.gz 2024-03-22T13:22:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48317.log 2024-03-22T13:22:41.600 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53875.log 2024-03-22T13:22:41.600 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-03-22T13:22:41.601 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56969.log 2024-03-22T13:22:41.601 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48317.log.gz 2024-03-22T13:22:41.602 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53875.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53875.log.gz 2024-03-22T13:22:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.56969.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57803.log 2024-03-22T13:22:41.604 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56969.log.gz 2024-03-22T13:22:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52621.log 2024-03-22T13:22:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56279.log 2024-03-22T13:22:41.606 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57803.log.gz 2024-03-22T13:22:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52621.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-22T13:22:41.608 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52621.log.gz 2024-03-22T13:22:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56279.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56279.log.gz 2024-03-22T13:22:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49613.log 2024-03-22T13:22:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49217.log 2024-03-22T13:22:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49613.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49613.log.gz 2024-03-22T13:22:41.613 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.49217.log: --verbose -- /var/log/ceph/ceph-client.admin.40901.log 2024-03-22T13:22:41.613 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49217.log.gz 2024-03-22T13:22:41.614 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41439.log 2024-03-22T13:22:41.615 INFO:teuthology.orchestra.run.smithi006.stderr: 90.5%/var/log/ceph/ceph-client.admin.40901.log: -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-22T13:22:41.615 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40901.log.gz 2024-03-22T13:22:41.616 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45033.log 2024-03-22T13:22:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41439.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-22T13:22:41.617 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41439.log.gz 2024-03-22T13:22:41.619 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45033.log: gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.43419.log 2024-03-22T13:22:41.619 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45033.log.gz 2024-03-22T13:22:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.43419.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43419.log.gz 2024-03-22T13:22:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48047.log 2024-03-22T13:22:41.625 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53707.log 2024-03-22T13:22:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48047.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48047.log.gz 2024-03-22T13:22:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47753.log 2024-03-22T13:22:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53707.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53707.log.gz 2024-03-22T13:22:41.627 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47170.log 2024-03-22T13:22:41.627 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47753.log.gz 2024-03-22T13:22:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47170.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47170.log.gz 2024-03-22T13:22:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43601.log 2024-03-22T13:22:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57971.log 2024-03-22T13:22:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43601.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43601.log.gz 2024-03-22T13:22:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58619.log 2024-03-22T13:22:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57971.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57971.log.gz 2024-03-22T13:22:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54356.log 2024-03-22T13:22:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58619.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58619.log.gz 2024-03-22T13:22:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55442.log 2024-03-22T13:22:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54356.log: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-22T13:22:41.638 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54356.log.gz 2024-03-22T13:22:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47837.log 2024-03-22T13:22:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55442.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55442.log.gz 2024-03-22T13:22:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55400.log 2024-03-22T13:22:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41313.log 2024-03-22T13:22:41.641 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47837.log.gz 2024-03-22T13:22:41.643 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55400.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55400.log.gz 2024-03-22T13:22:41.643 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47230.log 2024-03-22T13:22:41.645 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41313.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41313.log.gz 2024-03-22T13:22:41.645 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57449.log 2024-03-22T13:22:41.646 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47230.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47230.log.gz 2024-03-22T13:22:41.646 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47608.log 2024-03-22T13:22:41.647 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57449.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57449.log.gz 2024-03-22T13:22:41.648 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46876.log 2024-03-22T13:22:41.649 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47608.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47608.log.gz 2024-03-22T13:22:41.649 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49739.log 2024-03-22T13:22:41.651 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46876.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46876.log.gz 2024-03-22T13:22:41.651 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58181.log 2024-03-22T13:22:41.652 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49739.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49739.log.gz 2024-03-22T13:22:41.652 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43590.log 2024-03-22T13:22:41.654 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58308.log 2024-03-22T13:22:41.654 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58181.log.gz 2024-03-22T13:22:41.655 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57743.log 2024-03-22T13:22:41.655 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43590.log.gz 2024-03-22T13:22:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58308.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58308.log.gz 2024-03-22T13:22:41.657 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52915.log 2024-03-22T13:22:41.657 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57743.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57743.log.gz 2024-03-22T13:22:41.658 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45546.log 2024-03-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45075.log 2024-03-22T13:22:41.659 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52915.log.gz 2024-03-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54524.log 2024-03-22T13:22:41.661 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45546.log.gz 2024-03-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45075.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45075.log.gz 2024-03-22T13:22:41.662 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58265.log 2024-03-22T13:22:41.664 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54524.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54524.log.gz 2024-03-22T13:22:41.664 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51138.log 2024-03-22T13:22:41.665 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58265.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58265.log.gz 2024-03-22T13:22:41.665 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44173.log 2024-03-22T13:22:41.667 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56885.log 2024-03-22T13:22:41.667 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51138.log.gz 2024-03-22T13:22:41.668 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55190.log 2024-03-22T13:22:41.668 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44173.log.gz 2024-03-22T13:22:41.669 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44427.log 2024-03-22T13:22:41.669 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56885.log.gz 2024-03-22T13:22:41.671 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45294.log 2024-03-22T13:22:41.671 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55190.log.gz 2024-03-22T13:22:41.672 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44427.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30672.log 2024-03-22T13:22:41.672 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44427.log.gz 2024-03-22T13:22:41.673 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45294.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45294.log.gz 2024-03-22T13:22:41.674 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44595.log 2024-03-22T13:22:41.675 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.30672.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30672.log.gz 2024-03-22T13:22:41.675 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50135.log 2024-03-22T13:22:41.676 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44595.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44595.log.gz 2024-03-22T13:22:41.676 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43586.log 2024-03-22T13:22:41.678 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55130.log 2024-03-22T13:22:41.678 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50135.log.gz 2024-03-22T13:22:41.679 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43586.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43586.log.gz 2024-03-22T13:22:41.680 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45817.log 2024-03-22T13:22:41.681 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58577.log 2024-03-22T13:22:41.681 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55130.log.gz 2024-03-22T13:22:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45817.log: -- /var/log/ceph/ceph-client.admin.43377.log 2024-03-22T13:22:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58577.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.45817.log.gz 2024-03-22T13:22:41.684 INFO:teuthology.orchestra.run.smithi006.stderr:gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58577.log.gz -5 2024-03-22T13:22:41.684 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-22T13:22:41.685 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50928.log 2024-03-22T13:22:41.686 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43377.log.gz 2024-03-22T13:22:41.687 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46507.log 2024-03-22T13:22:41.689 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50928.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50928.log.gz 2024-03-22T13:22:41.689 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47482.log 2024-03-22T13:22:41.689 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46507.log.gz 2024-03-22T13:22:41.691 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57281.log 2024-03-22T13:22:41.692 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47482.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47482.log.gz 2024-03-22T13:22:41.693 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.57281.log: --verbose -- /var/log/ceph/ceph-client.admin.45117.log 2024-03-22T13:22:41.694 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57281.log.gz 2024-03-22T13:22:41.695 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43910.log 2024-03-22T13:22:41.697 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45117.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45117.log.gz 2024-03-22T13:22:41.698 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50177.log 2024-03-22T13:22:41.698 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43910.log.gz 2024-03-22T13:22:41.699 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54103.log 2024-03-22T13:22:41.700 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50177.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50177.log.gz 2024-03-22T13:22:41.701 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.54103.log: --verbose -- /var/log/ceph/ceph-client.admin.51618.log 2024-03-22T13:22:41.702 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54103.log.gz 2024-03-22T13:22:41.702 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47128.log 2024-03-22T13:22:41.704 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51618.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51618.log.gz 2024-03-22T13:22:41.705 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52267.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47128.log.gz 2024-03-22T13:22:41.705 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.707 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52351.log 2024-03-22T13:22:41.708 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52267.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52267.log.gz 2024-03-22T13:22:41.709 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58493.log 2024-03-22T13:22:41.710 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52351.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52351.log.gz 2024-03-22T13:22:41.711 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58493.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58493.log.gz 2024-03-22T13:22:41.712 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52747.log 2024-03-22T13:22:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47086.log 2024-03-22T13:22:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52747.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52747.log.gz 2024-03-22T13:22:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58997.log 2024-03-22T13:22:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47086.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47086.log.gz 2024-03-22T13:22:41.717 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58913.log 2024-03-22T13:22:41.719 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58997.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58997.log.gz 2024-03-22T13:22:41.719 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54398.log/var/log/ceph/ceph-client.admin.58913.log: 2024-03-22T13:22:41.720 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58913.log.gz 2024-03-22T13:22:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48695.log 2024-03-22T13:22:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54398.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54398.log.gz 2024-03-22T13:22:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52183.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48695.log.gz 2024-03-22T13:22:41.724 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.726 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48527.log 2024-03-22T13:22:41.726 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52183.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52183.log.gz 2024-03-22T13:22:41.727 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-22T13:22:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48527.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48527.log.gz 2024-03-22T13:22:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44385.log 2024-03-22T13:22:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.44385.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44385.log.gz 2024-03-22T13:22:41.733 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56006.log 2024-03-22T13:22:41.733 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-22T13:22:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55274.log 2024-03-22T13:22:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56006.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56006.log.gz 2024-03-22T13:22:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54920.log/var/log/ceph/ceph-client.admin.55274.log: 2024-03-22T13:22:41.737 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55274.log.gz 2024-03-22T13:22:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53665.log 2024-03-22T13:22:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54920.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54920.log.gz 2024-03-22T13:22:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.53665.log: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-22T13:22:41.741 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53665.log.gz 2024-03-22T13:22:41.743 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43594.log 2024-03-22T13:22:41.744 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46069.log 2024-03-22T13:22:41.746 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43594.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43594.log.gz 2024-03-22T13:22:41.747 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58661.log 2024-03-22T13:22:41.748 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46069.log.gz 2024-03-22T13:22:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-22T13:22:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58661.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58661.log.gz 2024-03-22T13:22:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49091.log 2024-03-22T13:22:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.49091.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49091.log.gz 2024-03-22T13:22:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50489.log 2024-03-22T13:22:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46213.log 2024-03-22T13:22:41.759 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50489.log.gz 2024-03-22T13:22:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42807.log 2024-03-22T13:22:41.765 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2024-03-22T13:22:41.768 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48839.log 2024-03-22T13:22:41.769 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42807.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45588.log 2024-03-22T13:22:41.770 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42807.log.gz 2024-03-22T13:22:41.771 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48839.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48839.log.gz 2024-03-22T13:22:41.771 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42765.log 2024-03-22T13:22:41.772 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45588.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45588.log.gz 2024-03-22T13:22:41.774 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43071.log 2024-03-22T13:22:41.777 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42765.log: /var/log/ceph/ceph-client.admin.43071.log: 93.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-22T13:22:41.779 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48881.log 2024-03-22T13:22:41.779 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43071.log.gz 2024-03-22T13:22:41.779 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42765.log.gz 2024-03-22T13:22:41.780 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40640.log 2024-03-22T13:22:41.782 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47963.log 2024-03-22T13:22:41.784 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40640.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40640.log.gz 2024-03-22T13:22:41.784 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55568.log 2024-03-22T13:22:41.784 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48881.log.gz 2024-03-22T13:22:41.787 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47963.log: /var/log/ceph/ceph-client.admin.55568.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55568.log.gz 2024-03-22T13:22:41.788 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55754.log 2024-03-22T13:22:41.789 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47963.log.gz 2024-03-22T13:22:41.790 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57221.log 2024-03-22T13:22:41.791 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55754.log.gz 2024-03-22T13:22:41.793 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45733.log 2024-03-22T13:22:41.795 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57221.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57221.log.gz 2024-03-22T13:22:41.797 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48215.log 2024-03-22T13:22:41.798 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45733.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45733.log.gz 2024-03-22T13:22:41.801 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58139.log 2024-03-22T13:22:41.803 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48215.log.gz 2024-03-22T13:22:41.806 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.58139.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58139.log.gz 2024-03-22T13:22:41.806 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43599.log 2024-03-22T13:22:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46708.log 2024-03-22T13:22:41.810 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43599.log.gz 2024-03-22T13:22:41.811 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44133.log 2024-03-22T13:22:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46708.log: /var/log/ceph/ceph-client.admin.44133.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51702.log 2024-03-22T13:22:41.815 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46708.log.gz 2024-03-22T13:22:41.816 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44133.log.gz 2024-03-22T13:22:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53395.log 2024-03-22T13:22:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51702.log: /var/log/ceph/ceph-client.admin.53395.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46918.log 2024-03-22T13:22:41.822 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51702.log.gz 2024-03-22T13:22:41.823 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53395.log.gz 2024-03-22T13:22:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44511.log 2024-03-22T13:22:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46918.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53521.log 2024-03-22T13:22:41.828 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46918.log.gz/var/log/ceph/ceph-client.admin.44511.log: 2024-03-22T13:22:41.829 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44511.log.gz 2024-03-22T13:22:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53101.log 2024-03-22T13:22:41.831 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53521.log.gz 2024-03-22T13:22:41.833 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43863.log 2024-03-22T13:22:41.835 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53101.log: 87.5%/var/log/ceph/ceph-client.admin.43863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44781.log 2024-03-22T13:22:41.836 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53101.log.gz 2024-03-22T13:22:41.836 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43863.log.gz 2024-03-22T13:22:41.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44781.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44781.log.gz 2024-03-22T13:22:41.839 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47002.log 2024-03-22T13:22:41.842 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57887.log 2024-03-22T13:22:41.843 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47002.log.gz 2024-03-22T13:22:41.845 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44637.log 2024-03-22T13:22:41.845 INFO:teuthology.orchestra.run.smithi092.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-22T13:22:41.849 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.57887.log: /var/log/ceph/ceph-client.admin.44637.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47524.log 2024-03-22T13:22:41.850 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57887.log.gz 2024-03-22T13:22:41.851 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44637.log.gz 2024-03-22T13:22:41.851 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44553.log 2024-03-22T13:22:41.856 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47524.log: /var/log/ceph/ceph-client.admin.44553.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49823.log 2024-03-22T13:22:41.856 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47524.log.gz 2024-03-22T13:22:41.857 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44553.log.gz 2024-03-22T13:22:41.859 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58871.log 2024-03-22T13:22:41.863 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49823.log: /var/log/ceph/ceph-client.admin.58871.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53311.log 2024-03-22T13:22:41.864 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49823.log.gz 2024-03-22T13:22:41.865 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58871.log.gz 2024-03-22T13:22:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53311.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53311.log.gz 2024-03-22T13:22:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46423.log 2024-03-22T13:22:41.871 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42639.log 2024-03-22T13:22:41.872 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46423.log.gz 2024-03-22T13:22:41.872 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50447.log 2024-03-22T13:22:41.875 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53059.log 2024-03-22T13:22:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50447.log: /var/log/ceph/ceph-client.admin.42639.log: 87.6% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42639.log.gz 2024-03-22T13:22:41.877 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50447.log.gz 2024-03-22T13:22:41.877 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58097.log 2024-03-22T13:22:41.880 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53059.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53059.log.gz 2024-03-22T13:22:41.881 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.58097.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58097.log.gz 2024-03-22T13:22:41.882 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54794.log 2024-03-22T13:22:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2024-03-22T13:22:41.885 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54794.log.gz 2024-03-22T13:22:41.887 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43932.log 2024-03-22T13:22:41.892 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43945.log: /var/log/ceph/ceph-client.admin.43932.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45420.log 2024-03-22T13:22:41.892 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2024-03-22T13:22:41.893 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43932.log.gz 2024-03-22T13:22:41.895 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52537.log 2024-03-22T13:22:41.899 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45420.log: /var/log/ceph/ceph-client.admin.52537.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49487.log 2024-03-22T13:22:41.899 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45420.log.gz 2024-03-22T13:22:41.900 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52537.log.gz 2024-03-22T13:22:41.902 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49487.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-22T13:22:41.905 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49487.log.gz/var/log/ceph/ceph-mds.a.log: 2024-03-22T13:22:41.905 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43890.log 2024-03-22T13:22:41.908 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50303.log 2024-03-22T13:22:41.909 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43890.log.gz 2024-03-22T13:22:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42723.log 2024-03-22T13:22:41.915 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50303.log: 87.2%/var/log/ceph/ceph-client.admin.42723.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42723.log.gz 2024-03-22T13:22:41.916 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50303.log.gz 2024-03-22T13:22:41.916 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56843.log 2024-03-22T13:22:41.918 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47314.log 2024-03-22T13:22:41.919 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56843.log: gzip -5 --verbose 88.4% -- /var/log/ceph/ceph-client.admin.58409.log -- replaced with /var/log/ceph/ceph-client.admin.56843.log.gz 2024-03-22T13:22:41.919 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:41.922 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53185.log 2024-03-22T13:22:41.924 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8%/var/log/ceph/ceph-client.admin.58409.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58409.log.gz 2024-03-22T13:22:41.924 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47314.log.gz 2024-03-22T13:22:41.928 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47879.log 2024-03-22T13:22:41.928 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53185.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53185.log.gz 2024-03-22T13:22:41.931 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47879.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47879.log.gz 2024-03-22T13:22:41.954 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/ceph-mds.a.log: file size changed while zipping 2024-03-22T13:22:41.954 INFO:teuthology.orchestra.run.smithi006.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-22T13:22:42.049 INFO:teuthology.orchestra.run.smithi092.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-22T13:22:42.304 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-22T13:22:42.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-22T13:22:42.680 INFO:teuthology.orchestra.run.smithi092.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-22T13:22:42.705 INFO:teuthology.orchestra.run.smithi092.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-22T13:22:42.714 INFO:teuthology.orchestra.run.smithi092.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-22T13:22:42.752 INFO:teuthology.orchestra.run.smithi092.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-22T13:22:42.754 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-03-22T13:22:42.754 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m1.542s 2024-03-22T13:22:42.754 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m6.630s 2024-03-22T13:22:42.754 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m0.255s 2024-03-22T13:22:42.831 INFO:teuthology.orchestra.run.smithi006.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-22T13:22:42.924 INFO:teuthology.orchestra.run.smithi006.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-22T13:22:43.044 INFO:teuthology.orchestra.run.smithi006.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-22T13:22:43.047 INFO:teuthology.orchestra.run.smithi006.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-22T13:22:43.050 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-22T13:22:43.051 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m1.849s 2024-03-22T13:22:43.051 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m6.114s 2024-03-22T13:22:43.051 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m1.752s 2024-03-22T13:22:43.051 INFO:tasks.ceph:Archiving logs... 2024-03-22T13:22:43.051 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/remote/smithi006/log 2024-03-22T13:22:43.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T13:22:43.297 INFO:tasks.ceph.mds.a.smithi006.stderr:==00:00:26:28.867 40641== 2024-03-22T13:22:43.297 INFO:tasks.ceph.mds.a.smithi006.stderr:==00:00:26:28.867 40641== Exit program on first error (--exit-on-first-error=yes) 2024-03-22T13:22:43.371 INFO:tasks.ceph.mds.a.smithi006.stderr:daemon-helper: command failed with exit status 42 2024-03-22T13:22:44.133 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/remote/smithi092/log 2024-03-22T13:22:44.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T13:22:44.833 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-22T13:22:45.065 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e8159ee84f8a432891f349bd6507487b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-22T13:22:45.069 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-22T13:22:45.077 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-22T13:22:45.119 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-22T13:22:45.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T13:22:45.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T13:22:45.263 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-22T13:22:45.271 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-22T13:22:45.271 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T13:22:45.278 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T13:22:45.291 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-22T13:22:45.295 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T13:22:45.295 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-22T13:22:45.295 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 -811us[ -810us] +/- 105ms 2024-03-22T13:22:45.295 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -665us[ -664us] +/- 72ms 2024-03-22T13:22:45.295 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1329us[+1329us] +/- 71ms 2024-03-22T13:22:45.295 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T13:22:45.320 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-22T13:22:45.329 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T13:22:45.329 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-22T13:22:45.330 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 -842us[ -842us] +/- 106ms 2024-03-22T13:22:45.330 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -699us[ -700us] +/- 72ms 2024-03-22T13:22:45.330 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +1279us[+1279us] +/- 71ms 2024-03-22T13:22:45.330 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T13:22:45.331 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-22T13:22:45.341 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-22T13:22:45.341 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-22T13:22:45.369 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T13:22:45.589 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T13:22:45.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T13:22:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T13:22:45.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T13:22:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T13:22:45.779 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-22T13:22:45.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-22T13:22:45.828 INFO:teuthology.task.internal:Duration was 2552.619285 seconds 2024-03-22T13:22:45.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-22T13:22:45.862 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-22T13:22:45.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T13:22:45.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T13:22:45.912 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T13:22:46.019 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T13:22:46.452 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-22T13:22:46.453 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-03-22T13:22:46.453 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T13:22:46.513 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-03-22T13:22:46.513 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T13:22:46.541 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-22T13:22:46.541 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T13:22:46.547 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T13:22:46.629 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-22T13:22:46.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T13:22:46.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T13:22:46.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-22T13:22:46.831 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-22T13:22:46.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T13:22:46.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T13:22:46.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-22T13:22:46.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T13:22:46.960 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T13:22:47.008 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-03-22T13:22:47.058 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-03-22T13:22:47.111 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T13:22:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T13:22:47.157 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T13:22:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T13:22:47.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-22T13:22:47.180 INFO:teuthology.task.internal:Transferring archived files... 2024-03-22T13:22:47.181 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/remote/smithi006 2024-03-22T13:22:47.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T13:22:47.298 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325/remote/smithi092 2024-03-22T13:22:47.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T13:22:47.336 INFO:teuthology.task.internal:Removing archive directory... 2024-03-22T13:22:47.336 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T13:22:47.344 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T13:22:47.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-22T13:22:47.398 INFO:teuthology.task.internal:Not uploading archives. 2024-03-22T13:22:47.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-22T13:22:47.428 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-22T13:22:47.428 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T13:22:47.435 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T13:22:47.448 INFO:teuthology.orchestra.run.smithi092.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 13:22 /home/ubuntu/cephtest 2024-03-22T13:22:47.481 INFO:teuthology.orchestra.run.smithi006.stdout: 394026 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 13:22 /home/ubuntu/cephtest 2024-03-22T13:22:47.484 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-22T13:22:47.494 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-22T13:22:47.557 INFO:teuthology.nuke:Checking targets against current locks 2024-03-22T13:22:47.584 DEBUG:teuthology.nuke:shortname: smithi006 2024-03-22T13:22:47.584 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T13:22:47.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 12:30:53.725282', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6BhfP/x4l8nDPTZnbV3xnODLJoELpLhMe8jZfm+FBBZzddHhUHg7GKzQLOknR/zo4xfve+LI+p66Lb2RTroU0='} 2024-03-22T13:22:47.611 DEBUG:teuthology.nuke:shortname: smithi092 2024-03-22T13:22:47.611 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T13:22:47.634 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 12:30:53.726772', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD4IWQLXvvgXJLm0v+K1UZa29kuyXyIouGFDfsGfDWmjnNHWLCxxQMQGuVH9JaD0cFOC6fhXVNlYmFKQo0WSwsQ='} 2024-03-22T13:22:47.664 INFO:teuthology.orchestra.console.smithi006:Power off 2024-03-22T13:22:47.664 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T13:22:47.687 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-03-22T13:22:47.688 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:22:47.703 INFO:teuthology.orchestra.console.smithi092:Power off 2024-03-22T13:22:47.703 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T13:22:47.712 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-22T13:22:47.725 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-03-22T13:22:47.726 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:22:47.749 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-22T13:22:48.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-22T13:22:51.714 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:22:51.751 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:22:51.825 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-22T13:22:51.861 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-22T13:22:53.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~6s 2024-03-22T13:22:53.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-22T13:22:53.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-03-22T13:22:53.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-22T13:22:53.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-22T13:22:53.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-03-22T13:22:55.828 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:22:55.862 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:22:55.940 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-22T13:22:55.971 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-22T13:22:58.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~11s 2024-03-22T13:22:58.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~5s 2024-03-22T13:22:58.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~5s 2024-03-22T13:22:58.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~5s 2024-03-22T13:22:58.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~5s 2024-03-22T13:22:58.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~5s 2024-03-22T13:22:59.942 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:22:59.973 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T13:23:00.052 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-03-22T13:23:00.083 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-03-22T13:23:00.154 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-03-22T13:23:00.184 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-03-22T13:23:00.289 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} duration: 2552.619284629822 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e8159ee84f8a432891f349bd6507487b status: fail success: false 2024-03-22T13:23:00.289 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T13:23:00.382 INFO:teuthology.run:FAIL